zrizDocsLearnSecuritySign inSign up

Run end-to-end tests in GitHub Actions

Start the service and a runner in the job. Then run the tests against localhost. A job has no browser, so the login is an API key from a secret.

Draft. Guide · zz 0.4.0 or later · Updated · Plain text for agents: /docs/github-actions-e2e-tests.md

The tests are the pipelines of the zriz project in your repository. zz reads its API key from the variable ZZ_API_KEY, so the job needs no zz login. The runner has its own token, also a secret.

Before you start

1. Start the service in a workflow

Do

Make the workflow file. The job starts the service on the CI machine and waits until it answers.

.github/workflows/e2e.yml:

name: e2e
on: [push]
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Start the service
        run: <start command> &
      - name: Wait for the health URL
        run: curl --fail --silent --retry 30 --retry-delay 2 --retry-connrefused <health url>

grep 'retry' .github/workflows/e2e.yml

Why

The & keeps the service in operation for the later steps of the job. curl tries again until the service answers, so no test starts too early.

You should see

--retry 30 --retry-delay 2 --retry-connrefused

If not

2. Store the API key as a secret

Do

Your user must approve this step. It copies the API key of your org from your machine to the secrets of the GitHub repository.

zz login wrote the key to the file credentials.json.

The key must be a key of the org in .zriz/project.json. With many logins, use the login of the project org: zz switch <login of the project org>.

Note: zz logout revokes a key from a browser sign-in, and this key is one. Then store a new key.

jq -r '.logins[.current]["api-key"]' "${XDG_CONFIG_HOME:-$HOME/.config}/zriz/credentials.json" | gh secret set ZZ_API_KEY
gh secret list

Why

A job cannot open a browser tab, so it cannot do zz login. A secret keeps the key out of the workflow file and out of the job log.

You should see

ZZ_API_KEY

If not

3. Store a runner token as a secret

Do

Your user must approve this step. It makes a runner token and copies it to the secrets of the GitHub repository.

The token fixes the name and the environment of the runner. The job then starts the runner with the same name and environment.

zz runners token create <name> --env <env> | jq -r '.data.token' | gh secret set ZRIZ_RUNNER_TOKEN
gh secret list

Why

A runner needs a token to connect. A secret keeps the token out of the workflow file and out of the job log.

You should see

ZRIZ_RUNNER_TOKEN

If not

4. Add the test run

Do

Add the last four steps to the workflow file. Keep the values of step 1.

.github/workflows/e2e.yml:

name: e2e
on: [push]
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Start the service
        run: <start command> &
      - name: Wait for the health URL
        run: curl --fail --silent --retry 30 --retry-delay 2 --retry-connrefused <health url>
      - name: Install zz
        run: curl -fsSL https://zriz.io/install | sh
      - name: Write the project files
        env:
          ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
        run: ~/.local/bin/zz init --url <base url>
      - name: Start a runner
        env:
          ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
          ZRIZ_RUNNER_TOKEN: ${{ secrets.ZRIZ_RUNNER_TOKEN }}
        run: |
          mkdir -p .zriz/runner
          printf 'ZRIZ_RUNNER_TOKEN=%s\n' "$ZRIZ_RUNNER_TOKEN" > .zriz/runner/runner.env
          chmod 600 .zriz/runner/runner.env
          ~/.local/bin/zz runners init --env <env> --name <name> --start
      - name: Run the tests
        env:
          ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
        run: ~/.local/bin/zz run

The step Start a runner writes runner.env first. zz runners init then writes only config.json and compose.yml, and makes no new token. It waits until the runner is connected.

zz run with no name runs each pipeline. Add --fail-fast to stop at the first pipeline that does not pass.

Note: zz init --url writes .zriz/environments/local.json with TARGET_URL only. If a pipeline uses a different value, write that file in the job. Pipelines gives its format.

grep 'secrets' .github/workflows/e2e.yml

Why

environments/local.json is not in the repository, so the job makes it. The runner is a container that reaches localhost of the CI machine as host.docker.internal.

You should see

ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}

If not

5. Push and read the result

Do

The job takes some minutes. The last command prints the state of the newest job.

git add .github/workflows/e2e.yml
git commit -m "Run end-to-end tests in CI"
git push
gh run list --workflow e2e.yml --limit 1 --json status,conclusion --jq '.[0] | .status + " " + .conclusion'

Why

zz run ends with exit code 0 only when each run has status pass. Each other exit code fails the job.

You should see

completed success

If not

Exit codes of the test run

Exit code of zz run Cause
0 Each run has status pass.
1 A run has status fail.
3 The key is absent, or the cloud refused it.
4 zz cannot reach the cloud.
5 The cloud answered with an error, or the org has no free place for 120 s.
6 A project file is not correct.
7 A run has status skip, and no run failed.
8 A run has status error.

A free org runs one pipeline at a time. Two jobs of one org wait for each other, up to 120 seconds. Then zz ends with exit code 5.

Where the runner runs

Each run needs a runner. This guide starts the runner in the job, next to the service. Nothing in your network must be reachable from outside.

The image tag comes from the zz release. This release writes ghcr.io/zriztech/runner:0.2.0.

Check that the tag exists before you rely on it. The runner README names the images.

A runner in the job fits a service that the job starts. A deployed runner fits a shared environment.

Deploy it next to the service: Deploy a zriz runner with Docker.

Then drop the step Start a runner. The file .zriz/environments/<env>.json must exist in the job. Run zz run --env <env>.

The job needs Docker with Compose. Check this on your CI image.

GitHub hosted ubuntu-latest runners are expected to have both. This guide did not check that.

zriz does not mark a run as a run from CI. The run shows in zz runs like any other run.