---
kind: guide
zz: "0.4.0"
updated: 2026-10-09
state: draft
rules: https://zriz.io/llms.txt
---
# Run end-to-end tests in GitHub Actions

Start the service and a runner in the job. Then run the tests against `localhost`. A job has no browser, so the login is an API key from a secret.

The tests are the [pipelines](https://zriz.io/docs/words.md#pipeline) of the zriz [project](https://zriz.io/docs/words.md#project) in your repository. `zz` reads its API key from the variable `ZZ_API_KEY`, so the job needs no `zz login`. The runner has its own token, also a secret.

## Before you start {#before-you-start}

- The project folder is in the repository: `git ls-files --error-unmatch .zriz/project.json`
- `zz` is logged in on your machine: `zz runners`
- The project has the sample pipeline: `zz validate hello`
- The GitHub CLI is logged in: `gh auth status`
- `jq` is installed: `jq --version`

## 1. Start the service in a workflow {#start-the-service}

### Do

Make the workflow file. The job starts the service on the CI machine and waits until it answers.

- `<start command>` is the command that starts your service. Example: `npm start`.
- `<health url>` is a URL of your service that answers status 200. Example: `http://localhost:3000/health`.

`.github/workflows/e2e.yml`:

```yaml
name: e2e
on: [push]
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Start the service
        run: <start command> &
      - name: Wait for the health URL
        run: curl --fail --silent --retry 30 --retry-delay 2 --retry-connrefused <health url>
```

```sh
grep 'retry' .github/workflows/e2e.yml
```

### Why

The `&` keeps the service in operation for the later steps of the job. `curl` tries again until the service answers, so no test starts too early.

### You should see

```text
--retry 30 --retry-delay 2 --retry-connrefused
```

### If not

- `No such file or directory`: Make the file in the root folder of the repository. Then do the command again.

## 2. Store the API key as a secret {#store-the-key}

### Do

Your user must approve this step. It copies the API key of your [org](https://zriz.io/docs/words.md#org) from your machine to the secrets of the GitHub repository.

`zz login` wrote the key to the file `credentials.json`.

The key must be a key of the org in `.zriz/project.json`. With many logins, use the login of the project org: `zz switch <login of the project org>`.

> Note: `zz logout` revokes a key from a browser sign-in, and this key is one. Then store a new key.

```sh
jq -r '.logins[.current]["api-key"]' "${XDG_CONFIG_HOME:-$HOME/.config}/zriz/credentials.json" | gh secret set ZZ_API_KEY
gh secret list
```

### Why

A job cannot open a browser tab, so it cannot do `zz login`. A secret keeps the key out of the workflow file and out of the job log.

### You should see

```text
ZZ_API_KEY
```

### If not

- `No such file or directory`: This machine has no login. Do `zz login`. Then do the commands again.

## 3. Store a runner token as a secret {#store-the-runner-token}

### Do

Your user must approve this step. It makes a [runner](https://zriz.io/docs/words.md#runner) token and copies it to the secrets of the GitHub repository.

- `<name>` is the name of the runner. Use the project name, the `name` in `.zriz/project.json`.
- `<env>` is the environment that the runner serves. The sample project has `local`.

The token fixes the name and the environment of the runner. The job then starts the runner with the same name and environment.

```sh
zz runners token create <name> --env <env> | jq -r '.data.token' | gh secret set ZRIZ_RUNNER_TOKEN
gh secret list
```

### Why

A runner needs a token to connect. A secret keeps the token out of the workflow file and out of the job log.

### You should see

```text
ZRIZ_RUNNER_TOKEN
```

### If not

- `not logged in`: Do `zz login`. Then do the commands again.
- `jq: error`: The reply of `zz` has no token. Run `zz runners token create <name> --env <env>` alone and read the error.

## 4. Add the test run {#add-the-test-run}

### Do

Add the last four steps to the workflow file. Keep the values of step 1.

- `<base url>` is the base URL of your service in the job. Example: `http://localhost:3000`.
- `<name>` and `<env>` are the values of step 3.

`.github/workflows/e2e.yml`:

```yaml
name: e2e
on: [push]
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Start the service
        run: <start command> &
      - name: Wait for the health URL
        run: curl --fail --silent --retry 30 --retry-delay 2 --retry-connrefused <health url>
      - name: Install zz
        run: curl -fsSL https://zriz.io/install | sh
      - name: Write the project files
        env:
          ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
        run: ~/.local/bin/zz init --url <base url>
      - name: Start a runner
        env:
          ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
          ZRIZ_RUNNER_TOKEN: ${{ secrets.ZRIZ_RUNNER_TOKEN }}
        run: |
          mkdir -p .zriz/runner
          printf 'ZRIZ_RUNNER_TOKEN=%s\n' "$ZRIZ_RUNNER_TOKEN" > .zriz/runner/runner.env
          chmod 600 .zriz/runner/runner.env
          ~/.local/bin/zz runners init --env <env> --name <name> --start
      - name: Run the tests
        env:
          ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
        run: ~/.local/bin/zz run
```

The step `Start a runner` writes `runner.env` first. `zz runners init` then writes only `config.json` and `compose.yml`, and makes no new token. It waits until the runner is connected.

`zz run` with no name runs each pipeline. Add `--fail-fast` to stop at the first pipeline that does not pass.

> Note: `zz init --url` writes `.zriz/environments/local.json` with `TARGET_URL` only. If a pipeline uses a different value, write that file in the job. [Pipelines](https://zriz.io/docs/pipelines.md) gives its format.

```sh
grep 'secrets' .github/workflows/e2e.yml
```

### Why

`environments/local.json` is not in the repository, so the job makes it. The runner is a container that reaches `localhost` of the CI machine as `host.docker.internal`.

### You should see

```text
ZZ_API_KEY: ${{ secrets.ZZ_API_KEY }}
```

### If not

- `No such file or directory`: Do step 1 again. Then add the four steps to the file.
- `runner-start-failed`: [A runner on this machine](https://zriz.io/docs/runner.md#on-this-machine) gives the fix for each `data.step`.
- `runner-unavailable`: The runner did not connect. Check that `<name>` and `<env>` are the values of the token.

## 5. Push and read the result {#push}

### Do

The job takes some minutes. The last command prints the state of the newest job.

```sh
git add .github/workflows/e2e.yml
git commit -m "Run end-to-end tests in CI"
git push
gh run list --workflow e2e.yml --limit 1 --json status,conclusion --jq '.[0] | .status + " " + .conclusion'
```

### Why

`zz run` ends with exit code 0 only when each [run](https://zriz.io/docs/words.md#run) has status `pass`. Each other exit code fails the job.

### You should see

```text
completed success
```

### If not

- `queued`: The job did not start yet. Wait 30 seconds. Then do the last command again.
- `in_progress`: The job is not complete. Wait 30 seconds. Then do the last command again.
- `completed failure`: Open the log of the job on GitHub. The step `Run the tests` shows the JSON reply of `zz run`.
- `org-mismatch`: The key in `ZZ_API_KEY` is of a different org than `org` in `.zriz/project.json`. Do `zz switch <login>` for the project org. Then do step 2 again.
- `not logged in`: The job log shows this text when the secret has no value. Do step 2 again.
- `"status":"fail"`: The job log shows the failed [step](https://zriz.io/docs/words.md#step) in the reply. Correct the service or the pipeline.

## Exit codes of the test run {#exit-codes}

| Exit code of `zz run` | Cause |
|---|---|
| 0 | Each run has status `pass`. |
| 1 | A run has status `fail`. |
| 3 | The key is absent, or the cloud refused it. |
| 4 | `zz` cannot reach the cloud. |
| 5 | The cloud answered with an error, or the org has no free place for 120 s. |
| 6 | A project file is not correct. |
| 7 | A run has status `skip`, and no run failed. |
| 8 | A run has status `error`. |

A free org runs one pipeline at a time. Two jobs of one org wait for each other, up to 120 seconds. Then `zz` ends with exit code 5.

## Where the runner runs {#where}

Each run needs a runner. This guide starts the runner in the job, next to the service. Nothing in your network must be reachable from outside.

The image tag comes from the `zz` release. This release writes `ghcr.io/zriztech/runner:0.2.0`.

Check that the tag exists before you rely on it. The [runner README](https://github.com/ZrizTech/runner#container-image) names the images.

A runner in the job fits a service that the job starts. A deployed runner fits a shared environment.

Deploy it next to the service: [Deploy a zriz runner with Docker](https://zriz.io/docs/runner.md).

Then drop the step `Start a runner`. The file `.zriz/environments/<env>.json` must exist in the job. Run `zz run --env <env>`.

The job needs Docker with Compose. Check this on your CI image.

GitHub hosted `ubuntu-latest` runners are expected to have both. This guide did not check that.

zriz does not mark a run as a run from CI. The run shows in `zz runs` like any other run.