End-to-end tests an AI agent can write
Give your coding agent one instruction. It installs the zriz CLI, writes one test for a flow of your service, and runs it. The work is done when the run has status pass.
The agent works in the folder of your service. You do two things: you paste the instruction, and you approve one login.
Before you start
- The computer has macOS or Linux:
uname -s curlis installed:curl --version- The current folder is the folder of your service, not the home folder:
test "$PWD" != "$HOME"
1. Give the agent the instruction
Do
Start your service. Start the agent in the folder of your service, and paste the text below.
<url>is the base URL of your service. Example:http://localhost:3000.<flow>is the flow to test, in your own words. Example:register a user, then read the profile.
Read https://zriz.io/llms.txt and obey its rules.
Do its start commands in this folder. The base URL of the service is <url>.
Run `zz guide`. Then write one pipeline that tests this flow: <flow>.
You are done only when `zz run` of that pipeline answers status `pass`.
If a step needs me, stop and tell me what to do.
The first line of the text needs a network. Check that this machine can read the rules:
curl -fsSL https://zriz.io/llms.txt
Why
Each command of zz prints one JSON object, and its hints name the next command. zz never asks a question, so the agent works alone until the login.
You should see
# zriz
## Rules
If not
Could not resolve host: The machine has no network. Connect it, then do the command again.
2. Approve the login
Do
Your user must approve the login in a browser tab. The zz login of the agent opens the tab. Sign in if asked, then click Yes, connect.
The step Log in of the quick start gives the case of a machine with no browser.
With many logins, the agent uses --org <login name or org id>, for example zz --org work run hello. It does not use zz switch <login>, because that changes the file for every terminal.
After the approval, check the login:
~/.local/bin/zz doctor
Why
The login gives this computer an API key of your org. An agent cannot sign in for you.
You should see
{"ok":true,"command":"doctor",
"name":"logged in","ok":true
If not
"ok":false: The login is not complete. Tell the agent to dozz loginagain, then approve in the tab.
3. Find the run of the agent
Do
Wait until the agent says that it is done. Then do the command in the folder of your service.
Find a row whose pipeline is not the sample hello. Its pipeline is the test that the agent wrote.
~/.local/bin/zz runs
Why
The command lists each run of this project. A row names the pipeline and the status, so you do not depend on the words of the agent.
You should see
{"ok":true,"command":"runs",
"pipeline":"<pipeline>"
"status":"pass"
If not
"runs":[]: The project has no run. The agent did not dozz run. Give it the instruction again."status":"fail": If no row of the new pipeline haspass, the agent is not done. Tell it to read the failed run and to correct the test.not a zriz project: Go to the folder of your service. Then do the command again.
4. Run the test
Do
Read the file .zriz/pipelines/<pipeline>.json. Its steps must be the flow that you named.
<pipeline>is the name of the pipeline from step 3.
~/.local/bin/zz run <pipeline>
Why
A run that you start shows that the test passes now, against your service as it is.
You should see
"status":"pass"
If not
"status":"fail": Give the reply to the agent. The command in itshintsshows the failed step.no pipeline: The name is not correct. Use the name from step 3, not a path.too-many-runs: The org is at the limit of the free tier. Read What happens at the limit?
Next
- For AI agents: the commands that an agent uses to find its way.
- Pipelines: the format that
zz guideprints. - Test an API call and the database row: a test that also reads your database.
- Security: what a run sends to zriz.io.