> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tinyfish.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Your Web App on localhost or a Preview URL

> Describe a user flow in plain English and an AI agent clicks through your localhost or preview deploy, then returns pass or fail as JSON

> [!NOTE]
>
> **Capability:** Strict mode + `output_schema` (Strict mode is beta — enroll at [agent.tinyfish.ai/beta](https://agent.tinyfish.ai/beta) to use it)

Use this to check a user flow — sign-up, checkout, a settings form — on a build that isn't live yet. Describe the flow in plain English, TinyAgent clicks through it in a real browser, and you get back pass or fail with a reason. Because that browser runs in TinyFish's cloud, your app needs a URL the internet can reach: a preview deploy or a tunnel to `localhost`.

![TinyAgent logs in to a demo shop, adds a backpack to the cart, checks out, and confirms the order page — returning passed: true](https://mintcdn.com/tinyfish/4TzYEN-cACiNG0l8/images/agent-tutorial/qa-your-app/run.gif?s=34779c0b5f6632d27375952b69e8cb79)

TinyAgent logs in to a demo shop, adds a backpack to the cart, checks out, and confirms the order page — returning `passed: true`.

## Steps

1. **Give the agent a URL it can reach.** Use your preview deploy's URL, or open a tunnel to your local server.
2. **Write the test as steps plus an expected result.** Number the steps, use exact values, end with "Expected result: …", and tell the agent to stop and report if any step can't be done as written.
3. **Run it in Strict mode with a pass/fail schema.** Strict mode stops at the first surprise instead of working around it — which is what you want when the surprise is a bug.
4. **Check `passed`, not the run status.** If it failed, open the run in the dashboard to see what happened at each step.

### Get a URL the agent can reach

> [tab] Preview deploy
>
> Use the preview URL your host gives you (Vercel, Netlify, etc.). If Vercel Deployment Protection is on, add your bypass secret to the URL — it's under **Project Settings → Deployment Protection → Protection Bypass for Automation**:
>
> ```txt theme={null}
> https://my-app-git-feature.vercel.app/?x-vercel-protection-bypass=YOUR_SECRET&x-vercel-set-bypass-cookie=true
> ```

> [tab] localhost
>
> Open a tunnel to your local server, then use the `https://` URL it prints:
>
> ```bash theme={null}
> # Cloudflare (no account needed)
> cloudflared tunnel --url http://localhost:3000
>
> # or ngrok
> ngrok http 3000
> ```
>
> Keep both your app and the tunnel running until the test finishes.

### Run the test

> [!NOTE]
>
> **Beta:**`agent_config.mode: "strict"` and `max_steps` are both beta — enroll at [agent.tinyfish.ai/beta](https://agent.tinyfish.ai/beta) to use either. A non-beta request returns `403 FORBIDDEN` for `mode: "strict"` just as it does for `max_steps`; `max_duration_seconds` alone is generally available, no enrollment needed.

Once enrolled in the beta, this example tests the checkout on a public demo shop, so you can run it as-is. Swap in your own URL and steps.

> [tab] Python
>
> ```py theme={null}
> from tinyfish import TinyFish, AgentConfig
>
> client = TinyFish()
>
> run = client.agent.run(
>     url="https://www.saucedemo.com",  # your preview or tunnel URL
>     goal="""Test the checkout flow:
>         1. Log in with username "standard_user" and password "secret_sauce"
>         2. Add "Sauce Labs Backpack" to the cart
>         3. Open the cart and click "Checkout"
>         4. Enter first name "Test", last name "User", postal code "94107", then click "Continue"
>         5. Click "Finish"
>
>         Expected result: the page shows "Thank you for your order!"
>
>         If any step can't be completed exactly as written, stop and report which step failed.
>         Set failed_step to null if every step passed.
>     """,
>     agent_config=AgentConfig(mode="strict", max_steps=30),
>     output_schema={
>         "type": "object",
>         "properties": {
>             "passed": {"type": "boolean"},
>             "failed_step": {"type": "integer", "nullable": True},
>             "reason": {"type": "string"},
>         },
>         "required": ["passed", "failed_step", "reason"],
>     },
> )
>
> if run.error:
>     raise SystemExit(f"Run did not complete: {run.error}")
> result = run.result or {}
> if not result.get("passed"):
>     raise SystemExit(f"FAILED at step {result.get('failed_step')}: {result.get('reason')}")
> print("PASSED:", result["reason"])
> ```

> [tab] TypeScript
>
> ```ts theme={null}
> import { TinyFish } from "@tiny-fish/sdk";
>
> type TestResult = { passed: boolean; failed_step: number | null; reason: string };
>
> const client = new TinyFish();
>
> const run = await client.agent.run({
>   url: "https://www.saucedemo.com", // your preview or tunnel URL
>   goal: `Test the checkout flow:
>     1. Log in with username "standard_user" and password "secret_sauce"
>     2. Add "Sauce Labs Backpack" to the cart
>     3. Open the cart and click "Checkout"
>     4. Enter first name "Test", last name "User", postal code "94107", then click "Continue"
>     5. Click "Finish"
>
>     Expected result: the page shows "Thank you for your order!"
>
>     If any step can't be completed exactly as written, stop and report which step failed.
>     Set failed_step to null if every step passed.
>   `,
>   agent_config: { mode: "strict", max_steps: 30 },
>   output_schema: {
>     type: "object",
>     properties: {
>       passed: { type: "boolean" },
>       failed_step: { type: "integer", nullable: true },
>       reason: { type: "string" },
>     },
>     required: ["passed", "failed_step", "reason"],
>   },
> });
>
> if (run.error) {
>   console.error("Run did not complete:", run.error);
>   process.exit(1);
> }
> const result = run.result as TestResult | null;
> if (!result?.passed) {
>   console.error(`FAILED at step ${result?.failed_step}: ${result?.reason}`);
>   process.exit(1);
> }
> console.log("PASSED:", result.reason);
> ```

**Result**

```json theme={null}
{
  "passed": true,
  "failed_step": null,
  "reason": "All steps completed successfully. Logged in, added Sauce Labs Backpack to the cart, checked out as Test User 94107, and clicked Finish. The page now shows 'Thank you for your order!'"
}
```

The script exits with an error when the test fails, so you can drop it into CI. To run several tests at once, start one run per test — see [Run TinyAgent Across Multiple URLs](https://docs.tinyfish.ai/agent/how-to/multiple-urls.md). For tests that log in, use a test account, and keep its password in Vault rather than the goal once you're past local testing — see [Login With Vault](https://docs.tinyfish.ai/agent/how-to/vault-login.md).

> [!NOTE]
>
> **Tip — from Claude Code, Cursor, ChatGPT or Claude:** paste your preview or tunnel URL with the steps, e.g. "Use TinyFish to test sign-up on https\://…: create an account with [test@example.com](mailto:test@example.com) and check the dashboard loads." See [Run TinyAgent From ChatGPT or Claude](https://docs.tinyfish.ai/agent/how-to/run-with-chatgpt-claude.md) or [MCP Integration](https://docs.tinyfish.ai/mcp-integration.md).

## Common Mistakes

> [accordion] Sending [http://localhost:3000](http://localhost:3000) as the URL
>
> The agent's browser runs in the cloud, so `localhost` points at that cloud machine, not yours — the run reports the site can't be reached. Open a tunnel and use its `https://` URL.

> [accordion] The run sees a Vercel login page instead of your app
>
> Vercel Deployment Protection is blocking the agent. Add `x-vercel-protection-bypass` and `x-vercel-set-bypass-cookie=true` to the URL, using the **Protection Bypass for Automation** secret.

> [accordion] The run is COMPLETED, but the test failed
>
> `COMPLETED` means the browser ran, not that your app works. Always check `result.passed` — see [Runs](https://docs.tinyfish.ai/agent/explanation/runs.md#completed-doesnt-mean-the-goal-succeeded).

> [accordion] The agent works around the bug and reports a pass
>
> In default mode the agent tries alternatives when something looks wrong — a missing button, an unexpected popup — which can hide the bug you're testing for. Use `mode: "strict"` so it stops instead. See [Agent Mode, Max Steps, and Max Duration](https://docs.tinyfish.ai/agent/explanation/agent-modes.md).

> [accordion] failed\_step is filled in even though the test passed
>
> Without an instruction, the agent may fill every field in the schema. Make `failed_step` nullable and add "Set failed\_step to null if every step passed" to the goal, as in the sample.

> [accordion] Yesterday's tunnel URL stopped working
>
> Quick tunnels get a new URL every time they restart. Copy the new one into your test. If your app checks allowed origins or OAuth redirect URLs, add the tunnel URL there too.

## Related

> [card] [Agent Mode, Max Steps, and Max Duration](https://docs.tinyfish.ai/agent/explanation/agent-modes.md)
>
> Why Strict mode suits tests

> [card] [Runs](https://docs.tinyfish.ai/agent/explanation/runs.md)
>
> Why COMPLETED doesn't mean your test passed

> [card] [Get a TinyAgent Run as Structured JSON](https://docs.tinyfish.ai/agent/how-to/structured-output.md)
>
> More on `output_schema`

> [card] [Run TinyAgent Across Multiple URLs](https://docs.tinyfish.ai/agent/how-to/multiple-urls.md)
>
> Run a whole test suite at once


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.