I need help integrating TinyFish Web Agent into my project. TinyFish Web Agent is a web automation API that uses natural language to control browsers - no CSS selectors or XPath needed.
**TinyFish Web Agent capabilities:**
- Navigate to websites and perform actions (clicks, form fills, scrolling)
- Extract structured data from any page as JSON
- Handle multi-step workflows with a single API call
- Work on authenticated and bot-protected sites
Please ask me these questions first:
1. What am I building?
- Data extraction / scraping
- Form automation
- Web monitoring
- AI agent with web browsing
- Something else
2. Which endpoint should I use?
- Synchronous (/run) - wait for result, simple code
- Async (/run-async) - start task, poll for result later
- Streaming (/run-sse) - real-time progress updates
3. What's my tech stack?
- TypeScript / JavaScript
- Python
- Other
4. Will I need anti-detection?
- No - standard websites
- Yes - sites with Cloudflare, CAPTCHAs, or bot protection
Then generate code using these patterns:
---
**Environment Setup**
```bash
# Get your API key at https://agent.tinyfish.ai/api-keys
# Add to .env file:
TINYFISH_API_KEY=sk-tinyfish-*****
```
---
**TypeScript - Streaming (Recommended)**
```typescript
import 'dotenv/config'
async function runAutomation(url: string, goal: string) {
const response = await fetch("https://agent.tinyfish.ai/v1/automation/run-sse", {
method: "POST",
headers: {
"X-API-Key": process.env.TINYFISH_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ url, goal }),
});
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop() ?? "";
for (const line of lines) {
if (line.startsWith("data: ")) {
const event = JSON.parse(line.slice(6));
if (event.type === "PROGRESS") {
console.log(`Action: ${event.purpose}`);
} else if (event.type === "COMPLETE") {
if (event.status === "COMPLETED") {
return event.result;
}
throw new Error(event.error?.message || "Automation failed");
}
}
}
}
}
// Example usage
const products = await runAutomation(
"https://example.com/products",
"Extract all product names and prices as JSON"
);
```
---
**TypeScript - Synchronous**
```typescript
import 'dotenv/config'
async function runAutomation(url: string, goal: string) {
const response = await fetch("https://agent.tinyfish.ai/v1/automation/run", {
method: "POST",
headers: {
"X-API-Key": process.env.TINYFISH_API_KEY!,
"Content-Type": "application/json",
},
body: JSON.stringify({ url, goal }),
});
const run = await response.json();
if (run.status === "COMPLETED") return run.result;
throw new Error(run.error?.message || "Automation failed");
}
```
---
**Python - Streaming**
```python
from tinyfish import TinyFish, EventType, RunStatus
client = TinyFish() # Reads TINYFISH_API_KEY from environment
def run_automation(url: str, goal: str):
with client.agent.stream(url=url, goal=goal) as stream:
for event in stream:
if event.type == EventType.PROGRESS:
print(f"Action: {event.purpose}")
elif event.type == EventType.COMPLETE:
if event.status == RunStatus.COMPLETED:
return event.result_json
raise Exception(event.error)
# Example usage
products = run_automation(
"https://example.com/products",
"Extract all product names and prices as JSON"
)
```
---
**Python - Synchronous**
```python
from tinyfish import TinyFish, RunStatus
client = TinyFish() # Reads TINYFISH_API_KEY from environment
def run_automation(url: str, goal: str):
result = client.agent.run(url=url, goal=goal)
if result.status == RunStatus.COMPLETED:
return result.result
raise Exception(result.error)
```
---
**Anti-Detection Mode**
For sites with bot protection, add stealth mode and proxy:
```typescript
body: JSON.stringify({
url: "https://protected-site.com",
goal: "Extract pricing data",
browser_profile: "stealth",
proxy_config: {
enabled: true,
country_code: "US", // Also: GB, CA, DE, FR, JP, AU
},
}),
```
---
**Writing Good Goals**
Be specific about what you want:
```
// Good - specific output format
"Extract product name, price, and availability. Return as JSON array."
// Good - multi-step with numbered actions
"1. Click 'Load More' 3 times 2. Extract all product cards 3. Return as JSON"
// Bad - too vague
"Get the data"
```
---
**Quick Test**
```bash
curl -N -X POST https://agent.tinyfish.ai/v1/automation/run-sse \
-H "X-API-Key: $TINYFISH_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://scrapeme.live/shop",
"goal": "Extract the first 3 product names and prices"
}'
```
---
After asking the questions, generate the appropriate code for my use case. Reference https://docs.tinyfish.ai for additional details.
