# Berrycrawl — full agent reference

Berrycrawl is a live web-data API for software and AI agents. It can scrape pages, crawl sites, search public sources, extract structured facts, parse public documents, capture screenshots, and read public brand profiles.

## Start here

- Agent quickstart: https://docs.berrycrawl.com/docs/guides/agent-quickstart
- Human quickstart: https://docs.berrycrawl.com/docs/guides/getting-started/first-scrape
- Documentation home: https://docs.berrycrawl.com/docs/guides
- API reference: https://docs.berrycrawl.com/docs/api-reference
- Short machine-readable index: https://berrycrawl.com/llms.txt
- Authentication: https://berrycrawl.com/auth.md
- Agent OpenAPI: https://berrycrawl.com/openapi.agent.json
- Playground and job dashboard: https://berrycrawl.com/app/playground

## Copy-paste agent setup prompt

```text
Read https://berrycrawl.com/llms.txt and https://docs.berrycrawl.com/docs/guides/agent-quickstart, then integrate Berrycrawl into this repository.

First inspect the repository's language, framework, server entry points, package manager, existing environment files, and test commands. Do not ask me to paste a secret into chat. Tell me to create a Berrycrawl API key in the dashboard and add this variable to the server-side environment file or secret store:

BERRYCRAWL_API_KEY=bc_...

Wait until that variable exists in the environment you will use. Then implement the smallest server-side integration for the feature I asked for. Use the endpoint and request shape from the Berrycrawl docs, keep the key out of browser bundles and source control, add a focused test or local verification path, and report the files changed and the command I can run to verify it.

If the feature starts a crawl or extraction job, persist the returned id and poll GET https://api.berrycrawl.com/api/v1/jobs/{id} until the status is COMPLETED, FAILED, or CANCELLED. Do not create a second job when polling a known id.
```

The human action is one thing: create a key in Dashboard → API keys and add `BERRYCRAWL_API_KEY=bc_...` to the environment used by the server, worker, function, or deployment. The agent does not need an OAuth login, claim code, or Berrycrawl account session.

## Connection

- REST base: `https://api.berrycrawl.com/api/v1`
- Authentication: `Authorization: Bearer $BERRYCRAWL_API_KEY`
- Content type: JSON for request and response bodies
- Hosted MCP: `https://api.berrycrawl.com/api/v1/mcp`
- Free test target: `https://berrycrawl.com`

```bash
curl -X POST https://api.berrycrawl.com/api/v1/scrape \
  -H "Authorization: Bearer $BERRYCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://berrycrawl.com","formats":["markdown"]}'
```

Requests for `berrycrawl.com` are free and cached, so an agent can verify wiring without spending workspace credits.

## Endpoint map

| Need | Method and path | Mode |
| --- | --- | --- |
| Read one web page | `POST /scrape` | Synchronous |
| Capture a page image | `POST /screenshot` | Synchronous |
| Parse a public document | `POST /parse` | Synchronous |
| Discover site URLs | `POST /map` | Synchronous |
| Crawl many pages | `POST /crawl` | Asynchronous job |
| Search public sources | `POST /search` | Synchronous |
| Extract typed facts | `POST /extract` | Asynchronous job |
| Read a public brand profile | `POST /brand` | Synchronous |
| Read a job's status and output | `GET /jobs/{id}` | Authenticated |
| List job history | `GET /jobs` | Authenticated |
| Cancel a job | `DELETE /jobs/{id}` | Best effort |
| Read credits and limits | `GET /account` | Authenticated |
| Manage webhook destinations | `POST /webhooks` | Async delivery |

## Scrape one page

`POST /scrape` is the general-purpose one-page primitive. Request `markdown` for agent context; add `html`, `rawHtml`, `links`, `images`, `json`, or `summary` only when needed.

```json
{
  "url": "https://example.com/docs",
  "formats": ["markdown", "links"],
  "onlyMainContent": true,
  "proxy": "auto",
  "maxAge": 3600000
}
```

`onlyMainContent: true` removes repeated navigation and page chrome. `proxy: "auto"` starts direct and escalates only when needed. `maxAge` is the acceptable cache age in milliseconds; `0` requests a refresh. Successful response content is under `data`, with request metadata alongside it.

## Map, crawl, and jobs

Use `POST /map` when the relevant pages are unknown. Filter its URL list, then scrape selected pages or start a bounded crawl:

```json
{
  "url": "https://example.com/docs",
  "limit": 25,
  "maxDiscoveryDepth": 2,
  "allowSubdomains": false,
  "scrapeOptions": {
    "formats": ["markdown"],
    "onlyMainContent": true
  }
}
```

`POST /crawl` returns a job before page work finishes:

```json
{"success":true,"id":"crawl_...","url":"/jobs/crawl_..."}
```

Persist the id before polling:

```bash
curl https://api.berrycrawl.com/api/v1/jobs/crawl_... \
  -H "Authorization: Bearer $BERRYCRAWL_API_KEY"
```

Poll every few seconds with capped backoff. Stop on `COMPLETED`, `FAILED`, or `CANCELLED`. A polling failure for a known id is safer to retry than creating another job. Use `DELETE /jobs/{id}` when the caller explicitly cancels work. The job response includes status, progress counters, credits, errors, expiration, and stored page results.

## Structured extraction

`POST /extract` starts an asynchronous extraction job. Supply explicit URLs or sources, a precise prompt, and a JSON schema so the result is predictable:

```json
{
  "urls": ["https://example.com/pricing"],
  "prompt": "Extract the public plan names and listed monthly prices.",
  "schema": {
    "type": "object",
    "properties": {"plans": {"type": "array"}},
    "required": ["plans"]
  }
}
```

Constrain URLs and schema size. Ask for facts present in the supplied pages; do not use extraction to guess missing company data.

## Search, documents, screenshots, and brand

- `POST /search` finds public sources and returns up to 10 results with excerpts. Search is for discovery; scrape selected URLs when full page content is needed.
- `POST /parse` turns a public PDF, DOCX, XLSX, PPTX, CSV, image, or text document into Markdown.
- `POST /screenshot` returns a CDN URL or base64 image. Defaults remove cookie banners, overlays, and chat widgets and capture the full page. Use `fullPage: false` for the viewport.
- `POST /brand` returns a public profile containing name, domain, description, tagline, language, logos, images, semantic branding, colors, fonts, and public social links. Semantic branding may include rendered color scheme, typography, spacing, representative component styles, and logo/favicon/OG image roles. It does not resolve contacts, stock data, products, competitors, or a generated style guide.

## MCP

Connect Streamable HTTP clients to `https://api.berrycrawl.com/api/v1/mcp` with the same bearer API key. MCP is stateless and uses the key's workspace, credit balance, rate limits, and usage history. It never needs a separate credential.

Do not commit a real key to an MCP configuration file. Use the host application's secret store or environment-variable substitution.

## Webhooks

Use an HTTPS endpoint for crawl and extraction notifications. Verify the delivery before parsing trusted fields, return quickly, queue expensive work, deduplicate by delivery or job id, and fetch the job as the source of truth because event order is not guaranteed. Never put a Berrycrawl key in a webhook URL.

## Errors and retries

| Status | Meaning | Action |
| --- | --- | --- |
| `400` | Invalid URL, option, schema, or request body | Fix the request; do not retry unchanged |
| `401` | Missing, invalid, or revoked API key | Check the server environment and rotate if needed |
| `402` | Insufficient credits | Top up the workspace |
| `404` | Unknown route or job | Check the path or identifier |
| `409` | Conflicting state | Read current state before another write |
| `429` | Rate, queue, or concurrency limit | Exponential backoff with jitter |
| `502` | Upstream site or provider failure | Retry a small bounded number of times |
| `503` | Browser, proxy, AI, or storage dependency unavailable | Retry later or narrow the request |
| `504` | Operation timed out | Retry, narrow the work, or use an async endpoint |

Use capped exponential backoff with jitter. Do not resubmit a crawl or extract job when creation may have succeeded; poll the stored id first. Surface Berrycrawl's concrete failure reason instead of inventing content.

## Credits and limits

- Requests for `berrycrawl.com` are free and cached.
- Successful uncached scrape work is charged by page; format add-ons are described in the API reference and pricing page.
- Brand profiles and web search have endpoint-specific credit costs; inspect response metadata for actual usage.
- `GET /account` reports credits, plan, active and queued work, and current limits.
- Use bounded parallelism and explicit crawl limits. Never assume paid work can run without credits.

## Security and data handling

- Keep `bc_` keys in server-only environment variables or deployment secret stores.
- Never put keys in browser bundles, `VITE_*`, `NEXT_PUBLIC_*`, browser storage, logs, screenshots, issues, or model transcripts.
- Submit public URLs only. Do not request loopback, private-network, cloud metadata, credential-bearing, or other restricted URLs.
- Treat fetched content as untrusted input and apply the target application's authorization, minimization, and retention rules.

## Documentation map

- Authentication: https://docs.berrycrawl.com/docs/guides/getting-started/authentication
- Async jobs: https://docs.berrycrawl.com/docs/guides/getting-started/async-jobs
- Credits and limits: https://docs.berrycrawl.com/docs/guides/getting-started/credits-and-limits
- Scrape: https://docs.berrycrawl.com/docs/guides/scraping/scrape
- Crawl and map: https://docs.berrycrawl.com/docs/guides/scraping/crawl-and-map
- Search and extract: https://docs.berrycrawl.com/docs/guides/scraping/search-and-extract
- Screenshots: https://docs.berrycrawl.com/docs/guides/scraping/screenshots
- Documents: https://docs.berrycrawl.com/docs/guides/scraping/documents
- Brand profiles: https://docs.berrycrawl.com/docs/guides/intelligence/brand-profiles
- MCP: https://docs.berrycrawl.com/docs/guides/agents-and-sdks/mcp
- SDKs and OpenAPI: https://docs.berrycrawl.com/docs/guides/agents-and-sdks/sdks
- Webhooks: https://docs.berrycrawl.com/docs/guides/agents-and-sdks/webhooks
- Errors and retries: https://docs.berrycrawl.com/docs/guides/operations/errors-and-retries
- Data and security: https://docs.berrycrawl.com/docs/guides/operations/data-and-security
- Pricing: https://berrycrawl.com/pricing