# Berrycrawl

Berrycrawl is a live web-data API for software and AI agents. It returns clean page content, screenshots, parsed documents, site maps, search results, structured extraction, crawl jobs, and public brand profiles.

## Start here

- Agent quickstart: https://docs.berrycrawl.com/docs/guides/agent-quickstart
- Human quickstart: https://docs.berrycrawl.com/docs/guides/getting-started/first-scrape
- Documentation home: https://docs.berrycrawl.com/docs/guides
- API reference: https://docs.berrycrawl.com/docs/api-reference
- OpenAPI contract: https://berrycrawl.com/openapi.agent.json
- Full machine-readable docs: https://berrycrawl.com/llms-full.txt
- Authentication details: https://berrycrawl.com/auth.md
- Playground and job dashboard: https://berrycrawl.com/app/playground

## Copy-paste agent setup

Give a coding agent this prompt:

```text
Read https://berrycrawl.com/llms.txt and https://docs.berrycrawl.com/docs/guides/agent-quickstart, then integrate berrycrawl into this repository. Inspect the repo's runtime, server entry points, env conventions, and test commands. Tell the human to create a Berrycrawl API key and add BERRYCRAWL_API_KEY=bc_... to the server environment or secret store. Never ask for the key in chat, never commit it, and never expose it in browser code. Once the variable exists, implement the smallest server-side wrapper for the requested feature, use the Berrycrawl docs for the request shape, add a focused test or local verification path, and report the files changed and verification command. For crawl or extract, save the returned id and poll GET /jobs/{id} until COMPLETED, FAILED, or CANCELLED.
```

The human action is one thing: add `BERRYCRAWL_API_KEY=bc_...` to the environment used by the server, worker, function, or deployment. The agent does not need an OAuth login, claim code, or Berrycrawl account session.

## Connection

- REST base: `https://api.berrycrawl.com/api/v1`
- Authentication: `Authorization: Bearer $BERRYCRAWL_API_KEY`
- Content type: JSON for request and response bodies
- MCP endpoint: `https://api.berrycrawl.com/api/v1/mcp`
- Free integration test: requests for `https://berrycrawl.com` are free and cached

Example:

```bash
curl -X POST https://api.berrycrawl.com/api/v1/scrape \
  -H "Authorization: Bearer $BERRYCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://berrycrawl.com","formats":["markdown"]}'
```

## Endpoint map

| Need | Method and path | Behavior |
| --- | --- | --- |
| Read one web page | `POST /scrape` | Synchronous; markdown, HTML, raw HTML, links, images, JSON extraction, or summary |
| Capture a web page | `POST /screenshot` | Synchronous; PNG, JPEG, or WebP URL/base64 output |
| Parse a document | `POST /parse` | Synchronous; public PDF, DOCX, XLSX, PPTX, CSV, image, or text to Markdown |
| Discover site URLs | `POST /map` | Synchronous URL discovery |
| Crawl many pages | `POST /crawl` | Asynchronous; returns a `crawl_...` job id |
| Search the web | `POST /search` | Synchronous structured search results |
| Extract typed data | `POST /extract` | Asynchronous; returns an `extract_...` job id |
| Read a brand profile | `POST /brand` | Synchronous public name, description, assets, semantic branding, colors, fonts, and socials |
| Read a job | `GET /jobs/{id}` | Status, progress, errors, and available output |
| List jobs | `GET /jobs` | Authenticated crawl and extraction job history |
| Cancel a job | `DELETE /jobs/{id}` | Cancel pending or running work |
| Receive job events | `POST /webhooks` in the dashboard or request webhook options | Signed asynchronous delivery |

## Async jobs

`POST /crawl` and `POST /extract` return a job id before page work completes. Persist the id, then poll:

```bash
curl https://api.berrycrawl.com/api/v1/jobs/crawl_... \
  -H "Authorization: Bearer $BERRYCRAWL_API_KEY"
```

Stop on `COMPLETED`, `FAILED`, or `CANCELLED`. Do not create a new job when a poll fails for a known id. Use explicit page, depth, URL, and schema limits. For long-running workflows, use webhooks and keep the job read as the source of truth.

## Response and cost rules

- Successful scrape responses place requested content under `data` and include metadata.
- Crawl responses include progress and stored page results; extraction responses include the final structured result when complete.
- `POST /scrape` is the general one-page primitive. `onlyMainContent: true` removes repeated navigation and page chrome.
- `proxy: "auto"` is the default escalation policy. Proxy credentials never belong in client requests.
- `maxAge` controls acceptable scrape cache age; `maxAge: 0` requests a refresh.
- Failed, blocked, and cached work is not charged as successful page work. Check the current pricing and credit rules in the docs before estimating spend.

## Error handling

- `400`: fix the URL, options, request body, or schema; do not retry unchanged.
- `401`: check that the server received a valid, unrevoked `BERRYCRAWL_API_KEY`.
- `402`: the workspace needs credits.
- `404`: check the route or job id.
- `429`, `502`, `503`, `504`: retry a bounded number of times with exponential backoff and jitter.

## Security rules

Keep `bc_` keys in server-only env files or deployment secret stores. Never put them in frontend bundles, browser storage, logs, screenshots, issues, or model transcripts. Validate caller-provided URLs and keep crawl scope bounded. Do not use Berrycrawl to access private networks, metadata services, or URLs containing credentials.

## Documentation map

- Authentication: https://docs.berrycrawl.com/docs/guides/getting-started/authentication
- Async jobs: https://docs.berrycrawl.com/docs/guides/getting-started/async-jobs
- Scraping: https://docs.berrycrawl.com/docs/guides/scraping/scrape
- Crawl and map: https://docs.berrycrawl.com/docs/guides/scraping/crawl-and-map
- Search and extract: https://docs.berrycrawl.com/docs/guides/scraping/search-and-extract
- Screenshots: https://docs.berrycrawl.com/docs/guides/scraping/screenshots
- Documents: https://docs.berrycrawl.com/docs/guides/scraping/documents
- Brand profiles: https://docs.berrycrawl.com/docs/guides/intelligence/brand-profiles
- MCP: https://docs.berrycrawl.com/docs/guides/agents-and-sdks/mcp
- SDKs and OpenAPI: https://docs.berrycrawl.com/docs/guides/agents-and-sdks/sdks
- Webhooks: https://docs.berrycrawl.com/docs/guides/agents-and-sdks/webhooks
- Errors and retries: https://docs.berrycrawl.com/docs/guides/operations/errors-and-retries
- Data and security: https://docs.berrycrawl.com/docs/guides/operations/data-and-security
- Pricing: https://berrycrawl.com/pricing
- Terms: https://berrycrawl.com/terms
- Privacy: https://berrycrawl.com/privacy