One key. Six endpoints. Every site on the open web — crawl, monitor, scrape, search, transform, render.
# Crawl every page on a site to clean markdown curl https://api.crawlcrawl.com/v1/crawls \ -H "Authorization: Bearer crk_..." \ -d '{"url":"https://news.ycombinator.com","max_pages":50}' # → { "id": 102, "status": "queued", "url": "..." }
# Watch a page; webhook on change only curl https://api.crawlcrawl.com/v1/crawls \ -H "Authorization: Bearer crk_..." \ -d '{ "url":"https://competitor.com/pricing", "cron":"0 */6 * * *", "webhook_url":"https://you.example/hook", "return_only_changed":true }'
# Bypass Cloudflare / Akamai / captchas curl https://api.crawlcrawl.com/v1/cloud/unblock \ -H "Authorization: Bearer crk_..." \ -d '{"url":"https://hard-site.example"}' # → clean markdown via residential proxies + headless browser
# Live SERP results, one call curl https://api.crawlcrawl.com/v1/cloud/search \ -H "Authorization: Bearer crk_..." \ -d '{"query":"rust async tutorial","num":10}' # → { "items": [...], "result_count": 10 }
# HTML or PDF in → clean markdown out curl https://api.crawlcrawl.com/v1/cloud/transform \ -H "Authorization: Bearer crk_..." \ -d '{"data":"<html>...</html>","return_format":"markdown"}'
# Run any URL through a real Chrome curl https://api.crawlcrawl.com/v1/cloud/render \ -H "Authorization: Bearer crk_..." \ -d '{"url":"https://spa-app.example","return_format":"raw"}'
Four jobs people usually buy four tools for — plus the helpers that make them actually usable. All from one key.
Point at any URL. We follow links, render JavaScript when needed, and hand back clean markdown plus the link graph. Ready for an LLM, a database, or whatever's next.
Crawl docs →Pin a cron schedule. We re-check the URL, hash the result, and webhook you only when content actually changed. HMAC-signed delivery.
Monitor docs →When a site fights back, we route through residential addresses and a real browser. Cloudflare, Akamai, PerimeterX, and captcha walls handled.
Anti-bot docs →Live web search results and backlink lookups in a single API call. Ground your AI agent's answer in fresh sources.
Search docs →Pipe raw HTML or PDF bytes in, get clean markdown out. Strips nav, footer, and chrome by default. The fastest path from a webpage to a vector store.
Transform docs →Run any URL through a real Chrome and get the post-JavaScript HTML back. The browser-as-a-service slice without managing your own fleet.
Render docs →Six more capabilities most APIs charge separately for.
Generate the new AI-discovery file for any site, ready for GPTBot, ClaudeBot, Perplexity.
POST /v1/llms-txt-build → AI-bot auditCheck which AI crawlers a site allows. Read the policy without parsing robots.txt yourself.
GET /v1/robots-policy → Orphan-page detectorFind pages reachable but un-linked. Screaming Frog feature as an API.
GET /v1/crawls/{id}/orphans → Per-domain scrapersPre-built configs for popular sites. Hit one path, get clean structured data.
POST /v1/cloud/fetch/{domain} → Backlink lookupWhat sites link to a URL. SEO-grade, fresh, single call.
POST /v1/cloud/links → Site diffCompare two crawls of the same site. Find what changed across runs.
GET /v1/crawls/{old}/diff/{new} →Every tier includes every endpoint. Tiers differ in volume, not features.
One API key. No card. 1,000 free pages every day.