Crw

安装配置

Docker Compose ships with `lightpanda` by default; `chrome` is opt-in:

README 摘要

fastCRW Self-hosted, Rust-native web crawler & scraper for AI agents The open-source alternative to Firecrawl. One static binary, ~50 MB RAM idle, Firecrawl-compatible REST API on **both `/v1/*` and `/v2/*`** (scrape, crawl, map, search, extract, plus v2 batch & parse) — a drop-in for the official Firecrawl SDKs — plus first-class MCP. Self-host free under AGPL-3.0, or hit our managed API at `api.fastcrw.com`. Reproducible 63.74% truth-recall on the public 1,000-URL dataset (`diagnose_3way.py`, 2026-05-08) — see [fastcrw.com/benchmarks](https://fastcrw.com/benchmarks). Built in Rust because every millisecond of agent latency compounds. Works with: [Claude Code](https://docs.fastcrw.com/mcp-clients/#claude-code) · [Cursor](https://docs.fastcrw.com/mcp-clients/#cursor) · [Windsurf](https://docs.fastcrw.com/mcp-clients/#windsurf) · [Cline](https://docs.fastcrw.com/mcp-clients/#cline) · [Copilot](https://docs.fastcrw.com/mcp-clients/#any-mcp-client) · [Continue.dev](https://docs.fastcrw.com/mcp-clients/#continue) · [Codex](https://docs.fastcrw.com/mcp-clients/#openai-codex-cli) · [Gemini CLI](https://docs.fastcrw.com/mcp-clients/#gemini-cli) --- ## Why fastCRW? - **Rust-native, single static binary** — no Redis, no Node.js, no Python venv, no headless-browser sidecar in the request path. One binary, one config file, one process. - **~50 MB RAM idle** — leaves headroom on a $5 VPS. Browser-render-first stacks (Firecrawl, Crawl4AI) carry a Chromium heap baseline measured in hundreds of MB before a single request lands. - **Firecrawl-compatible drop-in** — both the `/v1/*` and `/v2/*` surfaces (scrape, crawl, map, search, extract; plus v2-only batch & parse) with compatible request/response shapes. The v2 API is a drop-in for the official `firecrawl-py` v4 SDK (`FirecrawlApp(api_url="https://api.fastcrw.com")`) — swap the base URL and keep your code. - **Change tracking & monitoring** — diff a page against a prior snapshot (markdown git-diff, per-field JSON, or both) with an optional LLM "meaningful-change" judge. Stateless `changeTracking` primitive in the engine; scheduled monitors + signed-webhook/email alerts on the managed platform. See the [Monitoring docs](https://us.github.io/crw/monitoring). - **AGPL-3.0 open core + managed option** — self-host free, or point at `api.fastcrw.com` for managed proxy network, dashboard, and SLA without the AGPL obligations on your application code. --- ## Comparison Table Qualitative positioning vs. the three most-cited alternatives. Numerical claims trace to the inline sources noted; everything else is descriptive. | | **fastCRW** | Firecrawl | Crawl4AI | Spider | |---|---|---|---|---| | Language | Rust | Node.js + Playwright | Python + Playwright | Rust | | License | AGPL-3.0 (commercial avail.) | AGPL-3.0 (commercial avail.) | Apache-2.0 | Source-available / commercial ([spider.cloud](https://spider.cloud)) | | Self-host install size | Single static binary (~8 MB) | Multi-container (~500 MB+ image) | ~2 GB image (browser bundled) | Managed-first; self-host via crate | | Memory baseline (idle) | ~50 MB | Large (Chromium heap) | Large (Chromium heap) | Light (Rust) | | Firecrawl-compat API | Yes — **v1 + v2** (`/v1/*` and `/v2/*`) | Native | No | No | | MCP server | Built-in (`crw-mcp`) | Separate package | Community add-on | No first-party | | Hosted option | `api.fastcrw.com` (BYOK or managed) | firecrawl.dev | None official | spider.cloud (primary...