Tool profile
Crawl4AI: cost, fit and limits
Crawl4AI is an open-source Python web crawler, maintained under the GitHub user unclecode and licensed Apache 2.0, that turns pages into Markdown and structured JSON for LLM and RAG pipelines, with a separate hosted service called Crawl4AI Cloud.
What to know first
Crawl4AI has two ways in: the free self-hosted crawler under Apache License 2.0, where you pay for your own infrastructure, and Crawl4AI Cloud, with 1,000 free credits a month and paid plans from $10. The repository names no company or headquarters, so maker and country are not stated. It is built for developers, with a CLI, Docker image and REST routes.
Crawl4AI at a glance
| Maker | Maintained by the unclecode GitHub project, Not stated on the repository page |
|---|---|
| Free plan | Yes |
| Entry price | Quote only |
| API access | No |
| Category | Best AI Web Scraping Tools in 2026, rank 2 of 12 |
| Website | github.com/unclecode/crawl4ai |
Who Crawl4AI is best for
Developers building RAG pipelines
Fit Markdown filters and citation-style link lists produce cleaned page text for retrieval, which is the use the project is designed around.
Teams that want to self-host
The Apache-2.0 crawler ships with a Docker image, a crwl CLI, a dashboard and a playground, so a team can run it on its own servers and keep crawled content in-house.
Agent builders using coding assistants
The README documents adding Crawl4AI to Claude Code over MCP and lists Cursor, Codex and OpenCode, so assistants can call the crawler as a tool.
Crawl4AI plans and pricing
| Plan | Price | What it includes |
|---|---|---|
| Open source (self-hosted) | Free, Apache License 2.0 | The crawler, the unclecode/crawl4ai Docker image, the crwl CLI, a dashboard and playground; you cover your own infrastructure. |
| Crawl4AI Cloud | 1,000 free credits every month; plans from $10 | Hosted scrape, extract and search API on pay as you go, with live prices linked from the page and no credit card for the free credits. |
Checked October 2026.
What Crawl4AI does
- Markdown with Fit filters
- Output includes Fit Markdown through PruningContentFilterLXML, BM25ContentFilter or LLMContentFilter, plus citation-style link lists, so boilerplate is trimmed before the cleaned page text reaches any model you use.
- Extraction with or without an LLM
- Schemas in CSS, XPath or regex extract fields without any model call, and LLM extraction works with any provider LiteLLM supports, with OpenAI and Ollama shown as examples.
- Browser and crawl control
- Chromium, Firefox and WebKit are supported with persistent profiles, proxies with rotation, stealth mode and remote browsers over CDP. Deep crawls run BFS, DFS or best-first, with an adaptive crawler.
- Cloud endpoints
- Crawl4AI Cloud exposes /scrape, /extract, /search and batch routes. The page says up to 50 URLs per streamed call or 10,000 in a background job; /serp is beta and /answer experimental.
How Crawl4AI handles your data
For the self-hosted crawler, pages are fetched and processed on your own infrastructure, so what is stored depends on how you deploy it; the server can be secured with a CRAWL4AI_API_TOKEN. The repository page does not state retention, hosting regions, certifications or whether Crawl4AI Cloud uses customer content to train models, so the vendor's policy was not checked for those points. Any LLM extraction sends page text to the provider you configure.
Where Crawl4AI falls short in practice
- No named company behind it. The repository lists a maintainer handle but no company, headquarters or contract entity, which matters to procurement teams that need a legal counterparty for the open-source crawler.
- Cloud prices live elsewhere. Plans start from $10, but per-credit prices are linked from the page rather than shown, so the monthly cost of a given volume was not verified.
- Beta and experimental routes. The /serp endpoint is labeled beta and /answer experimental, so those cloud routes may change. Self-hosting also leaves servers, proxies and upgrades to your own team.
Works with
Alternatives to Crawl4AI
Crawl4AI: common questions
Is Crawl4AI free to use?
The crawler is open source under Apache License 2.0 and free to run yourself, but you pay for your own servers and any LLM provider. Crawl4AI Cloud gives 1,000 free credits every month with no credit card, and paid plans start from $10.
Does Crawl4AI work without an LLM?
Yes. CSS, XPath and regex schemas extract structured data with no model involved, and Markdown output with the BM25 or pruning filters also needs none. LLM extraction is optional and goes through any provider LiteLLM supports, such as OpenAI or Ollama.
Can I connect Crawl4AI to Claude Code?
The README shows a claude mcp add command for Claude Code, and also lists Cursor, Codex and OpenCode. That lets an assistant call the crawler as a tool. Follow the exact command on the repository page, since the setup depends on how you run the server.
How big can a Crawl4AI Cloud batch be?
The page says up to 50 URLs in one streamed call, or up to 10,000 URLs as a background job through /scrape/jobs. It does not state rate limits or concurrency, so check the live documentation before planning a large crawl.
Who makes Crawl4AI?
The repository lists the GitHub user unclecode as author and names no company. No headquarters country is stated on the pages read. Treat it as a community-maintained project if you need a contract party, and ask about Crawl4AI Cloud terms separately.