What it does
Firecrawl provides scraping, crawling, mapping, and extraction APIs. It converts supported web pages into Markdown or structured data, with capabilities that differ between the open-source and hosted versions.
Why it’s interesting
Content extraction removes much of the page boilerplate that distracts downstream retrieval. It is useful to study both the cleaning pipeline and the asynchronous work needed to crawl many pages.
Real-world use case
Index your product documentation into a vector database for a support assistant. Keep source URLs alongside each chunk so answers can link back to the original page.
What I can learn
- 01
Asynchronous crawling
coordinate queued tasks, concurrency, and retries.
- 02
Content extraction
preserve headings, code, and tables while removing boilerplate.
- 03
Browser automation
manage rendered pages and resource limits.
One thing to explore
API services and scraping pipeline
Follow the implementation and connect it to the learning points above.
Explore the source (opens in a new tab)10-Minute Challenge
Scrape one documentation page
Before you start: A hosted Firecrawl API key and available credits. This example uses the hosted API; self-hosted feature coverage differs.
- With a hosted Firecrawl account, put your API key in FIRECRAWL_API_KEY.
- Run the request below against a page you may scrape.
- Compare the returned Markdown with the page, checking headings and links. Self-hosting requires the separate setup guide.
curl -X POST https://api.firecrawl.dev/v2/scrape \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com","formats":["markdown"]}'