Firecrawl: The Open-Source LLM Web Scraper

By Nicholas St. Germain —

Updated May 2026: Originally published in July 2024, this post has been refreshed to cover Firecrawl's v2 SDK, the /extract, /map, /search, and Agent endpoints, action-based browsing, and MCP support - plus modernized Node.js code samples.

What is Firecrawl?

Firecrawl is not your average web scraper. It's a comprehensive API service that takes web crawling to the next level. With just a URL as input, Firecrawl crawls entire websites, converting them into clean markdown or structured data. What sets it apart? It doesn't require a sitemap, making it incredibly versatile for various web structures.

Key features of Firecrawl include:

  • Comprehensive crawling of all accessible subpages
  • Clean data output in markdown, HTML, JSON, or summary form
  • No sitemap required - /map discovers URLs on its own
  • Natural-language extraction via the /extract and Agent endpoints
  • Action-based browsing (clicks, scrolls, waits) before extraction
  • First-party MCP server for AI agents and Claude/Cursor integrations
  • API-first approach for easy integration
  • Open-source (AGPL-3.0) and self-hostable via Docker

The Minds Behind Firecrawl: Mendable

Firecrawl began as an internal tool at Mendable, a Y Combinator-backed AI startup focused on making LLMs useful for customer experience and sales. As Mendable worked on building AI systems that could understand and interact with vast amounts of web data, they realized the need for a more efficient, flexible, and powerful web scraping tool. Firecrawl was born out of this necessity, designed to bridge the gap between raw web content and LLM-ready data.

What started as a side project quickly outgrew its origins. By 2025 the team had effectively rebranded around the product - Firecrawl is now the headline name, with its own dashboard, paid tiers, and a rapidly growing GitHub presence (well over 100k stars at the time of this update).

By open-sourcing Firecrawl, the team isn't only solving its own challenges but also contributing to the broader tech community. They recognize that many developers, data scientists, and AI researchers face similar hurdles when it comes to collecting and structuring web data for AI applications. Firecrawl is their way of democratizing access to high-quality, structured web data, which is crucial for training and fine-tuning LLMs.

The decision to keep Firecrawl open-source aligns with the team's philosophy of fostering innovation through collaboration. By allowing the community to use, modify, and improve Firecrawl, they're accelerating the development of AI technologies and empowering developers worldwide to build more sophisticated AI-driven applications.

What's New Since 2024

A lot has shipped since the first release. If you wrote code against the v0 API, expect to make some changes:

  • v1 API (late 2024) replaced pageOptions and extractorOptions with a single formats array. Structured extraction moved under formats: ['json'] (originally 'extract') with a top-level jsonOptions/extract object holding your schema and prompt.
  • /extract endpoint - point it at one URL, many URLs, or a wildcard like https://example.com/* and let Firecrawl figure out which pages to visit. You describe the data in a prompt and an optional Zod/Pydantic schema.
  • /map endpoint - returns every URL it can find on a domain. Great for discovering a site's surface area before deciding what to crawl.
  • /search endpoint - search the web and return scraped content for the top results in a single call.
  • Actions - click, type, scroll, wait, screenshot, and press keys before extraction. This unlocks scraping behind logins, search forms, and "load more" buttons without spinning up your own Playwright stack.
  • Agent endpoint (v2) - describe what you need in plain English ("find every pricing tier and its features on these competitor sites") and an LLM-powered agent navigates, retrieves, and returns structured data, with model selection (Spark Fast/Mini/Pro) for cost-vs-quality tradeoffs.
  • PDF and Excel parsing - Firecrawl now handles .pdf and .xlsx files with fast/auto/OCR modes and page limits.
  • MCP server - first-party Model Context Protocol support means Claude, Cursor, and other MCP-aware tools can call Firecrawl directly.
  • Caching with minAge - re-use a recent scrape if it's fresh enough, instead of paying to re-fetch.
  • CLI - firecrawl scrape | search | crawl | map straight from your terminal.

The Node SDK package is still @mendable/firecrawl-js, but the default export is now Firecrawl (the older FirecrawlApp alias still works for backward compatibility).

Firecrawl in Action: Solving Real-World Problems

Firecrawl's versatility makes it an invaluable tool for a wide range of web scraping and data extraction scenarios. Let's explore some common use cases where Firecrawl shines:

Content Aggregation for News Apps: Firecrawl can efficiently crawl multiple news websites, extracting articles and their metadata. This structured data can then be used to power news aggregation apps or train AI models for content recommendation.

E-commerce Price Monitoring: For businesses needing to track competitor pricing, Firecrawl can regularly scrape e-commerce sites, extracting product information and prices in a structured format for easy analysis.

Research Data Collection: Academic researchers can use Firecrawl to gather large datasets from web sources, such as social media posts or forum discussions, for sentiment analysis or trend identification.

SEO Analysis: Digital marketers can leverage Firecrawl to extract metadata, headings, and content structure from websites, facilitating comprehensive SEO audits and competitor analysis.

Training Data for Chatbots: By crawling FAQs and knowledge bases, Firecrawl can compile extensive datasets for training customer service chatbots, ensuring they have up-to-date information.

In each of these scenarios, Firecrawl solves critical problems:

  • It eliminates the need for custom scraping scripts for each website.
  • It handles pagination and navigation automatically.
  • It provides clean, structured data ready for analysis or AI model training.
  • Its API-first approach allows for easy integration into existing workflows and applications.

Deep Dive: Scraping Airbnb Listings with Firecrawl

Now, let's dig into a more complex example: using Firecrawl to scrape Airbnb listings. This use case is particularly interesting because it involves dealing with dynamically loaded content, pagination, and extracting specific structured data.

Here's a step-by-step breakdown of how we can use Firecrawl to scrape Airbnb listings for San Francisco. We'll use the v2 SDK, which collapses what used to be a two-step scrapeUrl dance into a single extract call against a wildcard URL pattern:

import Firecrawl from '@mendable/firecrawl-js'
import 'dotenv/config'
import { z } from 'zod'

async function scrapeAirbnb() {
  const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY })

  const listingSchema = z.object({
    listings: z.array(
      z.object({
        title: z.string(),
        price_per_night: z.number(),
        location: z.string(),
        rating: z.number().optional(),
        reviews: z.number().optional(),
      })
    ).describe('Airbnb listings in San Francisco'),
  })

  // Firecrawl follows pagination on its own when you point /extract
  // at a wildcard URL and describe what you want.
  const result = await firecrawl.extract({
    urls: ['https://www.airbnb.com/s/San-Francisco--CA--United-States/homes*'],
    prompt:
      'Collect every visible Airbnb listing for San Francisco, including ' +
      'title, nightly price in USD, location, rating, and review count. ' +
      'Follow pagination until there are no more pages.',
    schema: listingSchema,
  })

  return result.data.listings
}

scrapeAirbnb().then((listings) => {
  console.log(`Scraped ${listings.length} Airbnb listings`)
  console.log(listings[0])
}).catch((error) => {
  console.error('An error occurred:', error.message)
})

If you want finer control - for example, only scraping a single page and shaping the response yourself - scrape with a JSON format still works in v2:

const page = await firecrawl.scrape(
  'https://www.airbnb.com/s/San-Francisco--CA--United-States/homes',
  {
    formats: [
      'markdown',
      {
        type: 'json',
        schema: listingSchema,
        prompt: 'Extract listings from the page.',
      },
    ],
    onlyMainContent: false,
    actions: [
      { type: 'wait', milliseconds: 1500 },
      { type: 'scroll', direction: 'down' },
    ],
  }
)

const listings = page.json.listings

Let's break down what's happening:

  1. Initialization: We set up the Firecrawl client with our API key.
  2. Schema Definition: We use Zod to declare the shape of the data we want. Firecrawl validates and coerces each result against it.
  3. Pagination Handling: With /extract we don't have to scrape the pagination bar ourselves - the wildcard pattern lets Firecrawl follow it. The older scrape flow gets pre-extraction actions for scrolling and waiting on lazy-loaded content.
  4. Single Round-Trip: What used to be three separate scrapeUrl calls in the v0 example is now one extract call.

This example showcases several key strengths of Firecrawl:

  • Flexibility: It can handle complex, multi-step scraping processes.
  • Structured Data Extraction: Zod schemas ensure we get precisely the data we need, with the right types.
  • Browser Actions: Click, scroll, wait, and type before extraction - no Playwright wrapper required.
  • Performance: Firecrawl parallelizes the underlying page fetches for you.

By using Firecrawl for this task, we've eliminated the need to deal with browser automation, AJAX requests, and complex CSS selectors. Instead, we can focus on defining what data we want and how to process it, making our scraping tasks significantly more manageable and maintainable.

Current Limitations and Challenges

Firecrawl ships with a built-in stealth proxy mode and automatic rotation, which handles a lot of the easy cases out of the box. But once you push beyond demos and into production, the same problems that affect every scraper start to show up:

  • Stealth-Mode Quotas: The hosted stealth proxy is metered separately and gets expensive fast at scale. You'll want your own pool if you're crawling millions of URLs.
  • Geo-Restricted Content: Some sites serve completely different content per country, currency, or even city. Hosted proxy pools rarely give you fine-grained control over exit location.
  • CAPTCHA and Hard Anti-Bot: PerimeterX, DataDome, Akamai Bot Manager, and Cloudflare Turnstile have all gotten dramatically harder since 2024. Sophisticated targets often need residential IPs with consistent session affinity to stay logged in.
  • Sticky Sessions: Many flows (carts, search refinements, dashboards) break if the IP changes mid-session. You need a proxy that lets you pin a session.
  • Cost Predictability: Hosted "anti-bot" proxy add-ons are usually billed per-page; a dedicated proxy provider is billed per-GB and is far cheaper for repeated crawls of the same domains.
  • Detection Avoidance: Sites are increasingly fingerprinting datacenter IP ranges. Residential and ISP IPs blend in far better than any shared cloud pool.

Enter Stat Proxies: Elevating Your Firecrawl Experience

This is where Stat Proxies comes into play, offering a powerful solution to complement and enhance Firecrawl's capabilities. As industry leaders in ethical residential and ISP proxies, Stat Proxies provides a robust infrastructure that addresses the gaps left by hosted scraping APIs.

How Stat Proxies Enhances Firecrawl:

  • IP Rotation and Anonymity: Stat Proxies' large pool of residential IPs allows for seamless IP rotation, significantly reducing the risk of rate limiting and IP blocks.
  • Global Coverage: With proxies from diverse geographic locations, you can access geo-restricted content and ensure comprehensive data collection.
  • Enhanced Scalability: Distribute your Firecrawl requests across multiple IPs, allowing for higher volume scraping without overloading a single IP.
  • Improved Success Rates: Residential proxies mimic real user behavior more closely, helping to bypass many anti-bot measures and reducing CAPTCHA occurrences.
  • Predictable Pricing: Per-GB billing instead of per-page fees, which makes large crawls dramatically cheaper than turning on a hosted stealth tier.
  • Ethical Compliance: Stat Proxies ensures all IPs are ethically sourced, maintaining legal and moral standards in your data collection efforts.

If you're self-hosting Firecrawl, point the underlying browser at Stat Proxies via the standard PROXY_SERVER, PROXY_USERNAME, and PROXY_PASSWORD environment variables on the Docker container - every request the worker makes will then egress through your proxy pool.

If you're using the hosted Firecrawl API, the cleanest pattern is to put your own proxy in front of a self-hosted Firecrawl worker, or to use Firecrawl just for the markdown/JSON conversion layer while doing the actual fetch from your side. Here's the self-hosted setup in action with the v2 SDK:

import Firecrawl from '@mendable/firecrawl-js'
import 'dotenv/config'
import { z } from 'zod'

async function scrapeAirbnbWithProxy() {
  // Point the SDK at your self-hosted Firecrawl instance. The worker
  // container is configured with the Stat Proxies credentials below
  // via PROXY_SERVER / PROXY_USERNAME / PROXY_PASSWORD env vars:
  //
  //   PROXY_SERVER=http://proxy.statproxies.com:3128
  //   PROXY_USERNAME=stat_user
  //   PROXY_PASSWORD=super_secret_password
  const firecrawl = new Firecrawl({
    apiKey: process.env.FIRECRAWL_API_KEY,
    apiUrl: 'http://localhost:3002', // self-hosted Firecrawl
  })

  const listingSchema = z.object({
    listings: z.array(
      z.object({
        title: z.string(),
        price_per_night: z.number(),
        location: z.string(),
        rating: z.number().optional(),
        reviews: z.number().optional(),
      })
    ).describe('Airbnb listings in San Francisco'),
  })

  const result = await firecrawl.extract({
    urls: ['https://www.airbnb.com/s/San-Francisco--CA--United-States/homes*'],
    prompt:
      'Collect every visible Airbnb listing for San Francisco, including ' +
      'title, nightly price in USD, location, rating, and review count.',
    schema: listingSchema,
  })

  return result.data.listings
}

scrapeAirbnbWithProxy().then((listings) => {
  console.log(`Successfully scraped ${listings.length} Airbnb listings using Stat Proxies!`)
  console.log(listings[0])
}).catch((error) => {
  console.error('An error occurred:', error.message)
})

By routing the underlying browser through Stat Proxies, every page Firecrawl fetches - whether that's during /scrape, /crawl, /extract, or an Agent run - exits from a residential IP that target sites have no reason to flag. You get Firecrawl's markdown and structured-data pipeline on top of an infrastructure layer that's actually built for high-volume scraping.

Conclusion: The Future of Web Scraping is Here

The combination of Firecrawl's powerful scraping capabilities and Stat Proxies' robust proxy infrastructure represents a new frontier in web data collection and preparation for LLMs. This powerful duo offers:

  • Unparalleled Data Access: Scrape even the most challenging websites with ease.
  • Scalability: Handle large-scale data collection projects effortlessly.
  • Ethical Compliance: Ensure your data collection methods adhere to legal and ethical standards.
  • LLM-Ready Data: Obtain clean, structured data perfect for training and fine-tuning language models.
  • Flexibility and Customization: Adapt to various scraping scenarios and requirements.

As the demand for high-quality, diverse datasets for AI and machine learning continues to grow, the importance of sophisticated, ethical web scraping tools cannot be overstated. Firecrawl, enhanced by Stat Proxies, stands at the forefront of this revolution, empowering developers, data scientists, and businesses to unlock the full potential of web data.

Ready to take your web scraping and data collection to the next level? Start your journey with Firecrawl today, and supercharge your efforts with Stat Proxies' ethical residential proxies.

Visit Stat Proxies to learn more about how our ethical residential proxies can elevate your web scraping game.