Crawlee: Build Reliable Web Scrapers That Don't Get Blocked
By Nicholas St. Germain —
What is Crawlee?
Crawlee is an open-source web scraping and browser automation library built by Apify. Available for both JavaScript/TypeScript and Python, Crawlee takes a fundamentally different approach to web scraping: instead of just giving you low-level tools, it handles the hard parts - anti-blocking, proxy rotation, request queuing, and browser management - so you can focus on what data to extract rather than how to avoid getting banned.
What makes Crawlee unique is its "human-like crawling" philosophy. Even with default settings, your scrapers appear as normal browser traffic, flying under the radar of modern bot protection systems. This is achieved through intelligent request timing, browser fingerprint management, and automatic session rotation.
Key features of Crawlee include:
- Built-in anti-blocking with human-like crawling behavior
- Unified interface for HTTP and headless browser scraping
- Automatic proxy rotation and session management
- Request queue with configurable concurrency and retry logic
- Multiple crawler types: HTTP-only, Cheerio/BeautifulSoup, Playwright, Puppeteer
- Automatic data storage and export
- Available in both JavaScript/TypeScript and Python
- Open source under the Apache 2.0 license
Who Built Crawlee?
Crawlee is built and maintained by Apify, a well-established web scraping platform company. The JavaScript version launched in summer 2022 as a successor to Apify's earlier SDK, and the Python version followed in March 2025.
Apify's decade of experience running web scrapers at massive scale directly informs Crawlee's design. Every anti-blocking technique, retry strategy, and concurrency pattern in Crawlee comes from real-world production experience handling billions of web requests. By open-sourcing this knowledge, Apify has given the community access to enterprise-grade scraping infrastructure.
Why Crawlee Stands Out
Most scraping libraries give you the building blocks - HTTP clients, HTML parsers, browser drivers - and leave you to handle the complex orchestration yourself. Crawlee takes the opposite approach:
Automatic Anti-Blocking: Crawlee automatically rotates user agents, manages browser fingerprints, and varies request timing to mimic human browsing patterns. No configuration required.
Unified Crawler Interface: Start with a lightweight HTTP crawler for static pages, then seamlessly upgrade to a full browser crawler for JavaScript-heavy sites - same API, same code structure.
Smart Request Management: Built-in request queuing handles deduplication, retry logic, and configurable concurrency. Failed requests are automatically retried with exponential backoff.
Automatic Data Storage: Crawlee manages data persistence for you, storing results in datasets that can be exported to JSON, CSV, or pushed to external storage.
Getting Started with Crawlee (JavaScript)
Install Crawlee and set up a basic crawler:
npx crawlee create my-crawler
cd my-crawler
npm start
Or install manually:
npm install crawlee
Here's a basic Cheerio crawler for scraping static pages:
import { CheerioCrawler, Dataset } from 'crawlee'
const crawler = new CheerioCrawler({
maxRequestsPerCrawl: 50,
async requestHandler({ request, $, enqueueLinks }) {
const title = $('title').text()
const headings = $('h1, h2, h3')
.map((_, el) => $(el).text().trim())
.get()
// Store the extracted data
await Dataset.pushData({
url: request.url,
title,
headings,
})
// Automatically find and enqueue links on the page
await enqueueLinks({
globs: ['https://example.com/**'],
})
},
})
await crawler.run(['https://example.com'])
// Export the dataset
const dataset = await Dataset.open()
await dataset.exportToCSV('results')
Browser-Based Scraping with Playwright
For JavaScript-heavy sites, switch to a Playwright crawler with the same code structure:
import { PlaywrightCrawler, Dataset } from 'crawlee'
const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 50,
headless: true,
async requestHandler({ request, page, enqueueLinks }) {
// Wait for dynamic content to load
await page.waitForSelector('.product-card')
// Extract product data from the rendered page
const products = await page.$$eval('.product-card', (cards) =>
cards.map((card) => ({
name: card.querySelector('.product-title')?.textContent?.trim(),
price: card.querySelector('.price')?.textContent?.trim(),
rating: card.querySelector('.rating')?.textContent?.trim(),
}))
)
await Dataset.pushData({
url: request.url,
products,
})
await enqueueLinks({
selector: '.pagination a',
})
},
})
await crawler.run(['https://example.com/products'])
Getting Started with Crawlee (Python)
pip install crawlee[beautifulsoup,playwright]
playwright install
import asyncio
from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
async def main():
crawler = BeautifulSoupCrawler(
max_requests_per_crawl=50,
)
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext):
soup = context.soup
title = soup.title.string if soup.title else ''
headings = [h.get_text(strip=True) for h in soup.find_all(['h1', 'h2', 'h3'])]
await context.push_data({
'url': context.request.url,
'title': title,
'headings': headings,
})
await context.enqueue_links(
include=['https://example.com/**'],
)
await crawler.run(['https://example.com'])
asyncio.run(main())
Real-World Example: Scraping Product Listings
Here's a complete example that scrapes product data across multiple pages with automatic pagination handling:
import { PlaywrightCrawler, Dataset } from 'crawlee'
const crawler = new PlaywrightCrawler({
maxRequestsPerCrawl: 200,
maxConcurrency: 5,
navigationTimeoutSecs: 60,
async requestHandler({ request, page, enqueueLinks, log }) {
log.info(`Scraping ${request.url}`)
// Wait for product listings to render
await page.waitForSelector('[data-testid="listing"]', { timeout: 10000 })
// Extract all product listings on the page
const listings = await page.$$eval('[data-testid="listing"]', (items) =>
items.map((item) => ({
title: item.querySelector('h2')?.textContent?.trim() || '',
price: item.querySelector('[data-testid="price"]')?.textContent?.trim() || '',
location: item.querySelector('[data-testid="location"]')?.textContent?.trim() || '',
link: item.querySelector('a')?.href || '',
}))
)
await Dataset.pushData(
listings.map((listing) => ({
...listing,
sourceUrl: request.url,
scrapedAt: new Date().toISOString(),
}))
)
// Follow pagination links
await enqueueLinks({
selector: 'a[aria-label="Next page"]',
})
},
async failedRequestHandler({ request, log }) {
log.error(`Request failed after retries: ${request.url}`)
},
})
await crawler.run(['https://example.com/listings'])
const dataset = await Dataset.open()
await dataset.exportToJSON('listings')
console.log('Scraping complete!')
Current Limitations
Even with Crawlee's built-in anti-blocking, large-scale scraping operations face real challenges:
- IP-Based Blocking: Crawlee's anti-blocking handles browser fingerprinting and behavior, but can't solve IP-level rate limiting and blocking on its own.
- Residential IP Requirements: Some websites only allow traffic from residential IP ranges, blocking all datacenter traffic regardless of how human-like the behavior appears.
- Geographic Restrictions: Accessing region-locked content requires proxies in the target geography.
- Scale Limitations: Running hundreds of concurrent browser instances from a single server creates detectable traffic patterns.
- Cost of Failures: Each blocked request wastes compute resources, especially when running headless browsers.
Completing the Stack with Stat Proxies
Crawlee handles the application-level anti-blocking (fingerprints, behavior, timing), while Stat Proxies handles the network-level anti-blocking (IP rotation, residential traffic, geographic distribution). Together, they form a complete anti-blocking solution.
How Stat Proxies Enhances Crawlee:
- Residential IP Pool: Ensure all traffic originates from real residential IPs, bypassing datacenter IP blocks.
- Automatic IP Rotation: Fresh IPs for each session, preventing pattern-based blocking.
- Geographic Targeting: Route requests through specific locations to access region-locked content.
- Unlimited Scale: Distribute browser sessions across thousands of IPs for truly parallel scraping.
- Ethical Sourcing: All IPs are responsibly sourced, maintaining compliance with ethical standards.
Here's how to configure Crawlee with Stat Proxies:
import { PlaywrightCrawler, Dataset, ProxyConfiguration } from 'crawlee'
// Configure Stat Proxies
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://stat_user:super_secret_password@proxy.statproxies.com:3128',
],
})
const crawler = new PlaywrightCrawler({
proxyConfiguration,
maxRequestsPerCrawl: 200,
maxConcurrency: 10,
async requestHandler({ request, page, enqueueLinks }) {
await page.waitForSelector('.product-card')
const products = await page.$$eval('.product-card', (cards) =>
cards.map((card) => ({
name: card.querySelector('.title')?.textContent?.trim(),
price: card.querySelector('.price')?.textContent?.trim(),
rating: card.querySelector('.rating')?.textContent?.trim(),
}))
)
await Dataset.pushData(products)
await enqueueLinks({
selector: '.pagination a',
})
},
})
await crawler.run(['https://example.com/products'])
console.log('Scraping complete with Stat Proxies!')
And in Python:
import asyncio
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration
async def main():
proxy_config = ProxyConfiguration(
proxy_urls=[
'http://stat_user:super_secret_password@proxy.statproxies.com:3128',
],
)
crawler = PlaywrightCrawler(
proxy_configuration=proxy_config,
max_requests_per_crawl=200,
max_concurrency=10,
)
@crawler.router.default_handler
async def handler(context: PlaywrightCrawlingContext):
page = context.page
await page.wait_for_selector('.product-card')
products = await page.eval_on_selector_all(
'.product-card',
'''cards => cards.map(card => ({
name: card.querySelector(".title")?.textContent?.trim(),
price: card.querySelector(".price")?.textContent?.trim(),
}))'''
)
await context.push_data(products)
await context.enqueue_links(selector='.pagination a')
await crawler.run(['https://example.com/products'])
asyncio.run(main())
With Crawlee handling the intelligent crawling logic and Stat Proxies providing the residential proxy infrastructure, you have a scraping stack that's both smart and resilient.
Conclusion
Crawlee represents the most complete open-source scraping framework available today. Its built-in anti-blocking, unified crawler interface, and automatic request management eliminate most of the boilerplate and complexity that makes web scraping painful.
When combined with Stat Proxies' residential proxy network, Crawlee becomes an enterprise-grade scraping solution that delivers:
- Near-zero block rates through combined application and network-level anti-blocking
- Effortless scaling across thousands of concurrent requests
- Global data access through geographically distributed residential IPs
- Lower costs by reducing wasted compute on blocked requests
- Ethical compliance with responsibly sourced proxy infrastructure
Whether you're building in JavaScript or Python, Crawlee and Stat Proxies give you everything you need to scrape the web reliably at any scale. Get started with Crawlee, and add Stat Proxies for the proxy infrastructure to make it production-ready.