The Basics: Web Scraping with Cheerio and Node.js

By Nicholas St. Germain —

Using Cheerio for Web Scraping

When a site renders its content server-side, spinning up a full headless browser is overkill. Cheerio gives you jQuery-style HTML parsing in Node.js with none of the overhead, making it the fastest way to pull structured data out of static pages.

What is Cheerio?

Cheerio is a fast, flexible, and lean implementation of core jQuery designed for the server. It parses HTML or XML into a traversable DOM and exposes the same selector API developers already know from the browser. Because there is no rendering engine, scraping a page is little more than an HTTP request followed by a few CSS selectors.

Key Features

  1. Familiar Syntax: Uses the jQuery selector API for traversing and manipulating the DOM
  2. Blazing Fast: Parses thousands of pages per second with minimal CPU and memory cost
  3. No Browser Overhead: Skips JavaScript rendering when the data already lives in the HTML
  4. Composable: Pairs cleanly with any HTTP client like axios, got, or node-fetch
  5. Streaming Friendly: Works with htmlparser2 under the hood for large or chunked documents

Setting Up Cheerio

Installation

Create a project directory and initialize Node.js, then install Cheerio along with an HTTP client:

npm install cheerio axios

Fetching and Parsing a Page

const axios = require('axios');
const cheerio = require('cheerio');

async function scrape(url) {
    const { data: html } = await axios.get(url);
    const $ = cheerio.load(html);

    const title = $('title').text();
    const headings = $('h2')
        .map((_, el) => $(el).text().trim())
        .get();

    return { title, headings };
}

scrape('https://example.com').then(console.log);

Extracting Structured Data

Cheerio shines when you need rows of repeating elements - product listings, search results, articles:

async function scrapeArticles(url) {
    const { data: html } = await axios.get(url);
    const $ = cheerio.load(html);

    return $('article.post')
        .map((_, el) => ({
            title: $(el).find('h2 a').text().trim(),
            link: $(el).find('h2 a').attr('href'),
            author: $(el).find('.author').text().trim(),
            date: $(el).find('time').attr('datetime'),
        }))
        .get();
}

Advanced Features

Custom Headers and User Agents

Most servers reject bare-bones HTTP clients. Send a realistic header set on every request:

const headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9',
    'Accept-Language': 'en-US,en;q=0.9',
};

const { data: html } = await axios.get(url, { headers });

Concurrent Requests

Cheerio is synchronous once the HTML is loaded, so the bottleneck is the network. Run requests in parallel with a small worker pool:

const pLimit = require('p-limit');
const limit = pLimit(10);

const urls = [/* ...hundreds of URLs */];
const results = await Promise.all(
    urls.map((url) => limit(() => scrape(url)))
);

Handling Common Challenges

JavaScript-Rendered Content

If the data you want is injected by client-side JavaScript, Cheerio alone will not see it. In that case, render the page with Puppeteer or Playwright first, then hand the resulting HTML to Cheerio for parsing:

const puppeteer = require('puppeteer');

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle2' });

const html = await page.content();
const $ = cheerio.load(html);
await browser.close();

Pagination

For multi-page content, walk the "next" link until it disappears:

let url = baseUrl;
const items = [];

while (url) {
    const { data: html } = await axios.get(url);
    const $ = cheerio.load(html);

    $('.item').each((_, el) => items.push($(el).text().trim()));

    const next = $('a.next').attr('href');
    url = next ? new URL(next, url).toString() : null;
}

Rate Limiting and Retries

Add a backoff layer so a single 429 does not kill the run:

async function fetchWithRetry(url, attempts = 3) {
    for (let i = 0; i < attempts; i++) {
        try {
            return await axios.get(url, { timeout: 10_000 });
        } catch (err) {
            if (i === attempts - 1) throw err;
            await new Promise((r) => setTimeout(r, 2 ** i * 1000));
        }
    }
}

Integrating with Stat Proxies

Static-HTML scraping breaks down the moment a site starts blocking your datacenter IP. Routing axios through residential ISP proxies fixes that without changing a line of your parsing logic:

const axios = require('axios');
const { HttpsProxyAgent } = require('https-proxy-agent');
const cheerio = require('cheerio');

const proxyAgent = new HttpsProxyAgent(
    'http://user:pass@proxy.statproxies.com:3128'
);

async function scrape(url) {
    const { data: html } = await axios.get(url, {
        httpsAgent: proxyAgent,
        proxy: false,
    });

    const $ = cheerio.load(html);
    return $('h1').first().text().trim();
}

For larger jobs, rotate through a pool of session-pinned proxies so each worker keeps a sticky IP for the duration of its task while the overall fleet spreads requests across many addresses.

Conclusion

Cheerio is the right tool whenever the data you need is already in the HTML. Pair it with a sensible HTTP client, realistic headers, and a residential proxy pool, and you have a scraper that is faster, cheaper, and less detectable than a browser-driven alternative. Save the headless browser for the pages that genuinely need it.