How to Scrape Walmart with Python in 2026: A Practical Guide

By Nicholas St. Germain — May 4, 2026

Walmart is the second-most-scraped ecommerce site on the open web, behind only Amazon. Repricing engines, brand monitoring tools, and AI shopping agents all need fresh Walmart data - and almost all of them get blocked the first time they try to fetch it from a cloud server.

This guide walks through scraping Walmart product pages with Python, parsing the JSON blob the site embeds in the HTML, and routing requests through residential ISP proxies so the run actually finishes.

Why Walmart Is Harder Than It Looks

Walmart product pages render most of their data server-side, which sounds like a gift to anyone with a static HTML parser. The catch is the perimeter: Walmart fronts every product URL with PerimeterX (now HUMAN), and the bot detection fires on three independent signals before the page even reaches your parser.

ASN classification. Requests from AWS, GCP, DigitalOcean, and other hosting ASNs get a JavaScript challenge or a 403 immediately. There is no warm-up period.
TLS fingerprinting. requests and httpx ship a JA3 fingerprint that Walmart's perimeter has on a known-bad list. Even from a clean residential IP, a bare requests.get() often gets challenged.
Behavioral pacing. A single IP hammering 100 product URLs in a minute trips a soft block - the page returns 200 OK, but the embedded __NEXT_DATA__ blob is missing the price and availabilityStatus fields.

Two of those three problems are solved by routing through residential or ISP proxies. The third (TLS) is solved by using a TLS-impersonating client like curl_cffi.

Setting Up the Project

mkdir walmart-scraper && cd walmart-scraper
python -m venv venv && source venv/bin/activate
pip install curl_cffi beautifulsoup4 lxml

curl_cffi is a drop-in replacement for requests that impersonates real browser TLS fingerprints. It is the single biggest reliability upgrade you can make to a Python scraper in 2026.

Fetching a Product Page

Walmart product URLs follow the pattern https://www.walmart.com/ip/<slug>/<itemId>. Pick any item and try a bare fetch first to see what you're up against:

from curl_cffi import requests

url = "https://www.walmart.com/ip/Apple-AirPods-Pro-2nd-Generation/720123456"

response = requests.get(url, impersonate="chrome120")
print(response.status_code, len(response.text))

From a residential IP this will usually return a 200 with the full HTML. From a datacenter IP you'll get a 403 or a challenge page that's a few KB instead of the expected 800 KB+.

Parsing the Embedded JSON

Walmart serves a Next.js app, which means every product page contains a <script id="__NEXT_DATA__"> tag with the entire product object as JSON. Parsing this blob is dramatically more reliable than scraping CSS selectors that change every quarter.

import json
from bs4 import BeautifulSoup

def extract_product(html: str) -> dict:
    soup = BeautifulSoup(html, "lxml")
    script = soup.find("script", id="__NEXT_DATA__")
    if not script:
        raise ValueError("Product data not found - likely soft-blocked")

    data = json.loads(script.string)
    product = data["props"]["pageProps"]["initialData"]["data"]["product"]

    return {
        "id": product["id"],
        "name": product["name"],
        "brand": product.get("brand"),
        "price": product["priceInfo"]["currentPrice"]["price"],
        "currency": product["priceInfo"]["currentPrice"]["currencyUnit"],
        "in_stock": product["availabilityStatus"] == "IN_STOCK",
        "rating": product.get("averageRating"),
        "review_count": product.get("numberOfReviews"),
        "seller": product.get("sellerName"),
    }

The __NEXT_DATA__ schema occasionally shifts when Walmart ships a redesign, but the top-level keys (product, priceInfo, availabilityStatus) have been stable for years. Wrap each .get() in a guard if you scrape across categories - grocery items, marketplace listings, and Walmart+ exclusives each have slight variations.

Routing Through ISP Proxies

A single residential IP will get you a few hundred requests before pacing kicks in. To run at any kind of scale, rotate through a pool:

from curl_cffi import requests

PROXY = "http://user-session1:pass@proxy.statproxies.com:3128"

def fetch(url: str) -> str:
    response = requests.get(
        url,
        impersonate="chrome120",
        proxies={"http": PROXY, "https": PROXY},
        timeout=20,
    )
    response.raise_for_status()
    return response.text

The session1 segment in the username pins the request to a sticky IP for the duration of that session. Rotate the session ID per worker (session1, session2, ...) so each thread keeps its own consistent identity instead of teleporting between IPs mid-scrape - Walmart's behavioral layer flags rapid IP changes inside a single session.

For why session-pinned residential exit nodes outperform rotating datacenter pools on retail sites, see our complete guide to ISP proxies.

Scaling with a Worker Pool

Walmart tolerates roughly 1 req/sec per IP before pacing engages. With 50 session-pinned IPs you can comfortably sustain 30–40 req/sec across the fleet:

import concurrent.futures as cf

def scrape_one(item_id: int) -> dict:
    url = f"https://www.walmart.com/ip/-/{item_id}"
    html = fetch(url)
    return extract_product(html)

item_ids = [720123456, 720123457, 720123458, ...]

with cf.ThreadPoolExecutor(max_workers=50) as pool:
    results = list(pool.map(scrape_one, item_ids))

For the multithreading patterns that keep this stable under load (per-thread sessions, retry logic, queue-based dispatch), see our guide on using proxies with multithreading in Python.

Detecting Soft Blocks

Walmart almost never returns a 4xx when it blocks you. Instead, the page loads with a stripped __NEXT_DATA__ payload - same status code, same length-ish, but the product key is missing. Build the check into your extractor:

def is_soft_blocked(data: dict) -> bool:
    try:
        return "product" not in data["props"]["pageProps"]["initialData"]["data"]
    except KeyError:
        return True

When the check trips, mark the session as burned, swap to a fresh session ID, and retry. Most of the data quality issues people blame on parser bugs are actually undetected soft blocks.

Handling ZIP-Code-Specific Pricing

Walmart prices, availability, and shipping windows vary by store. The site stores the active store ID in a cookie (locationData) seeded from a ZIP code. To pin your scrape to a specific market, set the cookie before the request:

cookies = {
    "locationData": json.dumps({
        "postalCode": "10001",
        "stateOrProvinceCode": "NY",
        "city": "New York",
    })
}

response = requests.get(url, impersonate="chrome120", cookies=cookies, ...)

Pair this with proxies that exit in the same region - a New York ZIP cookie on a Texas residential IP is exactly the kind of inconsistency the perimeter flags as automation.

Conclusion

Walmart scraping breaks down into four problems: TLS fingerprint, IP reputation, session consistency, and soft-block detection. Solve them with a TLS-impersonating client, session-pinned residential proxies, sentinel-string monitoring, and ZIP-code-aligned exits, and you have a scraper that runs cleanly for weeks instead of minutes. Skip any one of them and you'll be debugging "missing price" rows for the rest of the quarter.