Skip to content

Does Cloudflare Block Crawlers That Ignore 429s?

By . Published

Cloudflare publishes no rule that blocks crawlers for ignoring 429s, but it now scores them. On October 16, 2026, it said it grades each Verified bot on whether it slows or pauses after an HTTP 429 and follows robots.txt, and plans to let customers set a trust threshold. During a 5-minute limit, a crawler holding 2 requests per second sends 600; one that honors Retry-After sends 1. Static ISP proxies ($2.50 per IP per month at Stat) give a polite crawler stable addresses but can't fix one that ignores 429s.

That last sentence is the whole story for anyone who runs crawlers or AI agents at volume. For years, the 429 was a speed bump you could drive around. Cloudflare just said it is writing down who drives around it.

What did Cloudflare announce on October 16?

Cloudflare announced a system that keeps grading Verified bots after they're approved. In Continuous accountability for Verified bots, Chris Tsoukaladelis splits the question in two. Web Bot Auth answers "are you who you say you are?" The new system answers "can we trust you?", and it starts with two behaviors:

  1. robots.txt over time. Cloudflare looks for "repeated patterns of behavior that disregards site preferences" and says "isolated violations and other types of one-offs may not be significant."
  2. The response to a 429. In Cloudflare's words, "A 429 is a directive for the visitor to reduce its request rate." The system checks whether a bot slows down, pauses, or keeps the same pace.

The output is a trust band: "a range showing how consistently a bot respects site preferences and which observed behaviors informed that assessment." Consistently slowing or pausing raises it. Repeatedly ignoring the signal lowers it.

Here is what is live and what isn't, based on the post:

Item Status on October 22, 2026
Grading Verified bots on robots.txt and 429 behavior Live, used internally, in an initial phase to build history
Trust scores published on Cloudflare Radar Planned, no date given
Customers choosing a trust threshold for Verified bot traffic Planned, no date given
More behaviors beyond the first two Planned, no list given
Number of bands, observation window, penalties Not published

The post does not say how a 429 response is timed, how many ignored 429s move a band, or whether Retry-After is checked. Anyone telling you the exact threshold is guessing.

Does Cloudflare block crawlers that ignore 429s today?

There's no published automatic block, but there's a precedent and a direction. The precedent is that Cloudflare removed Perplexity's bot from the Verified bot list in 2025 after it reportedly reached content sites had opted out of. The direction is that the trust band turns that kind of judgment from a one-off decision into a running score, and once customers can pick a threshold, a low band becomes something each site acts on.

For crawlers that aren't Verified bots, nothing about this announcement is new. They already face Cloudflare's per-request bot score and whatever rate limits each site sets, which we cover in why AI agents get blocked by Cloudflare. The change is at the top: the bots that were trusted by default now have to keep earning it.

Why does the 429 response matter so much now?

It's the most direct request a site can make in the middle of a crawl: slow down. HTTP 429 Too Many Requests is defined in RFC 6585 (2012), which says the response MAY include a Retry-After header. Retry-After is defined in RFC 9110 and comes in two forms: a number of seconds (Retry-After: 120) or an HTTP date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT). It also appears on 503 responses, where it tells the client how long the service expects to be down.

Google has been steering sites toward these codes for years. Its guidance is to slow Googlebot with 500, 503 or 429 responses for a short period, a couple of hours or 1 to 2 days, and never with 403 or 404. So when a site under load sends 429s, it is using the signal Google asked sites to use, and Cloudflare is now grading which bots listen.

What counts as backing off after a 429?

Backing off means the next request to that host comes later than it would have, and keeps coming later for a while. Cloudflare's post names three outcomes: slow down, pause, or keep the same pace. The gap between them is large. Take one host that rate-limits a crawler for 5 minutes and answers every request in that window with a 429:

Bar chart of requests sent to one host during a 5-minute rate limit by a crawler running at 2 requests per second, computed from illustrative inputs: ignores the 429 and keeps its pace, 600 requests; retries every 1 second, 300; waits a fixed 10 seconds, 30; exponential backoff starting at 1 second, doubling, capped at 60 seconds, 10; honors Retry-After: 300 and pauses, 1.

The crawler that ignores the 429 sends 600 requests the site asked it not to send. Retrying on a short fixed timer still sends 300. Exponential backoff without reading the header gets down to 10, at 0, 1, 3, 7, 15, 31, 63, 123, 183 and 243 seconds. Reading the header gets it to 1: the request that triggered the limit.

Strategy Requests sent into the 5-minute limit What the site sees
Ignores the 429, keeps 2 requests per second 600 The same pace, as if nothing was said
Retries every 1 second 300 Half the pace, still nonstop
Waits a fixed 10 seconds 30 Slower, never pauses
Exponential backoff, 1 s doubling to a 60 s cap 10 Slows down sharply
Honors Retry-After: 300 1 Pauses until asked to return

These are computed, not measured. Real limits vary in length, and many sites send no Retry-After at all, which is why a crawler needs both behaviors: read the header when it's there, back off on its own when it isn't.

One more detail separates a polite crawler from a retry loop. After a 429, the steady pace for that host should stay slower for a while, not snap back to full speed the moment one request succeeds. Cloudflare grades how a bot behaves over time, and a crawler that hits the limit, waits, then sprints into it again looks like the second row with extra steps.

How should a crawler handle robots.txt status codes?

Follow RFC 9309, the Robots Exclusion Protocol, which is the standard Cloudflare cites. The parts crawlers most often get wrong are what to do when robots.txt doesn't come back as a clean 200:

robots.txt response What RFC 9309 says to do
200 Parse it and follow the group that matches your user agent (case-insensitive), or the * group
Redirect Follow at least five consecutive redirects, even across hosts
4xx (such as 404) The file is "unavailable": you may crawl any resource
5xx or network error The file is "unreachable": assume complete disallow
Cached copy Don't use it for more than 24 hours, unless the file is unreachable
Large file Parse at least the first 500 KiB

The 5xx rule surprises people. A site that's down or overloaded has not said "go ahead"; the standard says to stay out until robots.txt comes back. Crawl-delay is not part of RFC 9309, but many sites still publish it, and honoring it costs you nothing.

Can a proxy fix a crawler that ignores 429s?

No. A proxy changes the address a request comes from, and that's all it changes. The trust band is a score for a Verified bot, and a bot is verified by who it is, so changing exit IPs either leaves the score where it was or drops the bot out of verification entirely. Spreading a crawler that ignores 429s across more IPs makes things worse: the site asked for less load and got the same load from more places.

Where proxies do fit is identity and steadiness. Cloudflare's own Web Bot Auth post calls IP-based bot verification brittle, partly because addresses can be "shared by multiple users" through proxies and VPNs, and because published IP ranges change over time. Its answer is cryptographic: the bot signs requests under RFC 9421 HTTP Message Signatures, sending Signature, Signature-Input and Signature-Agent headers, so the identity travels with the request whatever the exit IP. If you crawl through proxies and want to be a Verified bot, signing is the path.

Dedicated static IPs then make the rest easier to reason about. Each address is yours alone, it doesn't change between runs, and its request rate to any host is exactly what your scheduler sent, with no other customer's traffic mixed in. That's useful when you're trying to prove to yourself, and to a site owner reading their logs, that your crawler paced itself.

What did the Cloudflare and OpenAI pilot show?

It showed how much crawler behavior Cloudflare can see, and how much of it is waste. On October 15, Cloudflare and OpenAI reported early results from a pilot that started in July: with network signals from a cohort of about 1% of Cloudflare's network, OpenAI discovered and indexed hundreds of millions of new pages. The signals included bots requesting files that don't exist, sites that redirect nearly every address to the homepage, and multiple addresses serving the same content.

Two lines from that post are worth keeping. "A crawler that repeatedly downloads an unchanged page adds load without bringing any benefit." And: "A signal about a page doesn't grant permission to access it." Read together with the trust bands a day later, the message to crawler operators is consistent. Cloudflare sees what your crawler wastes, and it's starting to grade it.

A polite crawler in 57 lines

This Python script fetches a list of URLs through a proxy and keeps one compliance record per host. It reads robots.txt with the RFC 9309 status rules from the table above, honors Crawl-delay, reads Retry-After in both forms on 429 and 503, falls back to doubling the gap with random jitter when there's no header, keeps the slower pace after a 429 and speeds back up by 10% per success, and parks a host entirely when asked to wait longer than 5 minutes. It needs pip install httpx.

"""Polite crawler: robots.txt (RFC 9309), 429/503 Retry-After, backoff, per-host log."""
import email.utils, json, os, random, sys, time
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser
import httpx

UA = "ExampleBot/1.0 (+https://example.com/bot)"  # a real name and contact page
PROXY = os.environ.get("PROXY_URL", "http://USER:PASS@proxy.statproxies.com:8080")
BASE_GAP, MAX_WAIT, MAX_TRIES = 1.0, 300.0, 5

def retry_after(value):  # delay-seconds or HTTP-date, per RFC 9110
    if not value: return None
    if value.strip().isdigit(): return float(value)
    try: return max(0.0, email.utils.parsedate_to_datetime(value).timestamp() - time.time())
    except (TypeError, ValueError): return None

class Host:
    def __init__(self, client, origin):
        self.robots, self.next_at = RobotFileParser(), 0.0
        self.log = {"host": origin, "fetched": 0, "robots_skip": 0, "throttled": 0, "gave_up": 0, "waited_s": 0.0}
        try:
            r = client.get(origin + "/robots.txt")
            if r.status_code >= 500: raise httpx.HTTPError("robots.txt 5xx")
            self.robots.parse(r.text.splitlines() if r.status_code == 200 else [])  # 4xx: allow
        except httpx.HTTPError:  # unreachable: RFC 9309 says assume complete disallow
            self.robots.parse(["User-agent: *", "Disallow: /"])
        self.base = self.gap = max(BASE_GAP, float(self.robots.crawl_delay(UA) or 0))

def fetch(client, hosts, url):
    origin = "{0.scheme}://{0.netloc}".format(urlsplit(url))
    host = hosts.get(origin) or hosts.setdefault(origin, Host(client, origin))
    if host.next_at - time.monotonic() > MAX_WAIT: host.log["gave_up"] += 1; return None  # parked
    if not host.robots.can_fetch(UA, url): host.log["robots_skip"] += 1; return None
    for _ in range(MAX_TRIES):
        pause = host.next_at - time.monotonic()
        if pause > 0: time.sleep(pause); host.log["waited_s"] = round(host.log["waited_s"] + pause, 1)
        try: r = client.get(url)
        except httpx.TransportError: r = None  # timeouts and resets count as "slow down" too
        if r is not None and r.status_code not in (429, 503):
            host.gap = max(host.base, host.gap * 0.9)  # speed back up slowly
            host.next_at = time.monotonic() + host.gap
            host.log["fetched"] += 1; return r
        host.log["throttled"] += 1
        host.gap = min(MAX_WAIT, host.gap * 2)  # and stay slower for the rest of the run
        hint = retry_after(r.headers.get("retry-after")) if r is not None else None
        wait = hint if hint is not None else host.gap + random.uniform(0, host.gap)
        host.next_at = time.monotonic() + max(wait, host.gap)
        if wait > MAX_WAIT: break  # a long pause was asked for: park the host
    host.log["gave_up"] += 1; return None

if __name__ == "__main__":
    hosts = {}
    with httpx.Client(proxy=PROXY or None, headers={"User-Agent": UA}, follow_redirects=True, timeout=30) as client:
        for url in open(sys.argv[1]).read().split():
            r = fetch(client, hosts, url)
            print(url, r.status_code if r else "skipped", file=sys.stderr)
    for host in hosts.values(): print(json.dumps(host.log))

Run it with a file of URLs, one per line: PROXY_URL=http://USER:PASS@proxy.statproxies.com:8080 python3 polite_crawl.py urls.txt. Set PROXY_URL= to an empty value to run without a proxy. Replace the user agent with your bot's real name and a page that explains it; Cloudflare's first Verified bot rule is to identify yourself accurately.

We ran it on October 22, 2026 against a local test server that disallowed /private with Crawl-delay: 1, answered one path with two 429s carrying Retry-After: 2, one with a 503 carrying an HTTP-date three seconds out, one with two 429s and no header, and one with Retry-After: 3600. A second test host answered robots.txt with a 503 and a third with a 404. The log for the first host:

{"host": "http://127.0.0.1:8901", "fetched": 4, "robots_skip": 1, "throttled": 6, "gave_up": 2, "waited_s": 105.8}

The disallowed path was never requested. The host that sent Retry-After: 3600 was parked, and the next URL for it was skipped without a request. The 503 robots.txt host got zero page requests, and the 404 host was crawled. The 105.8 seconds of waiting is the point: by the end of the run the crawler was spacing requests 20 to 40 seconds apart on that host, and it stays slower until the host stops pushing back.

The script is sequential to stay short. To crawl many hosts at once, run one worker per host so each keeps its own pace, and size your IP count from your total request rate as in our proxy sizing guide. For large AI crawls, cost per terabyte and connection math matter more than any single hop.

When Stat is the wrong pick

  • Your crawler ignores 429s and you want a fix. Fix the crawler. No proxy, ours included, changes how a site scores what you send.
  • You need content from outside the US. Every Stat IP is in the US, and pages that vary by country need IPs there.
  • You need a new address for every request. Stat is static only. Sites that block any repeat visitor need a rotating pool, which we don't sell.
  • Your crawl is small or one-off. Plans start at 25 IPs. A few thousand pages a month doesn't need that.

If you run a steady, well-paced crawler against US sites, static ISP proxies start at $2.50 per IP per month with unlimited bandwidth and no concurrency limit, and they go live the moment you check out. The AI data use case page covers setups for dataset refreshes and agent workloads.

FAQ

Does Cloudflare block crawlers that ignore 429 rate limits?

Cloudflare has not published an automatic block. On October 16, 2026, it said it grades Verified bots on whether they slow down or pause after an HTTP 429 and whether they follow robots.txt over time, producing a trust band. It plans to publish scores on Radar and let customers choose a trust threshold for Verified bot traffic. In 2025 it removed Perplexity's bot from the Verified list after it reportedly reached opted-out content.

What is Cloudflare's Verified bot trust band?

A trust band is Cloudflare's running assessment of how consistently a Verified bot respects site preferences, along with which observed behaviors informed it. The first two behaviors are robots.txt compliance over time and the response to HTTP 429. Slowing down or pausing after a 429 raises trust; repeatedly ignoring it lowers trust. Cloudflare uses it internally today and has not published the number of bands or thresholds.

How long should a crawler wait after a 429?

If the response has a Retry-After header, wait at least that long: it's either a number of seconds or an HTTP date (RFC 9110). If there's no header, back off exponentially, for example 1 second doubling up to a cap, with random jitter so many workers don't retry together. Then keep that host's pace slower for a while instead of returning to full speed after one success.

What should a crawler do if robots.txt returns a 503?

Treat the whole site as disallowed. RFC 9309 calls a 5xx or network error "unreachable" and says the crawler must assume complete disallow until robots.txt can be fetched again. A 4xx such as 404 is different: it means the file is unavailable, and the crawler may access any resource. Cached copies shouldn't be used for more than 24 hours unless the file is unreachable.

Can rotating IPs get around a low Cloudflare trust score?

No. Cloudflare's trust band scores a Verified bot by its identity, which Web Bot Auth can tie to cryptographic signatures instead of an IP address, so changing exit IPs doesn't change the bot being graded. Spreading a crawler that ignores 429s across more addresses also multiplies the load the site asked to reduce. The fix is pacing: honor Retry-After, back off, and follow robots.txt.

Sources