How to Fix HTTP Errors When Web Scraping (403, 429, 503, 520)

I've written a lot of scrapers, and almost all of them have failed in the same way at some point. The script runs fine on Monday. By Wednesday every request comes back as a 403 Forbidden or a 429 Too Many Requests, and I'm staring at a stack trace wondering what changed. Usually nothing changed on my end. The site just got tired of me.

That's the thing most tutorials skip: HTTP errors when web scraping aren't random noise. They're the target site talking to you. Sometimes it's a server hiccup. More often it's an anti-bot layer telling you that your requests look automated, arrive too fast, or come from an IP it doesn't trust. Once you learn to read these signals, debugging gets a lot less painful.

This guide walks through the HTTP errors web scraping practitioners actually hit in the wild. For each one I'll explain why it fires, how to confirm what you're dealing with, and give you code you can paste into a real project. If your end goal is feeding clean data to a model, the roundup of web scraping APIs built for AI is a useful companion. Otherwise, let's get into the errors.

403 vs. 429: The Short Answer

A 429 Too Many Requests usually indicates a pacing problem: your IP, account, session, or API key exceeded a request limit. A 403 Forbidden more often indicates an identity or access decision, such as missing authorization, a geo-policy, IP reputation, or bot mitigation. Neither status proves the cause by itself, so compare the headers, body, session behavior, and results of controlled tests before changing infrastructure.

Signal	More consistent with 429 rate limiting	More consistent with a 403 identity, access, or bot block	Next test to run
Status code	429 is the clearest pacing signal	403 means the server understood the request but refused it	Do not stop at the code; save the headers and a body preview
Response headers	Retry-After or other rate-limit metadata	CDN, WAF, challenge, authentication, or policy-related headers	Follow Retry-After; otherwise compare the failed response with a successful browser request
Response body	A message about quotas, request frequency, or waiting	A challenge page, access-denied message, login requirement, geo restriction, or application error	Identify whether the body came from the application, CDN, or anti-bot layer
Retry behavior	Often clears after the requested wait or after lowering request frequency	Usually persists if you repeat the same identity and request	Retry a 429 after the indicated delay; do not blindly retry a persistent 403
Cookies and session	A limit may be tied to a session or account, but fresh cookies do not necessarily fix it	Seeding legitimate cookies or reusing an authenticated session may change the result	Repeat once with the same client and cookies used by a successful browser session
IP effects	A different IP may reset an IP-based limit	A controlled IP change may reveal reputation, network, or geo-policy effects	Keep headers, cookies, URL, and timing fixed while changing only the IP
Concurrency effects	Error rates rise as workers or request frequency increase	A 403 rate that stays high at one worker points away from pacing alone	Start with one worker, increase slowly, and record separate 403 and 429 rates

A Quick Mental Model: Three Buckets

Before the code, it helps to sort every status code into one of three buckets. I do this in my head every time a request fails, because it tells me whether to retry, fix something, or give up.

1. You broke the request. The server understood you fine, but your headers, body, or auth were wrong. These are on you to fix. Think 400, 401, 422.
2. The site refused or limited the request. Your request looked automated or abusive, so the anti-bot layer rejected it. This is the interesting bucket. Think 403, 429, and a lot of disguised 503 and 520 responses.
3. The server had a problem. Nothing to do with you. The origin is overloaded, down, or timing out. Just retry sensibly. Think 500, 502, 521, 522, 524.

The reason this matters: retrying a request from bucket one will never work, no matter how many times you try. Retrying bucket three usually works on its own. Bucket two is where you spend your engineering effort. A 429 usually calls for lower request volume and backoff. A 403 calls for evidence about authorization, cookies, IP or geo policy, and bot mitigation before you choose a fix.

HTTP Status Codes at a Glance: a Scraper's Reference

Here's the cheat sheet I keep open. Not every error means you're blocked, and treating them all the same is how people waste hours debugging the wrong thing.

Status Code	Name	What it usually means for scrapers	Blocked?
400	Bad Request	Malformed headers or request body	Sometimes, check your payload
401	Unauthorized	Missing credentials or auth token	Yes, you need to authenticate first
403	Forbidden	Identity or access refusal: auth, policy, geo restriction, IP reputation, or bot mitigation	Sometimes; inspect the response
404	Not Found	Page doesn't exist or was removed	No, skip and move on
407	Proxy Auth Required	Your proxy needs credentials	Yes, fix proxy config
408	Request Timeout	Server gave up waiting for your request	No, transient, retry
410	Gone	Resource permanently deleted	No, remove from queue
422	Unprocessable Entity	Server rejects your params or body	Sometimes, check your POST body
429	Too Many Requests	Rate-limited: too many requests too fast	Yes
451	Unavailable For Legal Reasons	Content geo-blocked or removed by law	Yes, need different IP or skip
500	Internal Server Error	Server bug, retry is fine	No, transient, retry with backoff
502	Bad Gateway	Upstream server gave a bad response	No, transient, retry
503	Service Unavailable	Overloaded or anti-bot challenge	Sometimes
504	Gateway Timeout	Upstream server took too long	No, transient, retry
520	Unknown Error (Cloudflare)	Cloudflare received an unexpected response from the origin; investigate before attributing it to bot mitigation	Sometimes
521	Web Server Is Down (Cloudflare)	Origin is unreachable	No, site is down, retry later
522	Connection Timed Out (Cloudflare)	TCP handshake timeout to origin	No, transient, retry
524	A Timeout Occurred (Cloudflare)	Origin took too long to respond	No, transient, retry

The short version: 4xx codes usually mean the server understood your request and refused it, often because it decided you're a bot. 5xx codes usually mean a server-side problem, frequently transient, but occasionally an anti-bot challenge wearing a 503 costume. We'll get to that disguise later because it trips people up constantly.

How to Actually Diagnose a Block

Use this decision tree before changing headers, proxies, or browser tooling:

1. Capture the evidence. Record the status, final URL, response headers, the first part of the body, elapsed time, worker count, IP or proxy identifier, and session identifier.
2. If the status is 429, inspect Retry-After. It can be delta-seconds or an HTTP date. Wait accordingly, reduce pacing or concurrency, and retry with the same session and identity.
3. If a slower retry clears the error, treat pacing as the leading cause. Continue at the lower rate and raise concurrency only through a controlled test.
4. If the status is 403, classify the body. Look for an application authorization error, login wall, geo-policy message, CDN denial, or JavaScript challenge. Do not assume every 403 is bot detection.
5. Compare with a successful browser request. Keep the URL and account fixed, then compare headers, cookies, redirects, and the response body. Reuse a legitimate session if the page requires one.
6. Change one variable at a time. Test cookies or session state first, then request headers, then a controlled IP change. Changing all three together may restore access without revealing why.
7. Interpret the IP test carefully. A changed result can point to IP reputation, an IP-based rate limit, or geo policy. It does not by itself prove bot detection.
8. Stop blind retries. If the same 403 persists at one worker with the same body, repeated retries are unlikely to help. Move to the relevant authorization, browser, proxy, or managed-infrastructure test.

Most people, myself included on a bad day, read the status code and immediately start guessing. Don't do that. Spend two minutes gathering evidence first. It's faster in the long run.

When a request fails, I check three things before I change a single line of scraping logic.

First, print the response body. A status code tells you almost nothing on its own. The body tells you everything. A real 403 from a CDN often includes a Cloudflare ray ID or a "Sorry, you have been blocked" page. A challenge page says "Checking your browser" or "Just a moment." A genuine application error returns a JSON message. You cannot tell these apart from the status code alone.

import httpx
 
resp = httpx.get("https://example.com/products")
print("status:", resp.status_code)
print("server:", resp.headers.get("server"))
print("body preview:", resp.text[:600])

Second, look at the response headers. The Server, CF-RAY, Retry-After, and cf-mitigated headers reveal who's blocking you and why. If you see server: cloudflare, you now know which anti-bot vendor you're up against. If you see Retry-After, the server is literally telling you how long to wait.

Third, reproduce it in curl. This is the step people skip, and it's the most useful one. If you can reproduce the failure with a plain curl command, you can iterate quickly without rerunning your whole pipeline. Copy the request as cURL straight from your browser's DevTools Network tab, run it, and see if it succeeds where your scraper fails. The difference between those two requests is your bug.

curl -sS -D - "https://example.com/products" \
  -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36" \
  -o /dev/null

Once you've got evidence, you can match it to the sections below. Now the fun part.

403 Forbidden: Identity, Access Policy, or Bot Mitigation

Why You're Getting It

A 403 Forbidden means the server understood the request and refused to authorize it. During scraping, that can come from application permissions, missing authentication, geo or network policy, IP reputation, or an anti-bot layer. Inspect the response body and headers before deciding which one you have. Here are the usual culprits, roughly in the order I check them:

• A missing or unrealistic User-Agent. python-requests/2.31.0 is a flashing neon sign that says "I am a script." Real browsers send long, specific UA strings.
• Missing secondary headers. Browsers send Accept, Accept-Language, Accept-Encoding, the Sec-Fetch-* family, and Cache-Control. A bare requests.get() sends almost none of that, and the gap is easy to detect.
• No cookies or session state. Plenty of sites check whether you accepted a consent banner or carry a session token from a previous page. Show up cold and you look suspicious.
• A TLS fingerprint that doesn't match. Python's ssl library and Chrome produce different TLS Client Hello messages. Tools like Cloudflare fingerprint this (JA3/JA4) at the TCP layer, before they even read your HTTP headers.
• A bad IP reputation. Data-center ranges (AWS, GCP, DigitalOcean) are heavily flagged because that's where scrapers live. Residential IPs get the benefit of the doubt.
• Geo-restrictions. Some pages only serve specific countries, and everyone else gets a 403.

I've seen all six of these in production. The good news is they stack from cheap to expensive, so you fix them in order and stop as soon as the block clears.

The Fix

Step 1: Send realistic headers. This alone clears a surprising number of 403s. Open DevTools, find the request in the Network tab, copy the headers a real browser sent, and feed them to your client.

import httpx
 
HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": (
        "text/html,application/xhtml+xml,application/xml;"
        "q=0.9,image/avif,image/webp,*/*;q=0.8"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "DNT": "1",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}
 
with httpx.Client(headers=HEADERS, follow_redirects=True) as client:
    resp = client.get("https://example.com/products")
    resp.raise_for_status()
    print(resp.text[:500])

One detail people miss: header order matters to some fingerprinting systems. Browsers send headers in a consistent order, and a few WAFs flag requests where the order looks wrong. Most of the time you won't need to worry about it, but if realistic header values aren't enough, header ordering is the next thing to investigate.

Step 2: Keep a session and seed cookies. If the target sets cookies on its homepage, visit the homepage first so your later requests carry a legitimate-looking session.

import httpx
 
with httpx.Client(headers=HEADERS, follow_redirects=True) as client:
    # Seed cookies by visiting the home page first
    client.get("https://example.com/")
    # Now hit the target page with the seeded session
    resp = client.get("https://example.com/products")

Step 3: Run a controlled IP test. If headers and cookies do not change the result, repeat the same request through a permitted alternate network or proxy while keeping the URL, timing, headers, and session fixed. If the response changes, investigate IP reputation, an IP-based limit, or geo policy. Only then decide whether rotating proxies are justified. A changed IP result is useful evidence, but it does not prove bot detection by itself.

import httpx
 
proxy = "http://user:pass@residential-proxy.example.com:8080"
 
with httpx.Client(headers=HEADERS, proxy=proxy, follow_redirects=True) as client:
    resp = client.get("https://example.com/products")

Step 4: Use a headless browser for JavaScript-gated 403s. Some 403s fire because the page runs JavaScript that inspects the browser environment before serving real content. No amount of header tuning fixes that, because requests and httpx never run the JavaScript. You need Playwright or Puppeteer.

from playwright.sync_api import sync_playwright
 
with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        user_agent=HEADERS["User-Agent"],
        locale="en-US",
        extra_http_headers={"Accept-Language": "en-US,en;q=0.9"},
    )
    page = context.new_page()
    page.goto("https://example.com/products")
    html = page.content()
    browser.close()

If you only need the fully rendered markup and don't want to babysit a browser, the guide on extracting raw HTML from any URL with a single API call covers the managed alternative to running Playwright yourself.

429 Too Many Requests: You're Being Rate-Limited

Why You're Getting It

A 429 Too Many Requests is the server politely asking you to slow down. Most production sites and APIs enforce rate limits: a cap on how many requests one IP or API key can make in a rolling window. Cross it and you get a 429. Many of these responses include a Retry-After header that tells you exactly how long to wait. Ignore it and keep hammering, and a 429 can quietly escalate into a 403 ban that sticks around for hours.

This is the single most common way amateur scrapers fall over at scale, and it's almost always the same mistake: a tight loop with no delays, hitting one IP, with a predictable URL pattern, at machine speed. No human browses like that, and the server knows it.

The Fix: Backoff With Jitter, and Respect Retry-After

Retry-After has two valid forms: a number of seconds or an HTTP date. Production code should parse both, cap the wait, add jitter to computed backoff, and stop after a finite number of attempts. If the server asks for a delay longer than your wait budget, stop and reschedule the work instead of retrying early. It should retry 429 and selected transient server errors, not a persistent 403.

The pattern that works is straightforward once you've written it once:

1. If the response has a Retry-After header, wait that long before retrying.
2. If it doesn't, use exponential backoff: wait base * 2^attempt seconds.
3. Add jitter (random noise) so a fleet of workers doesn't all retry at the same instant and re-trigger the limit. This thundering-herd problem is real and I've caused it more than once.
4. Cap the retries and raise an error once you hit the cap, instead of looping forever.

Python

import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
 
import httpx
 
RETRYABLE = {429, 500, 502, 503, 504, 521, 522, 524}
 
 
def retry_after_seconds(value: str | None) -> float | None:
    if not value:
        return None
    try:
        return max(0.0, float(value))
    except ValueError:
        try:
            retry_at = parsedate_to_datetime(value)
            if retry_at.tzinfo is None:
                retry_at = retry_at.replace(tzinfo=timezone.utc)
            return max(0.0, (retry_at - datetime.now(timezone.utc)).total_seconds())
        except (TypeError, ValueError, OverflowError):
            return None
 
 
def fetch_with_backoff(
    client: httpx.Client,
    url: str,
    *,
    max_attempts: int = 6,
    base_delay: float = 1.0,
    max_delay: float = 120.0,
) -> httpx.Response:
    for attempt in range(max_attempts):
        response = client.get(url)
 
        if response.status_code == 403:
            response.raise_for_status()
        if response.status_code not in RETRYABLE:
            response.raise_for_status()
            return response
        if attempt == max_attempts - 1:
            response.raise_for_status()
 
        retry_after = retry_after_seconds(response.headers.get("Retry-After"))
        if retry_after is not None:
            if retry_after > max_delay:
                raise RuntimeError("Retry-After exceeds wait budget; reschedule this URL")
            wait = retry_after
        else:
            ceiling = min(base_delay * (2 ** attempt), max_delay)
            wait = random.uniform(0, ceiling)
 
        time.sleep(wait)
 
    raise RuntimeError("unreachable")
 
 
with httpx.Client(headers=HEADERS, follow_redirects=True, timeout=20.0) as client:
    response = fetch_with_backoff(client, "https://example.com/api/products")
    print(response.status_code)

Node.js

const RETRYABLE = new Set([429, 500, 502, 503, 504, 521, 522, 524]);
 
function parseRetryAfter(value) {
  if (!value) return null;
 
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000);
 
  const retryAt = Date.parse(value);
  if (Number.isNaN(retryAt)) return null;
  return Math.max(0, retryAt - Date.now());
}
 
const sleep = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));
 
async function fetchWithBackoff(
  url,
  { maxAttempts = 6, baseDelayMs = 1000, maxDelayMs = 120000 } = {},
) {
  for (let attempt = 0; attempt < maxAttempts; attempt += 1) {
    const response = await fetch(url, { redirect: "follow" });
 
    if (response.status === 403) {
      throw new Error(`Persistent 403 requires diagnosis: ${url}`);
    }
    if (!RETRYABLE.has(response.status)) {
      if (!response.ok) throw new Error(`HTTP ${response.status}: ${url}`);
      return response;
    }
    if (attempt === maxAttempts - 1) {
      throw new Error(`HTTP ${response.status} after ${maxAttempts} attempts: ${url}`);
    }
 
    const retryAfterMs = parseRetryAfter(response.headers.get("retry-after"));
    await response.body?.cancel();
    if (retryAfterMs !== null && retryAfterMs > maxDelayMs) {
      throw new Error(`Retry-After exceeds wait budget; reschedule: ${url}`);
    }
    const ceiling = Math.min(baseDelayMs * (2 ** attempt), maxDelayMs);
    const waitMs = retryAfterMs === null
      ? Math.random() * ceiling
      : retryAfterMs;
 
    await sleep(waitMs);
  }
 
  throw new Error("unreachable");
}
 
const response = await fetchWithBackoff(
  "https://example.com/api/products",
);
console.log(response.status);

Throttle Before You Get Throttled

Backoff is reactive. It kicks in after you've already been told off. A better habit is to pace yourself so you rarely see a 429 in the first place. The cleanest way I know is a token-bucket throttle: you refill tokens at a fixed rate, and every request spends one. When the bucket is empty, you wait. It smooths your traffic into something that looks deliberate instead of frantic.

import time
import threading
 
class TokenBucket:
    """Allow up to `rate` requests per second, with a small burst allowance."""
 
    def __init__(self, rate: float, burst: int = 1):
        self.rate = rate
        self.capacity = burst
        self.tokens = burst
        self.last = time.monotonic()
        self.lock = threading.Lock()
 
    def take(self) -> None:
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
            self.last = now
            if self.tokens < 1:
                sleep_for = (1 - self.tokens) / self.rate
                time.sleep(sleep_for)
                # Account for the time we just slept so the next call doesn't
                # recount it and let pacing drift faster than `rate`.
                self.last = time.monotonic()
                self.tokens = 0
            else:
                self.tokens -= 1
 
 
# One request every two seconds, no bursting
bucket = TokenBucket(rate=0.5, burst=1)
 
for url in urls:
    bucket.take()
    resp = httpx.get(url, headers=HEADERS)

Concurrency Limits

When you run workers in parallel, cap concurrency so you don't accidentally fire fifty simultaneous requests from one IP. In asyncio with httpx, a semaphore does the job, and I add a small random delay on top so the requests don't land in lockstep.

import asyncio
import random
import httpx
 
CONCURRENCY = 5  # max simultaneous in-flight requests
 
async def scrape_all(urls: list[str]) -> list[str]:
    sem = asyncio.Semaphore(CONCURRENCY)
 
    async def fetch(client: httpx.AsyncClient, url: str) -> str:
        async with sem:
            # small per-request delay in addition to the semaphore
            await asyncio.sleep(random.uniform(0.5, 1.5))
            r = await client.get(url, headers=HEADERS)
            r.raise_for_status()
            return r.text
 
    async with httpx.AsyncClient(follow_redirects=True) as client:
        tasks = [fetch(client, url) for url in urls]
        return await asyncio.gather(*tasks)

Treat concurrency changes as an experiment rather than a guess:

1. Start with one worker and a fixed per-domain delay.
2. Run a representative batch and record total requests, latency, and separate 403 and 429 counts.
3. Increase only one variable, such as worker count or inter-request delay. Keep the URL mix, session strategy, headers, and IP pool unchanged.
4. Run the same-sized batch and compare error rates rather than raw counts.
5. If 429 responses increase, roll back and lower pacing or concurrency. If 403 responses increase, stop and inspect their bodies and session or IP correlation instead of adding more workers.
6. Keep the highest setting that does not materially worsen errors, and repeat the test when the target changes.

This test is intentionally conservative. It gives you evidence about whether load triggers the failures without opening at full throttle and turning a small pacing problem into a longer-lived refusal.

503 Service Unavailable: Overloaded, or a Challenge in Disguise

Why You're Getting It

503 Service Unavailable is the ambiguous one. It can mean the server is genuinely overloaded and wants you to back off. It can also be an anti-bot challenge page wearing a 503 status code. Some CDNs and web application firewalls return a 503 with an HTML body that says "Checking your browser before you access" or "DDoS protection by [vendor]." If you print the body and see that kind of text, it isn't a real 503 at all. It's a JavaScript challenge, and no amount of backoff will solve it.

This is exactly why the diagnosis step earlier matters. The status code lies; the body doesn't.

The Fix

For a genuine 503 (real overload): retry with exponential backoff, the same way you handle a 429. The fetch_with_backoff function above already covers it, because it treats 503 and 429 identically.

For a challenge-page 503: detect it, then switch tools. A simple keyword heuristic catches most of them.

import httpx
 
CHALLENGE_SIGNALS = [
    "checking your browser",
    "ddos protection",
    "one more step",
    "please wait",
    "just a moment",
    "enable javascript",
]
 
def is_challenge_page(resp: httpx.Response) -> bool:
    if resp.status_code not in (503, 403):
        return False
    body_lower = resp.text.lower()
    return any(signal in body_lower for signal in CHALLENGE_SIGNALS)
 
 
resp = httpx.get("https://example.com", headers=HEADERS)
if is_challenge_page(resp):
    print("Challenge page detected, switch to headless browser or scraping API")
else:
    print(resp.text[:200])

Once you've confirmed it's a challenge, you have two real options: drive a headless browser (Playwright, as shown in the 403 section) or hand the request to a managed scraping API that solves challenges for you. Which one you pick depends on how often you hit it and how much maintenance you're willing to own, which is a tradeoff I'll come back to at the end.

520, 521, 522, 524: The Cloudflare Family

What They Mean

Cloudflare sits in front of a huge slice of the web, so you will run into its custom error codes constantly. They fire when Cloudflare can't successfully proxy your request to the origin server:

• 520 Unknown Error. Cloudflare received an unexpected or empty response from the origin. Bot mitigation may be one explanation in a scraping workflow, but the code alone does not establish whether the cause was fingerprinting, IP reputation, application behavior, or an origin failure.
• 521 Web Server Is Down. Cloudflare is fine, but the origin's TCP port is closed. The site is genuinely down. Retry later.
• 522 Connection Timed Out. Cloudflare's TCP handshake to the origin timed out. Usually transient, so retry with backoff.
• 524 A Timeout Occurred. Cloudflare connected, but the origin took too long to respond. Also transient.

Three of those four (521, 522, 524) are not about you. The one that keeps scraper authors up at night is 520.

Why 520 Is the Tricky One

A 520 during scraping needs the same evidence-first treatment as other ambiguous failures. Compare the body, Cloudflare headers, timing, browser behavior, and repeated results before treating fingerprinting as the leading explanation. Cloudflare's Bot Management looks at signals that have nothing to do with your HTTP headers:

1. TLS Client Hello (the JA3/JA4 fingerprint). Python's ssl module orders cipher suites differently than Chrome does. Cloudflare can block you on that alone, before reading a single header.
2. HTTP/2 fingerprint. Chrome and httpx send HTTP/2 settings, header ordering, and pseudo-header sequences in distinct patterns. The mismatch is detectable.
3. Behavioral signals. No mouse movement, no scrolling, instant loads with none of the sub-resource fetching a real browser does.

So the fix isn't "send better headers." It's "look like a real browser at the network layer."

Use curl_cffi to impersonate Chrome's TLS fingerprint. This is the cheapest fix and it solves a real chunk of 520s on its own.

from curl_cffi import requests as cffi_requests
 
# impersonate="chrome120" patches the TLS and HTTP/2 fingerprint
resp = cffi_requests.get(
    "https://cloudflare-protected-site.com",
    impersonate="chrome120",
    headers={"Accept-Language": "en-US,en;q=0.9"},
)
print(resp.status_code, resp.text[:200])

If curl_cffi isn't enough, layer on stealth. Libraries like playwright-stealth mask the obvious headless tells: a missing plugin array, the wrong WebGL renderer, the navigator.webdriver flag that screams "automation." And if the site runs full Cloudflare Bot Management and you're still stuck, that's usually my signal that a managed scraping API is the better use of my time than another evening of fingerprint patching.

502 and 504: Gateway Errors

These two show up less in scraping write-ups, but you'll meet them. A 502 Bad Gateway means an upstream server returned an invalid response to the gateway. A 504 Gateway Timeout means the upstream server didn't respond in time. They usually indicate a gateway or upstream failure rather than a direct bot decision, but repeated responses still need investigation. Treat them exactly like a real 503: retry with exponential backoff and a sensible cap. If a particular URL returns 502 or 504 consistently across many retries and a long window, the origin probably has a genuine problem, so log it and move on rather than burning your whole retry budget on one dead endpoint.

The 200 OK That Is Actually a Block

Here's the failure mode that fooled me the longest. You get a 200 OK, your code happily moves on, and a week later you discover your database is full of garbage. The status was fine. The body was a captcha page, a login wall, or an empty shell that loads its real content over an API you never called.

These soft blocks are nastier than a clean 403 because nothing throws an error. You have to validate the content yourself. I now add a sanity check to every scraper: does the page contain a thing I actually expect? A product page should have a price. An article should have more than 500 characters of body text. If the expected marker is missing, I treat the 200 as a failure and route it through the same challenge-handling path as a 503.

def looks_like_real_content(html: str, must_contain: list[str]) -> bool:
    lowered = html.lower()
    if len(html) < 1000:
        return False
    return all(marker.lower() in lowered for marker in must_contain)
 
 
resp = httpx.get("https://example.com/products/123", headers=HEADERS)
if resp.status_code == 200 and not looks_like_real_content(resp.text, ["add to cart", "price"]):
    print("200 OK but content is missing, likely a soft block")

Don't trust the status code alone. Trust the bytes you got back.

Other Status Codes You'll Run Into

401 Unauthorized and 407 Proxy Authentication Required

A 401 means the endpoint wants authentication (an API key, an OAuth token, or a session cookie) and you didn't provide it. Add the right Authorization header, or log in first to obtain a session cookie, then reuse it.

A 407 is the same idea one layer down: your proxy server itself wants credentials. Add Proxy-Authorization, or embed user:pass@host:port in the proxy URL so the credentials travel with every request.

404 Not Found and 410 Gone

A 404 means the URL doesn't exist. Maybe the site restructured, the product sold out, or the page was deleted. Don't retry it. Log the URL and skip it. Retrying a 404 is pure wasted budget.

A 410 Gone is a 404 with conviction. The server is explicitly saying the resource is gone for good, so remove it from your crawl queue entirely and don't revisit.

451 Unavailable For Legal Reasons

A 451 means content was withheld because of a legal demand: a DMCA takedown, a court order, a regional regulation. The only real fix is an IP in a jurisdiction where the content isn't restricted. Often the honest answer is that the content is genuinely unavailable to you, so skip it.

422 Unprocessable Entity

A 422 usually fires on POST requests where your body is structurally valid JSON but fails the server's business rules. Check that your fields, required parameters, and data types match what the API expects. The good news is that most 422 responses include a machine-readable error in the body explaining what went wrong, so print resp.text and read it. The answer is almost always right there.

Being a Polite Scraper: robots.txt and Per-Domain Pacing

A lot of blocks are self-inflicted, and the cure is courtesy. Sites are far less likely to fight you when your traffic looks reasonable, so a little restraint goes a long way toward keeping your IPs clean.

Start by reading robots.txt. It's not legally binding everywhere, but it tells you which paths the site would rather you leave alone, and some robots.txt files specify a Crawl-delay that hints at how often you can request without annoying anyone. Python's standard library can parse it for you.

import urllib.robotparser
 
rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
 
if rp.can_fetch("MyScraper/1.0", "https://example.com/products"):
    print("Allowed by robots.txt")

Beyond that, pace yourself per domain, not globally. If you're crawling twenty sites at once, a single global rate limit either crawls everyone too slowly or hits one small site too hard. Track a separate throttle per hostname so a fragile site gets gentle treatment while a robust one gets the throughput it can handle. The TokenBucket from the 429 section works nicely here: keep a dictionary of buckets keyed by hostname and call take() on the right one before each request.

Caching helps too. If your pipeline re-requests the same URL during development (and it always does), cache responses locally so you're not re-hitting the live site every time you tweak a parser. Fewer requests means fewer chances to get blocked, and your tests run faster as a bonus.

A Reusable Resilient Fetch Pattern

Let's pull the useful pieces together: realistic headers, backoff with jitter, optional proxy rotation, and challenge detection in one function I can drop into a project and forget about.

This example reuses retry_after_seconds from the Python example above so both forms of Retry-After follow the same waiting policy.

import time
import random
import httpx
from typing import Sequence
 
BROWSER_HEADERS = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/125.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}
 
CHALLENGE_SIGNALS = [
    "checking your browser",
    "ddos protection",
    "just a moment",
    "enable javascript",
    "one more step",
]
 
RETRYABLE_CODES = {429, 500, 502, 503, 504, 521, 522, 524}
 
 
def resilient_fetch(
    url: str,
    proxy_list: Sequence[str] | None = None,
    max_retries: int = 5,
    base_delay: float = 1.5,
    max_delay: float = 90.0,
    extra_headers: dict | None = None,
) -> httpx.Response:
    """
    Fetch a URL with realistic browser headers, optional rotating proxies,
    exponential backoff with jitter on retryable codes, and challenge detection.
    """
    headers = {**BROWSER_HEADERS, **(extra_headers or {})}
    proxy_pool = list(proxy_list) if proxy_list else [None]
 
    for attempt in range(max_retries):
        # Rotate through proxy pool
        proxy_url = proxy_pool[attempt % len(proxy_pool)]
 
        try:
            with httpx.Client(
                headers=headers,
                proxy=proxy_url,
                follow_redirects=True,
                timeout=20.0,
            ) as client:
                resp = client.get(url)
        except (httpx.ConnectError, httpx.TimeoutException) as exc:
            print(f"[attempt {attempt + 1}] Network error: {exc}")
            if attempt == max_retries - 1:
                raise
            _backoff(attempt, base_delay, max_delay)
            continue
 
        # Only treat 403/503 bodies as possible challenge pages. Scanning every
        # response (including 200) would false-positive on real HTML that happens
        # to contain phrases like "just a moment".
        if resp.status_code in (403, 503) and any(
            sig in resp.text.lower() for sig in CHALLENGE_SIGNALS
        ):
            raise RuntimeError(
                f"Challenge page on {url}, use headless browser or scraping API"
            )
 
        if resp.status_code not in RETRYABLE_CODES:
            resp.raise_for_status()
            return resp
 
        if attempt == max_retries - 1:
            resp.raise_for_status()
 
        retry_after = retry_after_seconds(resp.headers.get("Retry-After"))
        if retry_after is not None:
            if retry_after > max_delay:
                raise RuntimeError("Retry-After exceeds wait budget; reschedule this URL")
            time.sleep(retry_after)
        else:
            _backoff(attempt, base_delay, max_delay)
 
        print(f"[attempt {attempt + 1}] {resp.status_code}, retrying")
 
    raise RuntimeError(f"All {max_retries} retries exhausted for {url}")
 
 
def _backoff(attempt: int, base: float, cap: float) -> None:
    wait = min(base * (2 ** attempt), cap)
    _backoff_for(wait)
 
 
def _backoff_for(max_wait: float) -> None:
    time.sleep(random.uniform(0, max_wait))
 
 
# --- Example usage ---
PROXIES = [
    "http://user:pass@proxy1.example.com:8080",
    "http://user:pass@proxy2.example.com:8080",
    "http://user:pass@proxy3.example.com:8080",
]
 
resp = resilient_fetch("https://example.com/products", proxy_list=PROXIES)
print(resp.status_code, len(resp.text))

For setting up the proxy pool itself, the guide to rotating proxies in web scraping covers sourcing and rotation strategy. If your end goal is feeding an LLM, the list of web scraping APIs built for AI is worth a look, and the writeup on extracting raw HTML from any URL shows how to skip the retry plumbing entirely.

A Real Debugging Session

Let me make this concrete with a job I actually did. I was scraping a company's address from the web for a few thousand domains, and about a fifth of them came back 403. My first instinct was the usual one: blame the IP. So I bought residential proxies, rerouted, and the failure rate barely moved. Money down the drain.

Then I went back to basics and printed the response body. Every failing site returned server: cloudflare and a "Just a moment" page. It wasn't an IP problem at all. It was a TLS fingerprint problem. My httpx requests advertised a cipher order no real Chrome would send, and Cloudflare was bouncing them at the network layer before my fancy proxies ever mattered.

Swapping httpx for curl_cffi with impersonate="chrome120" reduced the failures substantially in that job. The remaining sites required more involved browser and behavioral handling, and at that point I made a call: those domains weren't worth a custom Playwright rig with stealth patches, so I routed just that slice through a scraping API and moved on with my life.

The lesson I keep relearning: read the body first. I wasted half a day and real money guessing, when two minutes of looking at the actual response would have pointed me straight at the fix.

When to Stop Fighting and Use a Scraping API

There's a point of diminishing returns, and recognizing it is a skill in itself. If a site runs Cloudflare Bot Management, Akamai Bot Manager, Imperva, or DataDome, you are in an arms race against full-time engineers whose entire job is catching scrapers. Maintaining your own residential proxy pool, keeping TLS fingerprint patches current, solving captchas, and chasing behavioral detection can quietly cost more than the data is worth.

The practical alternative is managed scraping infrastructure that owns the proxy, browser, retry, and site-specific maintenance behind an API. I reach for it when the engineering and operating cost of maintaining those pieces is higher than the value of keeping the workflow in-house. Context.dev's web scraping API provides raw HTML and clean Markdown through one endpoint. You pass a URL and choose the output formats, and you get the content back after the service deals with the anti-bot mess for you. If you need to walk an entire site, the website crawling API handles the queue and pagination too.

curl -X POST "https://api.context.dev/v1/web/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/products","formats":{"markdown":true}}'

Read the returned Markdown from markdown.data. To request raw HTML as well, add "html": true to formats and read html.data.

You get clean Markdown or raw HTML back without writing a line of proxy rotation, backoff logic, or browser automation. If you're spending more time fighting blocks than using the data, the trade is worth it. If the sites you scrape are lightly protected, the DIY code in this post is genuinely all you need, and you should keep your money. Try the free tier and see which camp your targets fall into before committing more time. For a broader picture of where this fits, the guide to building an AI web research agent shows the API as one piece of a larger pipeline.

Frequently Asked Questions

Why do I get a 403 when scraping but not in my browser?

Your browser sends dozens of headers that requests and httpx don't, including User-Agent, Accept, Accept-Language, and the Sec-Fetch-* family. Sites also check whether your IP is residential or a known data-center range. Your browser runs on a residential ISP connection; your scraper probably runs from a cloud VM with a flagged IP. Add realistic headers first, and if that isn't enough, route through rotating residential proxies.

How do I fix 429 errors when web scraping?

Read the Retry-After header, which gives a delay in seconds or an HTTP date. If it's absent or invalid, use capped exponential backoff with jitter so parallel workers don't retry in sync. If the requested delay exceeds your wait budget, reschedule the work instead of retrying early. Lower your concurrency and pace requests per domain. The Python fetch_with_backoff and Node.js fetchWithBackoff examples implement the retry policy, and the TokenBucket throttle helps you avoid the 429 in the first place.

What is a 520 error in web scraping?

A 520 is a Cloudflare-specific code meaning the origin returned an unexpected response. In a scraping workflow, bot mitigation, TLS or HTTP fingerprint differences, IP reputation, and origin failures are all possibilities. Check the body, headers, timing, and browser comparison before choosing a fix. If the evidence points to a client fingerprint difference, curl_cffi or a real browser is a useful test; if maintaining that stack is not economical, consider managed scraping infrastructure.

How do I know if a 503 is a real outage or an anti-bot challenge?

Print the response body. A real 503 is usually short and generic. A challenge page contains text like "Checking your browser" or "Just a moment," and often a cf-mitigated header. The is_challenge_page helper above checks for these signals. Real outages get exponential backoff; challenge pages need a headless browser or a scraping API.

Is it legal to bypass these errors?

Bypassing anti-bot measures sits in a genuine gray area that depends on the site's terms of service, the data you're collecting, and your jurisdiction. Public data is generally treated more permissively than data behind a login, and personal data carries extra obligations under laws like GDPR. I'm a developer, not a lawyer, so the responsible answer is to review the target's terms and get legal advice before scraping anything sensitive or at scale.

How do I avoid getting blocked entirely?

The main levers are: look like a browser (realistic User-Agent, full header set, session cookies), pace yourself (rate-limit, add random delays, cap concurrency), use residential IPs through a rotating proxy service, and handle JavaScript-gated sites with a headless browser or a managed API. Do all four and you'll dodge the large majority of blocks. The guide to rotating proxies in web scraping is a good next read for the proxy side of that setup.

Wrapping Up

HTTP errors in web scraping aren't random, and once you stop treating them that way they get a lot easier to handle. A 429 usually points to request pacing, so respect Retry-After, reduce concurrency, and use capped backoff with jitter. A 403 more often points to identity, authorization, access policy, IP or geo rules, or bot mitigation, so inspect the body, headers, cookies, session, and controlled IP tests before choosing a fix. A 503 might be a transient overload or a JavaScript challenge, and a 520 remains ambiguous until the surrounding evidence identifies the cause. And a 200 isn't always a win, so validate the content you actually got.

The single most useful habit I can leave you with is the boring one: print the response body before you change anything. Most of the time, the site is telling you exactly what's wrong, and the fix is right there in the bytes. When the anti-bot stack is too aggressive to fight cost-effectively, Context.dev's scraping API is a clean escape hatch with no proxy management and no browser maintenance. Try it free and spend your time on the data instead of the delivery.