Debugging Web Scraper Failures: 403s, 429s, Timeouts, and Empty Results

TL;DR

403 Forbidden. The target may reject your headers, session, or IP reputation. Inspect the response, preserve cookies, use browser-like requests, and change IPs when needed.
429 Too Many Requests. Your scraper exceeds a rate or concurrency limit. Honor Retry-After, throttle requests, and retry with exponential backoff and jitter.
Timeouts. DNS, connection, or response delays stall the request. Identify the stalled stage, set separate timeouts, and retry temporary failures.
Empty or malformed results. JavaScript, login screens, or bot challenges can hide behind a 200 response. Inspect the body and render the page when necessary.
Silent redesign failures. Changed markup breaks selectors. Validate required fields and use semantic extraction.
Production reliability. You eventually need monitoring, retries, and validation, or a managed API that handles them.

Why scraping errors are hard to diagnose from the error alone

A scraper's visible error identifies a symptom, not the component that caused it. The same status code can come from the origin or an intermediary, while empty output can originate during page loading or extraction.

Start by reproducing the failed request with the same URL, headers, cookies, and network route. Then compare the raw response with the rendered page to separate transport problems from browser and parser problems.

Response signals narrow the search. Record the status code and redirects, then inspect headers, body content, and request timing. Logs should also preserve the proxy used, rendering settings, and extraction version. Each later checklist follows this sequence because the correct fix depends on which pipeline layer produced the symptom. This guide is the diagnostic companion to our status-code reference for HTTP errors in web scraping, which carries the fix code for each response.

403 Forbidden: blocked, not broken

An HTTP 403 Forbidden response means the server understood your request but refused it. First, inspect the response body and headers. Then test realistic headers, preserve cookies, reduce request frequency, and retry through a different IP. Those checks separate request formatting problems from session, rate, and IP blocks.

Diagnostic checklist

Read the body for a block reason, challenge page, or WAF rule identifier.
Inspect headers such as Server, CF-Ray, X-Cache, and x-amzn-RequestId to identify which service rejected the request.
Reproduce the request with curl -v, including the same authentication and headers.
Compare a bare request with one that sends a consistent User-Agent, Accept, Accept-Language, and Referer.
Test through another IP. If early requests succeed before later requests fail, lower your rate.

Fix

Send a complete and internally consistent browser header set. Use a persistent session that visits the home or login page before requesting protected pages. Rotate blocked IPs where you have permission, and add exponential backoff with jitter when failures appear after successful requests. Use browser rendering when the site requires JavaScript challenges or browser-created cookies.

Missing headers cause many immediate 403 responses because raw HTTP clients do not resemble normal browser traffic. Some sites also require a valid session cookie, CSRF token, or expected referrer. A persistent session preserves those values across requests, while isolated calls discard them. Our header and session walkthrough covers the code for both.

WAFs inspect more than visible headers. Cloudflare, DataDome, and Akamai can compare the TLS negotiation pattern, header order, JavaScript capability, request timing, and IP reputation. A request that claims to come from Chrome but uses a Python TLS signature can still trigger a block. Datacenter IP ranges and unexpected countries may also activate IP or geographic rules, which is why IP rotation and fingerprint matching have to move together.

Some infrastructure returns 403 after a rate threshold instead of returning 429. Treat a block that appears only after several successful requests as possible throttling. Slow the request rate, respect any retry guidance, and verify whether access returns after a cooldown.

A 403 can also represent a genuine permission decision. MDN's definition is explicit that re-authenticating makes no difference to a 403, because the refusal is tied to application logic rather than to missing credentials. If credentials, account permissions, or access policies exclude the resource, changing headers or IPs will not fix the request. For infrastructure you own, inspect WAF events, file permissions, server logs, and allowlists before changing scraper behavior.

429 Too Many Requests: your pace, not your access

An HTTP 429 Too Many Requests response means the target has temporarily throttled your scraper. Start with the response headers and your request logs.

Read Retry-After and pause for the specified number of seconds or until the specified HTTP date. Both formats are valid, so parse for either.
Log request timestamps, concurrency, endpoint, IP address, and credentials to identify which limit you crossed.
Compare X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset with your observed traffic.
Check whether several workers share one API key or IP address. Each worker may stay below its local limit while their combined traffic exceeds a shared quota.

Your retry handler should honor Retry-After first. When the server omits that header, use exponential backoff with jitter, such as waits near one, two, four, and eight seconds with a small random adjustment. Randomness prevents several throttled workers from retrying simultaneously and creating another traffic spike. AWS measured that effect directly and concluded that jittered backoff should be a standard approach for remote clients, because it cuts total call volume without extending completion time. Cap the retry count, queue requests at a controlled rate, and reduce concurrency before sending more work.

Servers calculate limits over different time models. A fixed window counts requests during a set interval and resets the count at the boundary, which lets a burst slip through around the reset. A sliding window counts requests over a continuously moving interval, which is why Cloudflare built its rate limiter on a sliding estimate rather than a resetting counter. A token bucket replenishes request tokens at a steady rate while allowing short bursts up to the bucket's capacity. Any of the three can reject bursty traffic even when your average request rate looks acceptable.

Rate limits may apply per IP address, credential, account, endpoint, or concurrent connection. Search endpoints may therefore return 429 while ordinary page requests continue working. Centralized throttling helps when several processes share the same quota because one limiter can track their combined usage.

Retry-After follows the HTTP standard, but X-RateLimit-* names remain conventions that providers may rename or omit. The IETF draft that standardizes these fields documents the problem plainly: the same header name carries different meanings across implementations, from seconds remaining to Unix timestamps to datetimes. Let the target's API documentation determine how you interpret reset times and quota headers. When the target supports batching or caching, use those features to reduce request volume before throttling begins.

Timeouts: where in the request the connection actually stalled

Use the request stage to identify which timeout you are hitting.

Run curl -v URL to see whether the request stalls during DNS lookup, TCP connection, TLS negotiation, or response reading.
Check the exit code. Curl code 28 is an operation timeout, while code 7 means curl resolved the host but could not establish a TCP connection. Codes 35, 52, and 56 indicate a TLS handshake failure, an empty reply, and a failure receiving data.
Run nc -zv host port to test the destination port. A long wait suggests dropped traffic, while an immediate refusal usually means no service is listening.
Run traceroute host or tracert host when failures affect one network or region. The route can reveal where packets stop, although individual hops may ignore probes.

Curl's write-out format turns that guesswork into numbers. This prints the elapsed time at each stage, so a slow DNS resolver looks nothing like a slow origin:

curl -sS -o /dev/null \
  --connect-timeout 5 --max-time 30 \
  -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  "https://example.com/products"

A DNS timeout occurs before your scraper obtains an IP address. Slow resolvers, broken records, and local network problems commonly cause it. Test another resolver, inspect the returned records, and cache successful lookups within their permitted lifetime.

A connect timeout occurs when DNS succeeds but the TCP handshake does not. Firewalls may silently drop packets, or an overloaded server may leave new connections waiting in a full queue. Set the connect timeout separately and use roughly three times the expected round-trip time as an initial value. Measure production traffic before finalizing it.

A read timeout occurs after the connection succeeds but before the server returns enough data. Slow application code can cause it, but your scraper can also exhaust its own connection pool, sockets, memory, or worker capacity. Compare timeout frequency with concurrency, pool utilization, and host resource usage before blaming the target site.

Set every timeout explicitly. Choose the read timeout from observed latency percentiles, such as p99.9 when you can accept a 0.1 percent false-timeout rate. Avoid extending it blindly because waiting requests continue occupying connections and workers.

Retry transient timeouts using the backoff and jitter approach described for 429 responses. Retry suitable 5xx responses, but do not retry other 4xx responses by default. A circuit breaker should pause requests after repeated failures so retries do not add load during an outage.

Empty or malformed results on a 200 OK

A 200 OK can be a fake success when the response body contains no usable target data. Run these checks before accepting the response.

Inspect the body, content type, and final URL. Look for CAPTCHA text, login prompts, consent screens, soft 404 messages, and unusually short HTML.
Compare the raw HTML with the page shown in a browser. If the browser contains data that the HTML lacks, enable JavaScript rendering and wait for a stable target element.
Capture a screenshot of the rendered response. A screenshot can expose challenge pages, region blocks, and interaction gates that never appear in status logs.
Validate required fields, data types, and minimum record counts. Reject the response when expected content is missing, even if the server returned 200.
Check whether the crawler followed hidden or off-screen elements. Remove those URLs from the crawl queue and restrict discovery to relevant links.

JavaScript-heavy sites often send a nearly empty HTML shell and load the useful data through later browser requests. A plain HTTP client downloads the shell before those requests run, so the parser receives no target content. You can query the page's internal JSON endpoint directly when one exists. Otherwise, use browser rendering and wait for a specific element rather than relying on a fixed delay. Our breakdown of how scraping APIs handle JavaScript-rendered content covers the wait strategies and the data that arrives through XHR or Fetch calls.

Anti-bot systems may return a challenge page with a 200 status instead of a 403. Your pipeline should classify the body before extraction because an LLM can otherwise summarize a CAPTCHA or index a navigation shell as source content. Body classification should detect known challenge phrases, missing required elements, and abnormal content length, and our guide to how CAPTCHA solvers work explains which challenges a rendered request can clear.

Honeypots create another path to empty results. Sites hide links or fields with CSS, and indiscriminate crawlers reveal themselves by interacting with them. Skip elements that are hidden, transparent, or positioned off-screen, and avoid queuing every link in the DOM. Selective link discovery reduces the chance of triggering these traps.

These checks address content missing when the request runs. If the expected content reaches the scraper but existing selectors no longer find it, investigate site redesign and DOM drift instead.

Silent failures after a site redesign

Catch selector drift by validating extracted data instead of relying on HTTP status codes.

Reject records when required fields are null, empty, or outside expected formats and ranges.
Alert when field completeness, record counts, or value distributions change sharply.
Fingerprint important elements using nearby text and relative position rather than one CSS selector.
Run scheduled canary extractions against known pages, and compare each result with a stored good sample.
Keep page snapshots so an alert shows which markup changed.

A schema is the cheapest version of that first check, and it belongs between extraction and storage rather than in a dashboard you read later:

import logging
from pydantic import BaseModel, Field, ValidationError
 
class Product(BaseModel):
    name: str = Field(min_length=1)
    price: float = Field(gt=0)
    currency: str = Field(pattern=r"^[A-Z]{3}$")
    in_stock: bool
 
def accept(records: list[dict], expected_minimum: int) -> list[Product]:
    valid = []
    for record in records:
        try:
            valid.append(Product(**record))
        except ValidationError as error:
            logging.warning("rejected record: %s", error.errors())
    if len(valid) < expected_minimum:
        raise RuntimeError(f"{len(valid)} valid records, expected {expected_minimum} or more")
    return valid

Site redesigns break scrapers when extraction logic depends on a specific DOM path. CSS-in-JS tooling can generate class names that change during deployment, while A/B tests may serve different layouts across requests. A redesign can also move visible data into a different HTML structure without changing what a user sees. Each case leaves the page available but causes a selector to return null or capture the wrong element. Playwright's own guidance says the same thing about test automation: selectors tied to DOM structure break when the DOM changes, so user-visible attributes make a more durable anchor.

Resilient extraction identifies fields by their meaning and expected schema. For example, a product extractor can require a name, price, currency, and availability value while allowing their HTML tags and positions to change. You can first isolate a stable container such as main, then map its rendered content into a strict JSON schema.

A hybrid strategy keeps routine requests inexpensive. Run CSS selectors first, validate the output, and send failed records to semantic extraction. Element fingerprints can also select a fallback when the primary path disappears.

Auto-healing selectors can reduce manual repairs, but they need controls. When a selector fails, an LLM or matching algorithm can inspect the current DOM and propose a replacement. Your pipeline should test that replacement against known samples before saving it, since an unchecked repair may capture plausible but incorrect data.

Catching failures before they reach production: monitoring, alerting, and retry design

A production scraper needs to detect missing or corrupt data before downstream systems consume it. One data team discovered that its scraper had silently stopped collecting from two government portals only after an analyst noticed low record counts. The team had already lost three days of data.

Your monitoring stack should track throughput and freshness for every source. It should also measure error rates by category and latency at each processing stage. Historical baselines help distinguish normal traffic changes from sudden record drops, retry spikes, or stale datasets. Validate at the source, the transformation, and the destination, because each stage can fail without the others noticing.

Schema validation should reject unexpected field types, missing required values, and malformed records. Content checks should also flag valid JSON that contains empty titles, implausible prices, or record counts far below the usual range. These checks catch selector drift and challenge pages that still return HTTP 200.

Structured logs should attach a correlation ID to every request and carry it through fetching, parsing, retries, and storage. The same ID lets you reconstruct one failed job across workers and services without searching unrelated log entries.

Alerts should use severity tiers based on user impact and data loss. A temporary retry increase can create a warning, while stale production data can page an operator. Each alert should include the affected source, time window, request count, and recent error distribution. Google's SRE book draws the line that matters here: spend your effort catching symptoms rather than causes, and make every page actionable, so a retry blip never wakes anyone at 3am.

Retry workers should apply exponential backoff with jitter and enforce a retry limit. Circuit breakers should pause requests when repeated failures indicate a source-wide problem. Failed jobs then need a durable queue for later replay rather than silent deletion.

Build vs. buy: in-house resilience stack vs. a managed scraping API

A DIY stack gives you maximum control, but you must build and maintain retry queues, browser workers, proxy rotation, schema validation, and alerts. Managed APIs move those responsibilities into vendor infrastructure. The table rates available capability, while setup overhead reflects the engineering effort required.

Approach	Retry handling	JS rendering	Structured output	Setup overhead
DIY monitoring stack	High	High	Custom	High
Apify	High	High	Medium	Medium
Firecrawl	High	High	High	Medium
ScrapingBee	High	High	Medium	Low
Bright Data	High	High	Medium	High
Zyte	High	High	Medium	High
Context.dev	High	High	High	Low

Apify suits projects that benefit from its broad Actor marketplace. Firecrawl serves Markdown-oriented RAG workflows well. ScrapingBee offers strong proxy and anti-bot handling, while Bright Data and Zyte provide enterprise-grade access for difficult targets. Their product choices and configuration requirements can add work compared with a unified API.

Context.dev owns request handling, rendering, and extraction behind one API. Managed request handling reduces HTTP 403 Forbidden failures caused by fingerprints, sessions, or blocked IPs. Automatic retry and pacing logic handles HTTP 429 Too Many Requests responses and transient timeouts. Browser rendering addresses empty or malformed results caused by JavaScript shells, and one call returns clean Markdown or structured JSON. Structured extraction reduces dependence on brittle selectors, while Context.dev Monitors evaluates site changes to catch silent failures after redesigns. Our comparison of DIY change detection against a monitoring API covers where that line falls.

A DIY stack remains suitable when you need full control over request behavior or unusual extraction logic. Context.dev fits AI pipelines that need clean JSON or Markdown without maintaining crawler infrastructure, including full-site crawls where the queue and pagination are someone else's problem.

FAQs

Is 403 or 429 worse for a scraper?

A 403 usually signals denied access, while a 429 signals temporary rate limiting. Context.dev manages request handling behind one API. You avoid diagnosing many access and pacing failures yourself.

Should I retry after a 403?

A scraper should not retry a 403 immediately because unchanged requests will probably fail again. Context.dev handles retries and rendering within its managed scraping infrastructure. You avoid retry loops that increase traffic without fixing the block.

How many retries should a scraper attempt before giving up?

Most scrapers should attempt two or three retries for temporary failures before sending the request to a failed-job queue. Context.dev manages retry behavior for supported requests. A retry limit prevents one unavailable target from consuming workers indefinitely.

Does rotating IPs fix HTTP 429 Too Many Requests?

IP rotation can help when a website applies limits per IP, but it cannot bypass limits tied to an account, API key, or global quota. Context.dev manages request delivery without requiring you to maintain proxy rotation. You can focus on the returned data instead of proxy health and IP reputation.

Closing takeaway

Most recurring scraping failures expose missing pipeline controls rather than isolated bugs. Production reliability comes from request pacing, retries, rendering, validation, and monitoring that work together.

You can build and maintain those controls internally or buy them as managed infrastructure. Context.dev provides a single API for crawling and structured extraction, with no browser or retry infrastructure for your team to maintain.