TL;DR
- Anti-bot systems combine signals into a composite trust score, then allow, challenge, or block each request.
- Four detection layers shape that score. Sites inspect IP reputation and ASN, TLS and HTTP fingerprints, headless browser fingerprints, and behavioral signals such as timing and session consistency.
- Reliable scraping matches each weak layer with a specific countermeasure, including proxy rotation, browser-compatible handshakes, stealth patches, and realistic pacing. Teams that do not want to maintain those controls can use Context.dev to handle proxies, fingerprint management, and headless rendering through one managed API.
Why scrapers get blocked: the trust-scoring model
Anti-bot systems usually block scrapers by combining weak signals into a composite trust score. A suspicious IP lowers the score, but a mismatched TLS fingerprint, headless browser tell, or mechanical request pattern can lower it further. Some systems reportedly evaluate 40 to 60 browser signals rather than relying on one bot indicator.
Five input stages feed the risk engine. Network checks examine the IP address, ASN, location, and request rate. TLS and HTTP checks compare the connection handshake and protocol behavior with known browser profiles. Browser checks inspect JavaScript properties and hardware fingerprints. Behavioral checks evaluate timing, scrolling, mouse movement, and session consistency. Historical reputation connects the current request with earlier activity from the same IP, fingerprint, account, or cookie.
The risk engine updates its score as those signals arrive, then allows, challenges, or blocks the request. A CAPTCHA or JavaScript challenge usually reacts to a low score. It rarely serves as the first detection step.
Practical countermeasures map to four controllable layers. You manage network identity with suitable proxies and rotation. You match browser transport behavior at the TLS and HTTP layer. You remove headless browser tells while keeping fingerprints coherent. You pace requests and preserve believable session behavior. Historical reputation then reflects how consistently those four layers behave over time.
Layer 1: IP reputation and ASN checks
Anti-bot systems score each IP by its ownership, request history, location, and traffic rate. An autonomous system number, or ASN, identifies the network operator behind an address. Cloud hosting ASNs are easy to classify, while repeated requests from one IP can damage its reputation regardless of who owns it. A poor network score commonly produces a 403 Forbidden or 429 Too Many Requests response.
Proxy choice determines how closely your network identity resembles normal traffic. Datacenter proxies originate from hosting providers, so sites can identify their address ranges and apply stricter limits. Residential proxies route through consumer internet connections and inherit the stronger reputation of household IPs. ISP proxies use addresses registered to consumer providers but host them on server infrastructure, which provides residential-style ownership with longer, more stable sessions.
Proxy rotation limits the request volume and history attached to any single address. Keep one IP for a session when the target expects consistent cookies, location, or login state, then rotate between sessions. For stateless page requests, rotate more often while capping concurrency per IP. A larger pool reduces reuse, but rotation cannot repair an address range that the target already distrusts.
Treat a 429 as a likely pacing or per-IP quota problem. Treat an immediate 403 across many fresh IPs as evidence that another layer, such as TLS fingerprinting, may also be lowering the request score.
Proxy types compared: residential vs. datacenter vs. ISP
| Proxy type | Typical cost | IP reputation strength | Best use case |
|---|---|---|---|
| Residential | $5 to $15 per GB | High because traffic uses consumer connections | Protected sites and location-specific scraping |
| Datacenter | $1 to $3 per proxy monthly | Low because hosting-provider ranges are easy to classify | High-volume scraping on lightly protected sites |
| ISP | Provider-specific, usually priced per IP | Medium to high because consumer ISPs register the addresses | Stable sessions that need residential reputation |
Your rotation cadence should follow the target and session type. Keep one IP for a login, checkout, or multi-page visit so cookies and network identity remain consistent. For independent page requests, rotate before any address accumulates a suspicious request rate. Larger pools distribute traffic more safely, but rapid rotation cannot compensate for aggressive concurrency or inconsistent browser fingerprints.
Layer 2: TLS/JA3 and HTTP/2 fingerprinting
TLS fingerprinting can reveal a scraper before the server receives its HTTP request. During the TLS handshake, the client sends a ClientHello containing supported cipher suites, extensions, and their ordering. Anti-bot systems convert those attributes into a JA3 or JA4 fingerprint and compare it with the browser claimed by the User-Agent header. Python requests produces a different TLS signature than Chrome, so copying Chrome headers cannot make the connection look like Chrome. The mismatch often explains why a page works in a browser but returns 403 in code.
HTTP/2 gives the server another transport fingerprint. Chrome sends characteristic SETTINGS values, arranges frames in a particular order, and negotiates protocols in predictable ways. Libraries such as httpx can produce different patterns. A credible request must therefore match both the advertised browser and its network behavior.
Tools such as curl_cffi and curl-impersonate reproduce browser TLS and HTTP/2 profiles without running a full browser. They work well when the target serves content directly and checks the connection fingerprint. However, these tools cannot execute JavaScript challenges or produce browser signals such as canvas and WebGL output. JavaScript-heavy targets still require a real browser, a fortified headless browser, or a managed scraping API.
Layer 3: Headless browser and fingerprint tells
Headless browser detection starts with properties that automation frameworks expose. Chrome launched through WebDriver commonly sets navigator.webdriver to true. Headless sessions may also report no plugins, omit language settings, or lack browser objects that normal Chrome sessions expose. Some anti-bot products inspect Chrome DevTools Protocol activity, including calls that ordinary browsing does not generate.
Websites also compare hardware-related fingerprints across JavaScript APIs. Canvas and WebGL rendering can reveal cloud virtual machines through their graphics drivers, while audio output can identify combinations of operating system and hardware. Screen resolution, fonts, and permissions add supporting signals to the same browser profile. Cloud rendering often produces recognizable Canvas and WebGL outputs.
Stealth tools patch the most common browser tells. A basic patch uses Object.defineProperty to override navigator.webdriver with a getter that returns false. playwright-stealth and puppeteer-extra-plugin-stealth patch several properties automatically. Camoufox and nodriver take different approaches to reducing automation signals, while SeleniumBase offers UC mode for Chrome automation.
A coherent fingerprint works better than broad randomization. A session that claims macOS in its user agent should not expose Windows fonts or a conflicting WebGL renderer. Stable values across repeat visits also look more plausible than a new canvas hash on every page load because detection systems can evaluate identity over time. Frequent changes can make a browser profile less credible.
Self-managed stealth requires continuous testing because anti-bot vendors add checks and browser releases alter exposed behavior. Undetected-chromedriver illustrates that maintenance treadmill. It remains available, but many developers have moved to newer options as its older patches have become easier to detect.
Layer 4: Behavioral signals and pacing
Behavioral detectors flag sessions whose actions look mechanically uniform. They measure mouse-path curvature, click timing, and scrolling pauses. Real users vary their speed and often stop while reading, while basic automation moves directly to coordinates and scrolls continuously. Sites also place hidden links or fields as honeypots, which bots may activate while parsing every element.
Request pacing should vary within clear per-site limits. Cap concurrency by hostname, add jitter between requests, and back off after errors instead of retrying immediately. For browser automation, ghost-cursor and randomized delays of 300 to 2,000 milliseconds can produce curved mouse paths and less uniform timing. Staged scrolling should include pauses, and clicks should target visible rendered elements rather than every link found in the DOM.
Session consistency still controls whether those actions appear credible. Keep cookies and the browser fingerprint stable during one visit. Avoid changing the user agent or IP region halfway through a navigation sequence. A fresh identity on every request breaks the continuity that normal browsing creates, even when each individual request looks plausible.
Detection layers vs. countermeasures at a glance
| Detection layer | Primary countermeasure | Typical cost or effort |
|---|---|---|
| IP reputation and ASN | Rotate residential, ISP, or datacenter proxies based on target difficulty | Residential proxies often cost $5 to $15 per GB |
| TLS and HTTP/2 fingerprint | Match browser handshakes with curl_cffi or curl-impersonate | Low to medium setup and version upkeep |
| Headless browser fingerprint | Use stealth tooling and maintain a coherent browser profile | High recurring testing and patch maintenance |
| Behavioral signals | Pace requests, preserve sessions, and simulate plausible interaction timing | Medium ongoing tuning by target |
Start with the layer producing the weak signal. Harder targets often require coherent countermeasures across all four layers because anti-bot systems combine their observations into one trust score.
CAPTCHA solving: when a challenge is unavoidable
A CAPTCHA usually appears after weak IP, browser, or behavioral signals lower the request’s trust score. Solving the challenge may recover one request, but it does not correct the signals that triggered it.
2Captcha, CapSolver, and CapMonster provide APIs that submit challenges and return solution tokens. Published estimates put solving costs around $1 to $3 per 1,000 attempts, with 10 to 60 seconds of added latency and accuracy ranging from 60 to 95 percent. Challenge type, traffic volume, and provider capacity affect the actual figures, so test each service against your target sites.
A solving API also creates another integration to monitor, fund, and retry alongside your proxies and browser infrastructure. Managed scraping APIs can handle challenges within the scraping request, which suits AI pipelines that do not need direct control over the solver.
Troubleshooting 403, 429, and 503 errors
Status codes provide clues, not proof. Anti-bot platforms can return the same code for different failures, so change one variable per test and compare the response body, headers, and cookies.
| Code | Likely detection layer | Diagnostic next step |
|---|---|---|
| 403 Forbidden | IP or ASN reputation, TLS and HTTP fingerprinting, or browser fingerprinting can produce a 403. | Retry through a clean IP, then retry with a real browser on the original IP. An IP-only fix suggests reputation trouble. A browser-only fix suggests a transport or browser fingerprint mismatch. |
| 429 Too Many Requests | Per-IP rate limits, concurrency, or repetitive timing usually triggered the response. | Check the Retry-After header and request logs. Run a slower test with lower concurrency while keeping the IP and fingerprint unchanged. |
| 503 Service Unavailable | A JavaScript challenge or low browser and behavioral trust may have intercepted the request. The origin server could also be overloaded. | Inspect the response HTML for CAPTCHA, Turnstile, or challenge scripts. Compare the page with a normal browser and check whether cookies or JavaScript redirects complete the request. |
Repeated tests should preserve cookies and session identity. Rotating every variable at once can hide the failing layer and create new inconsistencies.
Vendor difficulty: Cloudflare, Akamai, DataDome, Kasada, and PerimeterX
Anti-bot difficulty depends on the vendor configuration, but the typical difficulty mapping shows how quickly isolated fixes stop working.
| Vendor | Typical difficulty | Layers that usually need attention |
|---|---|---|
| Cloudflare | Medium | IP reputation, browser fingerprint, and challenge handling |
| Akamai | High | Residential IP, TLS and HTTP profile, and real browser execution |
| DataDome | High | IP, transport, browser fingerprint, and behavior |
| Kasada | Very high | All four layers plus consistent challenge execution |
| PerimeterX or HUMAN | High | Browser fingerprint, realistic behavior, and stable network identity |
Cloudflare deployments often yield to a reputable proxy and fortified browser, although individual site configurations vary. Akamai can reject a request when its TLS profile conflicts with its claimed browser, even if the IP looks acceptable.
DataDome, Kasada, and PerimeterX commonly evaluate signals across the full session. A residential IP cannot compensate for an implausible canvas fingerprint or mechanical request timing. On these targets, every layer must describe the same browser, device, location, and user session. Managed scraping services become practical when maintaining that cross-layer consistency costs more than the extraction work itself.
Self-managed stack vs. managed API: what it actually costs
True cost = advertised cost × retry and failure overhead + engineering and QA hours. A proxy rate or API price says little unless you compare successful, usable pages. For example, a 50 percent success rate roughly doubles bandwidth and compute cost before you count debugging time.
Self-hosted browser fleets add substantial resource costs. Each headless Chrome instance can consume roughly 200 to 500 MB of RAM, so 100 concurrent sessions may require 20 to 50 GB before orchestration overhead. Residential proxies often cost $5 to $15 per GB, and failed requests still consume bandwidth. Engineers must also update stealth patches, test fingerprint consistency, manage sessions, and investigate target-specific failures.
Managed APIs shift those variable costs into a per-request or per-result fee. The unit price may exceed raw proxy pricing, but the comparison should include successful delivery rates and maintenance labor. Bright Data supports enterprise programs that need broad proxy and scraping infrastructure, though its product split requires an upfront architecture choice. Zyte offers enterprise unblocking for teams prepared to integrate and operate a more involved scraping stack. Both remain legitimate choices when you need infrastructure control and can support the recurring engineering work.
Where Context.dev fits: one API instead of four systems to maintain
Context.dev fits AI engineering teams that need reliable web data without operating an anti-bot stack. One managed API call handles proxy rotation, transport and browser fingerprints, headless rendering, and request pacing. Context.dev then returns clean JSON or LLM-ready Markdown instead of raw browser output.
Dedicated proxy networks and anti-detect browsers remain valid choices when you need direct control over IP pools, fingerprint profiles, and browser execution. That control requires you to monitor block rates, update stealth patches, manage retries, and keep session behavior consistent as detection systems change.
Context.dev moves that maintenance server-side. Your AI agent can request current web content through REST or MCP without coordinating separate proxy, browser, CAPTCHA, and extraction services. The managed approach suits teams that would rather ship agent features than re-patch stealth tooling each month.
FAQs
Is web scraping legal?
Web scraping is generally legal when you access public data without bypassing access controls or violating applicable laws. Site terms, copyright, privacy rules, and account restrictions can still limit collection and reuse. Consult counsel for regulated, personal, or commercially sensitive data.
How do I know which detection layer blocked me?
Status codes provide an initial clue, but they rarely identify one layer conclusively. A 429 often points to request rate, while a 403 can indicate IP reputation or fingerprint mismatch. A 503 or challenge page usually points to browser or behavioral scoring.
Do I need residential proxies for every site?
Residential proxies use consumer ISP addresses that usually carry stronger reputations than datacenter addresses. Public pages with light defenses may work through datacenter proxies or direct requests. Test the least expensive option before moving sensitive targets to residential pools.
Can rotating fingerprints too often cause blocks?
Frequent fingerprint rotation can make one session appear internally inconsistent. Anti-bot systems may compare cookies, IP location, timezone, headers, and browser properties over time. Keep one coherent identity per session, and rotate between sessions when needed.
How much does anti-bot bypass cost at scale?
Costs depend on bandwidth, browser compute, retries, CAPTCHA volume, and engineering time. Residential proxies commonly cost $5 to $15 per GB, while headless browsers add memory and operational overhead. Measure cost per successful page rather than cost per request.
Can Context.dev replace my proxy and stealth stack?
Context.dev provides managed scraping, browser rendering, proxy handling, and structured delivery through one API. AI agents and LLM pipelines can receive clean JSON or Markdown without maintaining fingerprint patches. Dedicated proxy and anti-detect tools remain better when you need direct control over browser identities.