TL;DR
Staying unblocked now costs more engineering time than parsing HTML or picking a framework. Once your scraper hits a hardened target, the fight is against detection, not extraction.
- Rotating IP addresses no longer beats modern anti-bot stacks, because vendors correlate IP reputation with TLS fingerprint, header order, and behavior.
- TLS/JA3 fingerprinting matters as much as IP reputation. A default
requestshandshake looks nothing like Chrome, and detection vendors flag that instantly. - DIY CAPTCHA solving with computer vision rarely pays off past small scale, since challenge formats change without notice and break your solver.
- A rising CAPTCHA rate signals a fingerprint or behavior problem upstream, not a solving problem to throw money at.
- Once maintenance hours cross a threshold, a managed API becomes the cheaper option than owning the arms race yourself.
Why proxy rotation alone stopped working
Rotating IPs used to be enough because detection ran mostly at the network layer. A site checked whether an IP belonged to a known datacenter range or had already tripped a rate limit, and a fresh proxy reset that check. That era is over. Modern anti-bot vendors like Cloudflare and Akamai correlate signals across four layers at once, so a clean IP paired with a Python default handshake gets flagged faster now than a dirty IP did in 2023.
The detection stack breaks into four layers, and every technique in this article maps onto one of them. At the network layer, vendors score IP reputation, ASN, and request rate per subnet. At the TLS layer, they fingerprint your client's handshake through JA3 and its successors, and the requests library produces a handshake that looks nothing like Chrome or Firefox. At the browser fingerprint layer, they inspect header order, header casing, and JavaScript-exposed properties like navigator.webdriver, canvas output, and WebGL details. At the behavioral layer, they watch timing, mouse movement, and scroll patterns to separate a script from a person.
The reason proxy rotation alone stopped working is correlation across those layers. When you rotate the IP but keep the same TLS fingerprint and header order on every request, you hand the vendor a stable identifier that survives the IP change. Ten proxies sending an identical Python fingerprint read as one bot behind ten addresses, not ten users. The vendor bans the fingerprint, and your whole pool dies together.
That layered model is why the rest of this guide is organized around a single diagnostic question. When your scraper gets blocked, you need to know which layer you are failing at, because the fix for a TLS mismatch is useless against a behavioral flag, and vice versa.
Rotating proxies in requests, httpx, and aiohttp
In requests, the cleanest pattern keeps one Session per proxy and rotates at the session level rather than swapping proxies mid-connection. You maintain a pool of live proxies, pull one per host or per batch, and mark it unhealthy when it returns a 403 or 429. A dictionary mapping proxy URLs to a failure count and a cooldown timestamp is usually enough. Once a proxy crosses a threshold, drop it from rotation for a few minutes instead of retrying it immediately, since correlated bans mean a burned proxy stays burned for a while.
Two rotation strategies dominate, and they solve different problems. Sticky sessions bind one proxy to one target host for the life of a crawl, which matters when the site tracks session cookies or expects IP continuity across a login flow. Random rotation picks a fresh proxy per request, which spreads load and dilutes any single IP's request rate. Use sticky sessions for authenticated or stateful targets, and random rotation for stateless, high-volume fetching where no single request depends on the previous one.
httpx changes the calculus because it supports async clients, and concurrency turns proxy selection into a contention problem. If you launch two hundred coroutines against a fifty-proxy pool, four requests share each IP simultaneously, and that burst pattern reads as automated far more clearly than a human browser would. Bound your concurrency to your pool size with an asyncio.Semaphore, and treat the semaphore limit as a function of how many proxies you actually have, not how many coroutines your event loop can hold. An async AsyncClient per proxy, reused across requests, avoids the handshake cost of rebuilding connections on every call.
aiohttp follows the same async logic but pushes proxy configuration down to the request level through the proxy= argument on session.get(), so a single ClientSession can fan out across your whole pool. Track proxy health in a shared structure guarded against concurrent writes, and back off with exponential delay on repeated 429s rather than hammering the next proxy in line. Retrying instantly across a fresh IP just spends your pool faster.
None of this survives contact with real scale without honest accounting. Datacenter proxies are cheap and get flagged early because their subnets are known, so a single ban often takes a whole /24 block with it. Residential proxies dodge that but cost ten to thirty times more per gigabyte, and heavy concurrent use burns through a pool of a few thousand IPs in hours against a hardened target. Pool exhaustion is the real ceiling. You can rotate perfectly and still run out of clean IPs faster than they recover, at which point rotation logic stops being the bottleneck and proxy sourcing becomes the recurring cost you actually manage.
Scrapy middleware for proxy rotation
Scrapy's asynchronous engine makes it a natural fit for high-throughput proxy rotation, but you still need middleware to wire a proxy pool into the request lifecycle. Two libraries dominate. scrapy-rotating-proxies rotates through a list you supply and tracks which proxies are alive by watching for failures, retrying dead ones after a cooldown. scrapy-proxy-pool leans on public proxy sources and refreshes its list automatically, which sounds convenient but pulls in free proxies whose reliability you cannot control.
Choose scrapy-rotating-proxies when you already have a paid proxy list and want the middleware to handle health tracking and rotation around it. It stays actively maintained and gives you knobs for ban detection and per-proxy backoff. scrapy-proxy-pool suits throwaway experiments and low-stakes crawls where free proxies are acceptable, but its dependence on scraped public lists means you inherit dead endpoints and shared bans that other users triggered before you.
A minimal setup for scrapy-rotating-proxies lives entirely in settings.py.
ROTATING_PROXY_LIST = [
'http://user:pass@proxy1:8000',
'http://user:pass@proxy2:8031',
]
DOWNLOADER_MIDDLEWARES = {
'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
}
ROTATING_PROXY_PAGE_RETRY_TIMES = 5The middleware priority numbers matter because they place proxy rotation ahead of Scrapy's built-in RetryMiddleware, which sits at 550 by default. When a request fails on a proxy, RotatingProxyMiddleware marks that proxy as bad and Scrapy's retry logic reissues the request, giving the rotation layer a fresh proxy on the next attempt. Set ROTATING_PROXY_PAGE_RETRY_TIMES deliberately, because a high value combined with RETRY_TIMES can multiply your request count against a hardened target and burn through a proxy pool fast.
The honest failure mode is that neither middleware sources proxies for you. Both handle the mechanics of picking a proxy, detecting a ban, and cycling to the next one, but the quality and reputation of the underlying IPs remain your problem. A rotating middleware with a pool of exhausted datacenter IPs fails just as reliably as no rotation at all, because the target already flagged that subnet. Proxy sourcing, buying residential IPs, monitoring their reputation, and replacing burned ones is a recurring operational cost the middleware never touches.
CAPTCHA handling: solving services vs. DIY
A CAPTCHA is a symptom, not the disease. When your scraper hits a reCAPTCHA or hCaptcha challenge, the target's anti-bot vendor has already decided your traffic looks automated. Solving that challenge lets one request through, but it does nothing about the fingerprint or behavior that triggered the challenge in the first place. Treat every CAPTCHA as a signal about an upstream detection problem, and you will spend your engineering time where it actually pays off.
Third-party solving services like 2Captcha and Anti-Captcha route challenges to human workers or ML solvers and return a token you inject back into the page. They earn their cost when your CAPTCHA rate is low and unavoidable, and you need a request to complete right now without stopping to re-engineer your stack. You pay per solve, latency runs several seconds per challenge, and neither number improves as you scale. At a few hundred solves a day, that tradeoff is reasonable. At tens of thousands, the bill and the added latency make it a tax on a problem you should have fixed elsewhere.
Building your own computer-vision solver for reCAPTCHA or hCaptcha is a maintenance trap, and I would avoid it entirely. Google and hCaptcha change challenge formats, image sets, and scoring logic without notice, and every change breaks a DIY solver until someone retrains or rewrites it. You end up staffing a permanent research project to keep pace with two companies whose full-time job is defeating exactly what you are building. The solving services exist because that arms race is not worth fighting alone.
Here is the decision rule. Solve reactively with a paid service when your CAPTCHA rate is low, when the volume is small enough that per-solve cost stays negligible, and when the alternative is blocking a pipeline you need running today. When your CAPTCHA rate starts climbing, stop buying solves and fix the cause. A rising rate almost always means your TLS fingerprint, header ordering, or request timing is exposing you as automated, and the next section covers exactly those tells. Cheaper fingerprint work upstream eliminates challenges you would otherwise pay to solve one at a time, so measure your CAPTCHA rate as a leading indicator rather than a line item.
Anti-bot fingerprinting: TLS/JA3, headers, and headless detection
Modern anti-bot vendors identify scrapers before they ever look at your IP reputation, and the tell starts with your TLS handshake. When a Python client opens an HTTPS connection, it negotiates a specific set of cipher suites, extensions, and elliptic curves in a specific order. Cloudflare, Akamai-style stacks, and PerimeterX-style systems hash that handshake into a JA3 fingerprint. A default requests or httpx connection produces a JA3 that no real Chrome or Firefox build has ever generated, so the request is flagged as automated before a single header is parsed.
Header ordering and casing
Real browsers send headers in a consistent order with consistent casing, and detection systems check that order against known browser profiles. A hand-built header dictionary in Python preserves whatever order you wrote it in, which rarely matches Chrome's actual sequence. Even when you copy the exact header values, mismatched ordering or lowercase casing on headers a browser sends capitalized gives you away. The User-Agent claiming Chrome while the TLS fingerprint says Python is the contradiction these systems are built to catch.
Headless browser tells
Switching to a headless browser solves the TLS problem and introduces a new set of signals. navigator.webdriver returns true under automation unless you patch it. Headless Chrome ships with missing or inconsistent plugin lists, and its canvas and WebGL rendering produces fingerprints that cluster differently from real GPUs. Detection scripts also measure how fast JavaScript executes on the page, since automation environments often run challenges faster or slower than a human's browser would. Tools like undetected-chromedriver and Playwright with stealth patches close many of these gaps, but each patch targets a specific check a vendor already knows about.
Timing and behavioral signals
Beyond static fingerprints, vendors watch how requests arrive over time. A scraper firing requests at exact one-second intervals, or hitting fifty pages with zero mouse movement, looks nothing like a person browsing. For headless-browser work, convincing evasion means randomized delays, simulated scrolling, and mouse paths that curve instead of jumping in straight lines. These behavioral layers are harder to fake than a header list, because they require modeling what a human session actually looks like rather than copying a fixed value.
Spoofing every layer at once is possible, and libraries like curl_cffi and tls-client let you match a real browser's JA3 from Python without a full browser. The honest problem is durability. Detection vendors update their fingerprint databases and add new checks on their own schedule, and a fingerprint that passed last month starts failing without any change on your end. Keeping a spoofed client convincing means tracking what Cloudflare and its peers change, then re-patching your handshake, headers, and behavior to match. That tracking never ends, which is the real cost most teams underestimate when they decide to own this in-house.
Where this breaks down at scale
Every technique in this article shares one weakness. Better proxies, TLS impersonation libraries, CAPTCHA services, and fingerprint spoofing all work until the target site's detection vendor pushes an update, and then you start over. Cloudflare, Akamai, and PerimeterX ship changes on their schedule, not yours, so a scraper that ran clean for months can start returning 403s overnight with no code change on your end. You aren't solving a problem once. You're maintaining a position against an opponent that keeps moving.
The costs compound in three directions at once. Proxy spend scales with request volume, and residential pools that beat modern detection run far more expensive than the datacenter IPs that no longer work. Engineer-hours disappear into re-patching JA3 fingerprints and header ordering each time a vendor tightens its checks, and that work never converts into a finished feature. A CAPTCHA solving service that costs a few dollars a month at pilot scale turns into a real line item once you hit hundreds of thousands of challenges.
"It works today" is the trap. A scraper passing every check in your test run tells you nothing about next Tuesday, because the detection logic sits on a server you don't control and changes without notice. Teams that treat unblocking as a one-time engineering task budget for it once and then absorb the recurring cost as a series of emergencies.
At low volume against soft targets, owning this stack is defensible and cheap. Past a certain scale, or against sites behind hardened anti-bot vendors, the arithmetic changes. When the ongoing maintenance cost exceeds what a managed API would charge, running your own proxy pools and fingerprint patches stops being the frugal choice and becomes the expensive one.
Comparison: proxy/anti-bot approaches by use case
Each approach below wins on a different axis. Pick based on your scraping volume and how hardened your target sites are, not on which one sounds most complete.
| Approach | Best for | Setup effort | Maintenance burden | Anti-bot resilience | Cost model |
|---|---|---|---|---|---|
| Self-managed proxy rotation (requests, httpx, aiohttp) | Low-volume scraping against simple targets you fully control | Low | High. You own pool sourcing, health checks, and retry logic | Weak alone. Rotating IPs does not defeat TLS or fingerprint checks | Proxy subscription plus your engineering time |
| Scrapy middleware (scrapy-rotating-proxies, scrapy-proxy-pool) | Structured crawls at medium scale already running on Scrapy | Medium | Medium. Middleware handles rotation, not proxy quality | Moderate when paired with TLS impersonation and good proxies | Proxy subscription plus middleware upkeep |
| CAPTCHA solving services (2Captcha, Anti-Captcha) | Reactive solving at low CAPTCHA rates | Low | Low to run, but rising rates signal an upstream problem | None. Buys time without fixing what triggers challenges | Per-solve pricing that scales with block rate |
| Context.dev | Teams that want clean, LLM-ready output without owning anti-bot infrastructure | Very low. Single API call | Near zero. Proxies, rotation, and fingerprinting are managed | Strong. Handled and updated for you | Usage-based, no proxy pool to fund separately |
Self-managed rotation stays cheapest at small scale and gives you full control. Scrapy middleware fits teams already committed to the framework and willing to source proxies. CAPTCHA services patch symptoms, so treat a climbing solve bill as a fingerprinting problem in disguise. Context.dev suits teams that would rather stop maintaining the anti-bot stack entirely.
Context.dev as the managed alternative
Once your team spends more hours patching TLS fingerprints and rotating proxy pools than parsing the data you actually came for, a managed API becomes the cheaper option. Context.dev handles JavaScript rendering, proxy rotation, and anti-bot evasion behind a single endpoint, and it returns clean Markdown or structured JSON ready for an LLM pipeline. You send a URL, you get back content that a model can consume, and you never touch a JA3 signature or a CAPTCHA solver.
The practical benefit is that Context.dev owns the arms race so you don't. When Cloudflare or Akamai updates its detection stack, the re-patching happens on Context.dev's side, not in your codebase. Direct integration through MCP means an AI agent can pull live web content without you standing up a crawler, a proxy budget, or a retry layer. For teams whose actual product is the downstream analysis rather than the scraping infrastructure, that trade removes an entire category of on-call maintenance.
Self-hosting still wins in two cases, and pretending otherwise would be dishonest. At low volume against simple targets, a requests script with a small rotating proxy pool costs almost nothing and runs fine for months. If you need full control over request behavior, header order, and session logic for a specialized target, owning the stack gives you knobs a managed API deliberately hides. Scrapy middleware plus a CAPTCHA service remains a defensible middle ground when your volume is real but your targets aren't running the hardest commercial anti-bot stacks.
Where Context.dev earns its place is the replace-your-internal-anti-bot-maintenance case. If you already run a crawler that breaks every few weeks when a target site tightens detection, consolidating that work into one API removes the proxy contracts, the fingerprint tracking, and the retry logic from your roadmap. You can try it at context.dev and measure the swap against your current engineer-hours before committing to owning the cat-and-mouse game for another year.
Decision framework: when to self-host vs. go managed
Three variables decide whether you self-host or go managed: your request volume, how hardened your target sites are, and how many engineer-hours you can spend on maintenance every month. Run through them honestly before you commit to either path.
Self-host with requests or httpx plus a rotating proxy pool when you scrape low volume from sites that don't run a serious anti-bot vendor. If your targets return data with a rotated datacenter IP and a spoofed User-Agent, you don't need more machinery, and building it would waste money. A weekend of setup and a cheap proxy plan cover this case for years.
Reach for Scrapy middleware plus a CAPTCHA service in the mid-range, where you crawl at meaningful volume but your targets aren't running the newest fingerprinting stacks. Scrapy's retry and rotation middleware handle the mechanics, and a solving service absorbs the occasional challenge. You still own proxy sourcing and quality, so budget a few engineer-hours a month for the pool. That cost stays defensible as long as your CAPTCHA rate stays low.
Go managed once your targets sit behind Cloudflare, Akamai, or PerimeterX-style detection and your CAPTCHA rate keeps climbing no matter how you patch fingerprints. At that point you're paying an engineer to track detection-vendor changes indefinitely, and that salary line dwarfs an API subscription. A managed API absorbs TLS impersonation, header ordering, and headless evasion so you stop re-patching every time a vendor ships an update.
The threshold is the same one the arms race sets. When keeping your scrapers unblocked costs more engineering than the data is worth, you've crossed the line where owning the infrastructure no longer pays.
FAQs
Are free proxies ever viable for production scraping? Free proxy lists work for quick experiments, not for anything you run continuously. Public proxies share IPs across thousands of users, so most are already flagged by the target sites you care about, and their uptime is unpredictable. For production Python proxy rotation, pay for datacenter or residential proxies from a provider that lets you monitor pool health.
Do I always need residential proxies? No. Residential proxies cost far more than datacenter proxies, and many targets never inspect IP type closely enough to justify them. Start with datacenter proxies, watch your 403 and 429 rates, and only move to residential when a specific site blocks datacenter ranges outright.
How do I tell if a block is IP-based or fingerprint-based? Rotate to a fresh, clean IP and retry the same request. If the block clears, the trigger was IP reputation. If a brand-new IP still gets blocked immediately, your TLS fingerprint, header order, or headless tells are giving you away, and no amount of proxy rotation will fix it.
Is proxy rotation or CAPTCHA solving legal? Rotating proxies and using solving services are legal in most jurisdictions, but they can violate a site's terms of service, and scraping certain data raises separate legal questions. Consult counsel before scraping personal data, copyrighted content, or sites behind authentication.
How does Context.dev handle anti-bot without proxy management? Context.dev runs proxy rotation, TLS fingerprinting, and anti-bot evasion behind a single API, so you send a URL and receive clean structured output. You get JavaScript rendering and LLM-ready Markdown or JSON without owning proxy pools or patching fingerprints yourself. Teams building AI pipelines use it to replace internal anti-bot maintenance with one managed endpoint.