Web Scraping & Crawling
What is a proxy?
A server that forwards your network requests, presenting its own IP address to the destination instead of yours.
A proxy sits between your client and the internet. You make a request to the proxy, the proxy makes the same request to the target server, and the response flows back through the proxy to you. From the target's perspective, the request originated from the proxy's IP, not yours.
Proxies show up everywhere in modern infrastructure: corporate egress proxies for compliance, CDN edge proxies for caching, reverse proxies like nginx in front of an app server, and forward proxies for scraping. Each pattern differs in where the proxy lives relative to client and server, but the core mechanic is the same.
For web scraping, proxies are how you avoid getting one IP banned. Datacenter proxies are cheap and fast but trivially identifiable; residential proxies route through real consumer ISP IPs and look indistinguishable from regular users; mobile proxies route through cellular carrier IPs and are even harder to block. Pricing tracks that hierarchy.
In the wild
- →HTTP/HTTPS forward proxies via Squid for outbound traffic policy
- →SOCKS5 proxies for protocol-agnostic tunneling
- →Rotating residential proxies for avoiding scraping IP bans
How Brand.dev uses proxy
Endpoints in the Brand.dev API where this concept comes up directly.
FAQ
Proxy vs VPN, what's the difference?
A VPN encrypts and tunnels all of your device's traffic at the OS level. A proxy is configured per-app or per-request and usually only handles HTTP(S) or SOCKS traffic. VPNs are heavier and broader; proxies are surgical.
What is a rotating proxy?
A proxy pool that hands you a different exit IP on every request (or every N seconds). It is the standard way to spread scraping traffic across thousands of IPs without managing them yourself.
Are free proxies safe?
Generally not. Free proxies are slow, unreliable, and have a documented history of injecting ads, malware, or stealing credentials in transit. Use a paid provider for anything production.
Related terms
A proxy that routes your traffic through an IP address assigned by a consumer ISP, making your requests look like ordinary home users.
Programmatically extracting structured data from websites that were designed to be read by humans.
A server-side policy that caps how many requests a client can make in a given window, returning 429 Too Many Requests when the cap is exceeded.
A program that systematically follows links between web pages to discover and index content at scale.
The application protocol the web is built on, a simple request/response format for asking a server for a resource.