HTTP 429 Too Many Requests: What It Means and How to Fix It
HTTP 429 Too Many Requests is a status code meaning the client has sent too many requests in a given amount of time, a condition usually called rate limiting. Defined in RFC 6585, the response may include a Retry-After header that says how long to wait, and caches must not store it. For scrapers, it is the clearest signal to slow down.
- Code
- 429
- Name
- Too Many Requests
- Class
- 4xx client error
- Retry?
- Yes, with backoff
What causes a 429 error?
- →More requests per second, minute, or hour than the API key, IP address, or account is allowed.
- →Too many concurrent connections from one client.
- →Bursts at the start of a job, even when the average rate is within limits.
- →A shared IP, such as a NAT gateway or proxy, where other clients used up the quota.
How do you fix a 429 error when web scraping?
- →Honor
Retry-After. It can be a number of seconds or an HTTP date. - →Back off exponentially with jitter when there is no
Retry-After, and cap the number of attempts. - →Size your worker pool to the limit instead of discovering it with errors. Spread requests evenly rather than in bursts.
- →Cache pages you have already fetched and skip URLs that have not changed. See rate limiting for how the limits are usually counted.
How do you fix a 429 error on your own server?
- →Send
Retry-Afterand rate-limit headers so clients can pace themselves. - →Limit per API key or account rather than per IP where you can, so shared networks are not punished.
- →Return a body that says which limit was hit and when it resets.
How do you handle a 429 error in a retry loop?
fetch() treats 429 as temporary. It waits for Retry-After when the server sends a number of seconds, otherwise backs off exponentially with jitter, caps every wait at 60 seconds, and gives up after five attempts.
import random
import time
import requests
RETRYABLE = {408, 429, 500, 502, 503, 504, 520, 521, 522, 523, 524}
def fetch(url: str, max_attempts: int = 5) -> requests.Response:
for attempt in range(max_attempts):
try:
response = requests.get(url, timeout=(10, 60))
except requests.Timeout:
time.sleep(2**attempt + random.uniform(0, 1))
continue
if response.status_code not in RETRYABLE:
response.raise_for_status()
return response
retry_after = response.headers.get("Retry-After", "")
backoff = 2**attempt + random.uniform(0, 1)
time.sleep(min(int(retry_after) if retry_after.isdigit() else backoff, 60))
raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")
How does Context.dev handle a 429 error?
Context.dev runs every scrape in a browser with managed proxies and handles proxy selection and fetch retries for you. If the site still serves an anti-bot challenge or access wall, the output fails with WEBSITE_BLOCKED inside an HTTP 200 response, and a request where every output fails is not charged. A 429 from the Context.dev API itself means your organization is using all of its concurrency slots: 1 on Free, 10 on Developer, 100 on Pro, 250 on Growth, and 500 on Scale. The response carries Retry-After and X-RateLimit-* headers, and Batches queue large URL lists for you.
See what the web scraping API does on every request, or read how to fix HTTP errors in web scraping for a longer walkthrough.
Frequently asked questions about a 429 error
What does 429 mean?
You sent more requests than the server allows in a time window. The server is asking you to slow down, not refusing you permanently.
How long should I wait after a 429?
As long as Retry-After says. Without it, start around one second and double the wait on each retry, with random jitter so parallel workers do not retry in lockstep.
Which status codes are related to 429?
Sources
Last reviewed