HTTP Error Codes: What They Mean and How to Fix Them
An HTTP error code is a three-digit status code in the 4xx or 5xx range that a server returns when it cannot complete a request. A 4xx code points at the request, a 5xx code points at the server or a gateway in front of it, and Cloudflare adds its own 52x and 1020 codes for failures at its edge.
Each page below explains one code: what causes it, how to fix it when you are scraping, how to fix it on your own server, and how a managed web scraping API such as Context.dev handles it.
What are 4xx client errors?
The server is answering, but it rejected something about the request: the URL, the method, the headers, the credentials, or the rate.
What are 5xx server errors?
The request may be fine. The server, or a gateway in front of it, failed while handling it. Most of these are temporary and safe to retry with backoff.
What are Cloudflare 52x and 1020 errors?
Cloudflare-specific codes that describe what went wrong between Cloudflare and the origin server, or which firewall rule blocked you.
Which HTTP errors should a scraper retry?
| Code | Meaning | What a scraper should do |
|---|---|---|
| 400 Bad Request | The server rejected the request as malformed or invalid. | Fix the cause, do not retry |
| 401 Unauthorized | The request has no valid authentication credentials. | Fix the cause, do not retry |
| 402 Payment Required | Reserved by the spec; APIs use it for billing problems. | Fix the cause, do not retry |
| 403 Forbidden | The server understood the request and refuses it. | Fix the cause, do not retry |
| 404 Not Found | Nothing exists at this URL, or the server will not say. | Fix the cause, do not retry |
| 405 Method Not Allowed | The URL exists but does not accept this HTTP method. | Fix the cause, do not retry |
| 406 Not Acceptable | No response matches the Accept headers in the request. | Fix the cause, do not retry |
| 408 Request Timeout | The server stopped waiting for the request to arrive. | Retry with backoff |
| 409 Conflict | The request conflicts with the current state of the resource. | Fix the cause, do not retry |
| 410 Gone | The resource was removed on purpose and is not coming back. | Fix the cause, do not retry |
| 422 Unprocessable Content | The request parsed fine, but its content failed validation. | Fix the cause, do not retry |
| 429 Too Many Requests | The client hit a rate limit and should slow down. | Wait for Retry-After, then retry |
| 499 Client Closed Request | An nginx log code: the client hung up before the response. | Retry after your own timeout |
| 500 Internal Server Error | The server failed unexpectedly while handling the request. | Retry with backoff |
| 502 Bad Gateway | A gateway or proxy got an invalid response from upstream. | Retry with backoff |
| 503 Service Unavailable | The server is overloaded or in maintenance, for now. | Wait for Retry-After, then retry |
| 504 Gateway Timeout | A gateway waited too long for the upstream server. | Retry with backoff |
| 520 Web Server Returns an Unknown Error | The origin sent Cloudflare an empty or unexpected response. | Retry with backoff |
| 521 Web Server Is Down | The origin refused Cloudflare’s connection. | Retry with backoff |
| 522 Connection Timed Out | Cloudflare timed out connecting to the origin. | Retry with backoff |
| 523 Origin Is Unreachable | Cloudflare has no route to the origin. | Retry with backoff |
| 524 A Timeout Occurred | The origin accepted the connection but answered too slowly. | Retry with backoff |
| 525 SSL Handshake Failed | The TLS handshake between Cloudflare and the origin failed. | Fix the cause, do not retry |
| 526 Invalid SSL Certificate | Cloudflare could not validate the origin’s certificate. | Fix the cause, do not retry |
| 1020 Access Denied | A firewall rule set by the site owner blocked the request. | Fix the cause, do not retry |
How do you retry HTTP errors with backoff?
Retry only codes that can succeed on a second try, honor Retry-After when the server sends it, add jitter so parallel workers do not retry in lockstep, and cap both the wait and the number of attempts. This Python helper does all four.
import random
import time
import requests
RETRYABLE = {408, 429, 500, 502, 503, 504, 520, 521, 522, 523, 524}
def fetch(url: str, max_attempts: int = 5) -> requests.Response:
for attempt in range(max_attempts):
try:
response = requests.get(url, timeout=(10, 60))
except requests.Timeout:
time.sleep(2**attempt + random.uniform(0, 1))
continue
if response.status_code not in RETRYABLE:
response.raise_for_status()
return response
retry_after = response.headers.get("Retry-After", "")
backoff = 2**attempt + random.uniform(0, 1)
time.sleep(min(int(retry_after) if retry_after.isdigit() else backoff, 60))
raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")
Frequently asked questions about HTTP errors
Which HTTP errors should a scraper retry?
Retry timeouts, 429, and transient server errors (500, 502, 503, 504, and Cloudflare 520 to 524) with exponential backoff and a cap. Do not retry 400, 401, 403, 404, 405, 406, 410, or 422 unchanged, because the same request fails the same way.
What is the difference between 4xx and 5xx errors?
A 4xx code says the request was the problem: wrong URL, missing credentials, too many requests. A 5xx code says the server or a gateway failed while handling a request that may have been fine.
Why does Cloudflare use codes like 520 and 1020?
They are Cloudflare extensions that describe failures the standard codes cannot: what went wrong between Cloudflare and the origin (520 to 526) or which firewall rule blocked a request (1020).
Where can you learn more about blocks and errors?
Ship an agent that actually knows things.
Free tier, 10-minute integration, and the same API powering agents at Mintlify, daily.dev, and Propane. No credit card to start.