HTTP 500 Internal Server Error: What It Means and How to Fix It
HTTP 500 Internal Server Error is a status code meaning the server hit an unexpected condition that stopped it from fulfilling the request. It is the generic server-side failure: an unhandled exception, a crashed worker, or a misconfiguration. The problem is on the server, so the same request may succeed later without any change from the client.
- Code
- 500
- Name
- Internal Server Error
- Class
- 5xx server error
- Retry?
- Yes, with backoff
What causes a 500 error?
- →An unhandled exception in application code.
- →A failed dependency: database, cache, or third-party API.
- →A bad deploy or configuration change.
- →Input the server did not expect and did not validate, which it should have answered with a 4xx.
How do you fix a 500 error when web scraping?
- →Retry with exponential backoff and a small attempt cap. Transient 500s often clear within seconds.
- →If one URL always returns 500, record it and move on. Something about that page breaks the server.
- →Check whether your request has unusual parameters or headers that a browser would not send.
How do you fix a 500 error on your own server?
- →Look up the request ID in your error tracker or logs to find the stack trace.
- →Validate inputs and return 4xx codes for bad requests, so 500 means a real server bug.
- →Roll back the last deploy if 500s started right after it.
How do you handle a 500 error in a retry loop?
fetch() treats 500 as temporary. It waits for Retry-After when the server sends a number of seconds, otherwise backs off exponentially with jitter, caps every wait at 60 seconds, and gives up after five attempts.
import random
import time
import requests
RETRYABLE = {408, 429, 500, 502, 503, 504, 520, 521, 522, 523, 524}
def fetch(url: str, max_attempts: int = 5) -> requests.Response:
for attempt in range(max_attempts):
try:
response = requests.get(url, timeout=(10, 60))
except requests.Timeout:
time.sleep(2**attempt + random.uniform(0, 1))
continue
if response.status_code not in RETRYABLE:
response.raise_for_status()
return response
retry_after = response.headers.get("Retry-After", "")
backoff = 2**attempt + random.uniform(0, 1)
time.sleep(min(int(retry_after) if retry_after.isdigit() else backoff, 60))
raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")
How does Context.dev handle a 500 error?
Context.dev retries the fetch for you, and by default Scrape can reuse a capture made in the last 3 days (maxAgeMs), so a page that is briefly down may still come back from cache. If no capture is available, the failed output carries an error_code and message inside an HTTP 200 response, and a request where every output fails is not charged. A 500 from the Context.dev API itself means the scrape could not be completed; output-level INTERNAL_ERROR is a temporary failure you can retry. The Context.dev SDKs retry twice with exponential backoff after connection errors and 408, 409, 429, and 5xx responses from the API itself.
See what the web scraping API does on every request, or read how to fix HTTP errors in web scraping for a longer walkthrough.
Frequently asked questions about a 500 error
Is a 500 error my fault?
Usually not. It means the server failed. A client can trigger it with unexpected input, but a well-built server answers that with a 4xx instead.
Should I retry a 500 Internal Server Error?
Yes, a few times with backoff for idempotent requests. If it keeps failing, the bug is persistent and retries only add load.
What is the difference between 500 and 502?
A 500 comes from the server handling the request. A 502 comes from a gateway or proxy that got a bad response from the server behind it.
Which status codes are related to 500?
Sources
Last reviewed