HTTP 504 Gateway Timeout: What It Means and How to Fix It
HTTP 504 Gateway Timeout is a status code meaning a server acting as a gateway or proxy did not get a timely response from the upstream server it needed to complete the request. The gateway itself is working; the application behind it is slow or unreachable, often because of a long query or an overloaded backend.
- Code
- 504
- Name
- Gateway Timeout
- Class
- 5xx server error
- Retry?
- Yes, with backoff
What causes a 504 error?
- →A slow database query or external API call in the upstream application.
- →An overloaded upstream with a long request queue.
- →A proxy timeout shorter than the time some requests legitimately need.
- →Network problems between the gateway and the upstream.
How do you fix a 504 error when web scraping?
- →Retry with backoff, and lower concurrency against that host.
- →Avoid pages that trigger expensive work, such as huge result pages or deep pagination, when a lighter URL has the same data.
- →Note which URLs time out consistently and retry them later, separately from the main job.
How do you fix a 504 error on your own server?
- →Find the slow requests in upstream logs and fix the query or call behind them.
- →Align timeouts so the proxy waits a little longer than the application’s own timeout.
- →Move long-running work to background jobs with polling.
How do you handle a 504 error in a retry loop?
fetch() treats 504 as temporary. It waits for Retry-After when the server sends a number of seconds, otherwise backs off exponentially with jitter, caps every wait at 60 seconds, and gives up after five attempts.
import random
import time
import requests
RETRYABLE = {408, 429, 500, 502, 503, 504, 520, 521, 522, 523, 524}
def fetch(url: str, max_attempts: int = 5) -> requests.Response:
for attempt in range(max_attempts):
try:
response = requests.get(url, timeout=(10, 60))
except requests.Timeout:
time.sleep(2**attempt + random.uniform(0, 1))
continue
if response.status_code not in RETRYABLE:
response.raise_for_status()
return response
retry_after = response.headers.get("Retry-After", "")
backoff = 2**attempt + random.uniform(0, 1)
time.sleep(min(int(retry_after) if retry_after.isdigit() else backoff, 60))
raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")
How does Context.dev handle a 504 error?
Context.dev retries the fetch for you, and by default Scrape can reuse a capture made in the last 3 days (maxAgeMs), so a page that is briefly down may still come back from cache. If no capture is available, the failed output carries an error_code and message inside an HTTP 200 response, and a request where every output fails is not charged. The Context.dev SDKs retry twice with exponential backoff after connection errors and 408, 409, 429, and 5xx responses from the API itself.
See what the web scraping API does on every request, or read how to fix HTTP errors in web scraping for a longer walkthrough.
Frequently asked questions about a 504 error
What causes a 504 Gateway Timeout?
The server behind a proxy or CDN took longer to answer than the proxy was willing to wait. The cause is usually slow application code or an overloaded backend.
Is 504 a client or server problem?
Server. The client’s request reached the gateway fine. Retrying later or reducing load is all a client can do.
Which status codes are related to 504?
Sources
Last reviewed