TL;DR
- Check whether JavaScript builds the missing content after the server returns its initial HTML shell.
- Inspect a 200 response for anti-bot challenge markers, unusually short HTML, or an empty body caused by IP blocking.
- Recheck your CSS selector or XPath against the live DOM, including localized and A/B-tested variants.
- Review robots.txt settings and compare responses with realistic User-Agent, Accept-Language, and Referer headers.
- Test for rate limiting by logging response sizes and adding delays. This guide provides diagnostic checks and code for each cause. Context.dev handles JavaScript rendering and anti-bot infrastructure through a managed API when you do not want to maintain those controls yourself.
Why "empty results" is its own failure mode
Empty results evade normal error handling because HTTP success and extraction success measure different things. A server can return 200 OK while sending an initial JavaScript shell, an anti-bot challenge, or a page variant without the target data. Your parser then returns [], None, or an empty object, and exception-based monitoring records a successful run.
Inspect the response before changing parsing logic. Save the raw HTML and record its byte length. Check the status code and response headers separately. Then compare the raw HTML with the DOM shown in browser DevTools. If the expected content never appears in the response, no CSS selector or XPath expression can extract it. If the content appears in the raw HTML, test whether your selector still matches the current structure.
The following sections cover five causes that produce empty payloads despite an apparently successful request. They focus on silent failures rather than explicit network errors. For requests returning 403, 429, or timeouts, use our web scraper failure debugging guide.
Cause 1: JavaScript-rendered content never reached your parser
A plain HTTP client retrieves the initial HTML response but does not execute the page’s JavaScript. Sites built with React, Vue, and similar frameworks may return little more than script tags and an empty <div id="root"> or <div id="app">. The browser runs those scripts, requests additional data, and inserts the visible content into the DOM. requests.get() and raw fetch() stop before those steps, so BeautifulSoup or another parser receives no content to select.
Compare the raw response with the rendered DOM before changing your parser. Open View Source and search for text that appears on the page. Then search for the same text in the browser’s Elements panel. Content found only in Elements was added after JavaScript ran. You can also print the raw HTML and check for an empty application container.
Python can expose the difference by running the same CSS selector against raw HTML and a rendered page.
import requests
from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright
url = "https://example.com/products"
selector = ".product-card"
response = requests.get(url, timeout=20)
soup = BeautifulSoup(response.text, "html.parser")
print("Raw HTML matches", len(soup.select(selector)))
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector(selector, timeout=10000)
products = page.locator(selector).all_text_contents()
print("Rendered matches", len(products))
browser.close()If the first count equals zero and the Playwright count is positive, JavaScript rendering caused the empty result. Waiting for the target selector is usually more precise than adding a fixed sleep because it ties extraction to the content you need.
Node.js Playwright follows the same sequence.
const { chromium } = require("playwright");
(async () => {
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://example.com/products", {
waitUntil: "domcontentloaded"
});
await page.waitForSelector(".product-card");
const products = await page
.locator(".product-card")
.allTextContents();
console.log("Rendered matches", products.length);
await browser.close();
})();A browser adds runtime and memory overhead, but it gives client-side code time to create the DOM that your web scraper needs.
Cause 2: Anti-bot systems returning empty shells instead of blocks
Anti-bot systems can return an empty shell while reporting 200 OK. Cloudflare, PerimeterX, and DataDome may serve a JavaScript challenge, CAPTCHA page, or nearly blank document instead of the requested content. Your parser then receives no matching elements, which can resemble a page that needs JavaScript rendering.
Inspect the body before changing your parser. Check its byte length and search for markers such as cdn-cgi, Just a moment, DataDome, _px, or captcha. Run the same request with identical headers through a residential connection and a datacenter or cloud IP. If only the datacenter request receives the shell, IP blocking or reputation filtering is a likely cause. Challenge markers provide evidence, but vendors can change them and legitimate pages can be short.
import os
import requests
session = requests.Session()
session.headers.update([
("User-Agent", "Mozilla/5.0 AppleWebKit/537.36 Chrome/124 Safari/537.36"),
("Accept", "text/html,application/xhtml+xml"),
("Accept-Language", "en-US,en;q=0.9"),
])
response = session.get(os.environ["TARGET_URL"], timeout=20)
body = response.text.lower()
expected = os.environ.get("EXPECTED_TEXT", "").lower()
markers = (
"just a moment",
"cdn-cgi/challenge-platform",
"datadome",
"_px",
"captcha",
)
has_expected = bool(expected) and expected in body
has_marker = any(marker in body for marker in markers)
suspected = not has_expected and (has_marker or len(response.content) < 1000)
print(response.status_code, len(response.content))
print(dict(response.headers))
print("challenge suspected" if suspected else "real response detected")Node.js can apply the same check without launching a browser.
const headers = new Headers([
["user-agent", "Mozilla/5.0 AppleWebKit/537.36 Chrome/124 Safari/537.36"],
["accept", "text/html,application/xhtml+xml"],
["accept-language", "en-US,en;q=0.9"]
])
const response = await fetch(process.env.TARGET_URL, { headers })
const text = await response.text()
const body = text.toLowerCase()
const expected = (process.env.EXPECTED_TEXT || "").toLowerCase()
const markers = [
"just a moment",
"cdn-cgi/challenge-platform",
"datadome",
"_px",
"captcha"
]
const hasExpected = Boolean(expected) && body.includes(expected)
const hasMarker = markers.some(marker => body.includes(marker))
const suspected = !hasExpected && (hasMarker || Buffer.byteLength(text) < 1000)
console.log(response.status, Buffer.byteLength(text))
console.log(Object.fromEntries(response.headers))
console.log(suspected ? "challenge suspected" : "real response detected")Record the status, headers, body size, and a short body sample in production logs. Those fields separate a blocked shell from a successful response before the CSS selector runs.
Cause 3: CSS or XPath selectors have drifted from the live DOM
Frontend deployments can invalidate a CSS selector or XPath expression without causing a request error. Build tools may generate new class names, while component updates may move an element under a different parent. Your parser still receives valid HTML, but select() or locator() matches zero elements and returns an empty collection.
Re-inspect the rendered page in browser DevTools before changing request logic. Run the selector against the current DOM, and log the match count before parsing fields. Save the HTML from failed runs so you can compare A/B tests, localized pages, and layouts selected by viewport size or cookies.
BeautifulSoup returns [] when a selector matches nothing. Logging the count separates selector failure from later extraction errors.
import requests
from bs4 import BeautifulSoup
response = requests.get(
"https://example.com/products",
headers={"User-Agent": "Mozilla/5.0"},
timeout=20,
)
soup = BeautifulSoup(response.text, "html.parser")
css_selector = ".product-card__3fK9"
cards = soup.select(css_selector)
print("status", response.status_code)
print("html bytes", len(response.content))
print("matched cards", len(cards))
if not cards:
with open("failed-page.html", "w", encoding="utf-8") as file:
file.write(response.text)Prefer selectors tied to meaning rather than generated presentation classes. Stable data-* attributes often survive design changes. Playwright can also locate visible text or select a parent based on a child element.
import { chromium } from "playwright";
const browser = await chromium.launch();
const page = await browser.newPage({ locale: "en-US" });
await page.goto("https://example.com/products", {
waitUntil: "networkidle",
});
const byAttribute = page.locator('[data-testid="product-card"]');
const byText = page.getByRole("heading", { name: "Featured" });
const byRelationship = page.locator(
'article:has(h2:has-text("Featured"))'
);
console.log("attribute matches", await byAttribute.count());
console.log("text matches", await byText.count());
console.log("relationship matches", await byRelationship.count());
await browser.close();Text-based selectors can break when the site changes language, so use them only when you control or record the locale. If match counts vary across otherwise identical runs, capture the URL, cookies, locale, and returned HTML. Those records can reveal an experiment or regional DOM variant rather than ordinary selector drift.
Cause 4: robots.txt and header-based blocking
A robots.txt rule can produce empty results before your scraper sends a request. Crawling frameworks such as Scrapy may check disallowed paths and skip them without raising a network error. Python requests and the browser fetch API do not enforce robots.txt automatically, so inspect your framework settings, middleware, and logs. Treat robots.txt as a site policy even when your client does not enforce it.
Servers can also return different HTML based on request headers. A missing User-Agent may identify a basic script, while Accept-Language can determine which localized page variant appears. Some servers also inspect Referer when a page normally follows an internal navigation. Compare the status, response size, and expected selector count for a bare request and a request with browser-like headers.
import os
import requests
url = os.environ["TARGET_URL"]
bare = requests.get(url, timeout=20)
headers = {}
headers["User-Agent"] = os.environ["SCRAPER_USER_AGENT"]
headers["Accept"] = "text/html,application/xhtml+xml"
headers["Accept-Language"] = "en-US,en;q=0.9"
headers["Referer"] = os.environ["EXPECTED_REFERER"]
browser_like = requests.get(url, headers=headers, timeout=20)
print(bare.status_code, len(bare.content))
print(browser_like.status_code, len(browser_like.content))Node.js supports the same comparison through fetch and Headers.
const url = process.env.TARGET_URL
const bare = await fetch(url)
const headers = new Headers()
headers.set("User-Agent", process.env.SCRAPER_USER_AGENT)
headers.set("Accept", "text/html,application/xhtml+xml")
headers.set("Accept-Language", "en-US,en;q=0.9")
headers.set("Referer", process.env.EXPECTED_REFERER)
const browserLike = await fetch(url, { headers })
console.log(bare.status, (await bare.text()).length)
console.log(browserLike.status, (await browserLike.text()).length)Use a truthful, stable User-Agent, and send Referer only when the navigation would normally include one. Playwright sets a user agent when creating a browser context. Selenium requires a browser option before starting the driver. Page-level request headers can supply language and referer values, but changing the HTTP header alone may not change JavaScript-visible browser properties.
Cause 5: Rate limiting disguised as empty responses
Some servers enforce rate limiting by returning an empty or truncated body while keeping the status at 200. Your parser then receives too little HTML to match a CSS selector, which makes throttling resemble a parsing failure.
Log each request time, status, and response size before parsing. A sequence that starts with normal payloads and becomes empty as request frequency rises points to throttling. Repeat the test with deliberate delays. Treat any minimum size as site-specific because some valid pages are naturally small.
import time
import requests
MIN_BYTES = 1000
session = requests.Session()
for url in urls:
for attempt in range(4):
response = session.get(url, timeout=20)
size = len(response.content)
print(time.time(), response.status_code, size, url)
if size >= MIN_BYTES:
parse(response.text)
break
delay = 2 ** attempt
print(f"Suspicious response. Retrying in {delay}s")
time.sleep(delay)
time.sleep(1)The Python example spaces normal requests by one second and applies exponential backoff when a response falls below the expected size. Production code should also cap retries and record failed URLs for later processing.
A sequential Node.js queue provides similar pacing without sending every request at once.
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms))
const MIN_BYTES = 1000
async function fetchWithBackoff(url) {
for (let attempt = 0; attempt < 4; attempt++) {
const response = await fetch(url)
const body = await response.text()
console.log(Date.now(), response.status, body.length, url)
if (body.length >= MIN_BYTES) return body
await sleep(1000 * 2 ** attempt)
}
throw new Error(`Repeated empty responses for ${url}`)
}
for (const url of urls) {
const html = await fetchWithBackoff(url)
await parse(html)
await sleep(1000)
}If slower requests restore full payloads, keep concurrency bounded and add retry jitter so multiple workers do not retry together. See the scraper failure debugging guide for detailed 429 handling, retry limits, and backoff strategies.
A quick diagnostic checklist
- Compare the raw HTML with the rendered DOM, and check whether JavaScript inserts the missing content after page load.
- Inspect the response body for challenge text, unusual scripts, or a nearly empty HTML shell despite a 200 status.
- Run each CSS selector or XPath expression in browser DevTools, and log how many elements it matches.
- Review your library’s robots.txt settings, and compare a bare request with one that sends realistic browser headers.
- Log response sizes and timestamps across several requests, then add delays to see whether request frequency triggers rate limiting.
When to stop debugging and use a managed scraping API
Production scrapers become expensive when empty results require repeated investigation. JavaScript rendering, anti-bot responses, selector changes, header variations, and silent rate limiting can each reappear whenever a target site changes. You then maintain browser infrastructure, proxies, retries, detection rules, and site-specific extraction logic alongside the data pipeline itself.
DIY tools remain sensible when you need a one-off script or direct control over a browser session. Selenium and Playwright let you manage interactions, cookies, and persistent logins. Requests and BeautifulSoup work well for stable server-rendered pages. Those tools also give you full control when a managed service cannot support a specialized session or interaction.
A managed API becomes a better fit when you run scrapers in production and spend recurring engineering time diagnosing the same failure modes. Context.dev's managed API handles JavaScript rendering and anti-bot infrastructure without requiring you to maintain browser fingerprints or proxy rotation. Managed request pacing and page retrieval also keep blocking logic outside your application.
Context.dev exposes scraping, crawling, and structured delivery through one API. Your application can consume JSON or markdown instead of depending directly on a site's current DOM structure, which reduces breakage caused by CSS selector drift. MCP integration can send the same output directly into LLM pipelines without a separate conversion layer.
A managed API cannot prevent target sites from changing or becoming unavailable. It moves the detection and adaptation work out of your codebase. For production pipelines that need current web data rather than persistent browser sessions, that trade reduces the infrastructure you own and the empty-result cases your application must diagnose.
FAQs
Why does my scraper return 200 but no data?
A 200 response only confirms that the server answered. The body may contain an initial JavaScript shell, an anti-bot challenge, or a stripped page variant instead of the expected content. Log the raw HTML and response size before checking your parser.
How do I tell if a site is JavaScript-rendered or blocking me?
Compare the raw response with the rendered DOM in browser DevTools. Content that appears only after rendering usually requires a browser. Challenge text, unusual scripts, or different responses across IP addresses suggest blocking.
Is my CSS selector wrong or did the site change?
Test the CSS selector against the current rendered DOM and log its match count. If an older saved page still matches, the site changed. Also check whether localized pages or A/B tests produce different markup.
Does robots.txt actually stop my scraper?
The file does not enforce access by itself, but some scraping libraries obey its rules and skip disallowed URLs. A server can separately return empty variants based on headers, cookies, or request behavior.
Can rate limiting cause empty results instead of errors?
Yes. Some servers return truncated or blank bodies instead of a 429 response. Compare response sizes with request timing, then retry more slowly. Use the scraper failure debugging guide for status-code, timeout, and backoff diagnostics.