Your web scraper keeps returning empty results because the expected content is missing from the response, added later by JavaScript, replaced by a challenge shell, missed by a stale selector, changed by a redirect or request context, or suppressed as request frequency rises. Diagnose those causes in that order: compare raw HTML with the rendered DOM, inspect the body for challenge markers, test selectors against the current page, verify the final URL and request context, and then repeat at a lower request rate.
TL;DR
- If content appears only in the rendered DOM, use Playwright or Selenium with a condition-based wait. Context.dev’s Web Scraping API provides managed JavaScript rendering, browser actions, and wait controls.
- If the body contains a challenge instead of the expected page, verify challenge markers and compare network paths. Context.dev can manage browser, proxy, and CAPTCHA infrastructure without promising that every target will succeed.
- If the expected content exists but your selector returns zero matches, recheck the selector against the current DOM and validate required fields after extraction.
- If responses vary by headers or URL policy, review robots.txt handling, locale, cookies, and request headers.
- If payloads shrink as request frequency rises, reduce concurrency, add spacing and backoff, and inspect
429andRetry-Afterbehavior. Context.dev supports managed batch retrieval, while your application still owns URL policy, expected-field validation, and failure handling.
| Empty-result symptom | Diagnostic test | Valid DIY fix | Context.dev capability that reduces maintenance |
|---|---|---|---|
| Content appears in DevTools but not in raw HTML | Compare View Source with the rendered DOM, then wait for the target element | Render with Playwright or Selenium and use a selector-based explicit wait | Managed JavaScript rendering, browser actions, and wait controls |
A 200 body contains a challenge or nearly empty shell | Log body size, final URL, headers, expected markers, and challenge markers | Use a compliant browser and proxy setup, and handle challenges where permitted | Managed browser, proxy, and CAPTCHA infrastructure |
| HTML is present, but the selector matches nothing | Run the selector against the live DOM and count matches | Update selectors and validate required fields before accepting output | Structured output formats that reduce direct DOM parsing, with customer-owned schema validation |
| Results change with headers, locale, or URL policy | Compare bare and browser-like requests, and review robots.txt and redirect behavior | Set truthful headers, preserve required cookies, and enforce an explicit URL policy | Managed page retrieval, while the customer retains URL-policy decisions |
| Responses become empty or truncated as request frequency rises | Log timestamps, status, response bytes, and Retry-After, then repeat at lower concurrency | Bound concurrency, space requests, add jitter, and cap retries | Batch retrieval that moves scheduling and pacing infrastructure out of application code |
Why "empty results" is its own failure mode
Empty results evade normal error handling because HTTP success and extraction success measure different things. MDN defines 200 OK as a successful HTTP request, but that status does not confirm that JavaScript rendered, the expected fields are present, or a selector matched. A response can contain an initial JavaScript shell, an anti-bot challenge, or a page variant without the target data. The parser then returns [], None, or an empty object, and exception-based monitoring may record a successful run.
Inspect the response before changing parsing logic. Save the raw HTML and record its byte length. Check the status code and response headers separately. Then compare the raw HTML with the DOM shown in browser DevTools. If the expected content never appears in the response, no CSS selector or XPath expression can extract it. If the content appears in the raw HTML, test whether your selector still matches the current structure.
The following sections move from symptom to likely cause for silent empty-result failures. If the request instead returns a 403, a confirmed 429, or a timeout, use our guide to debugging scraper HTTP failures, timeouts, Retry-After, and backoff.
Cause 1: JavaScript-rendered content never reached your parser
The raw HTTP response can contain only an application shell while the browser’s rendered DOM contains the visible data. In that case, a parser operating on the response body has no matching content to extract, even though the page appears complete in a browser.
Compare the raw response with the rendered DOM before changing your parser. Open View Source and search for text that appears on the page. Then search for the same text in the browser’s Elements panel. Content found only in Elements was added after JavaScript ran. You can also print the raw HTML and check for an empty application container. The Playwright Page API supports inspecting the current DOM and waiting for a target locator instead of assuming that navigation completion means the required content is ready.
Python can expose the difference by running the same CSS selector against raw HTML and a rendered page.
import requests
from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright
url = "https://example.com/products"
selector = ".product-card"
response = requests.get(url, timeout=20)
soup = BeautifulSoup(response.text, "html.parser")
print("Raw HTML matches", len(soup.select(selector)))
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector(selector, timeout=10000)
products = page.locator(selector).all_text_contents()
print("Rendered matches", len(products))
browser.close()If the first count equals zero and the Playwright count is positive, JavaScript rendering caused the empty result. Waiting for the target selector is usually more precise than adding a fixed sleep because it ties extraction to the content you need.
Node.js Playwright follows the same sequence.
const { chromium } = require("playwright");
(async () => {
const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://example.com/products", {
waitUntil: "domcontentloaded"
});
await page.waitForSelector(".product-card");
const products = await page
.locator(".product-card")
.allTextContents();
console.log("Rendered matches", products.length);
await browser.close();
})();A browser adds runtime and memory overhead, but it gives client-side code time to create the DOM that your web scraper needs. Selenium is also a valid DIY option. Its official waiting guidance explains that JavaScript can continue changing the page after navigation returns and recommends explicit waits tied to a specific condition rather than arbitrary sleeps.
If you do not want to operate browsers and wait logic yourself, Context.dev’s Web Scraping API provides managed JavaScript rendering, browser actions, and wait controls for content that depends on scripts, clicks, or scrolling. Your application must still verify that the expected fields are present and decide how to handle missing output.
Cause 2: Anti-bot systems returning empty shells instead of blocks
A response can report 200 OK while containing a challenge, CAPTCHA page, interstitial, or nearly blank shell instead of the expected content. Your parser then receives no matching elements, which can resemble a page that merely needs JavaScript rendering. As MDN’s definition of 200 OK makes clear, the status confirms that the request succeeded; it does not confirm that the returned representation contains the page you intended to extract.
Inspect the body before changing your parser. Check its byte length and search for markers such as cdn-cgi, Just a moment, DataDome, _px, or captcha. Run the same request with identical headers through a residential connection and a datacenter or cloud IP. If only the datacenter request receives the shell, IP blocking or reputation filtering is a likely cause. Challenge markers provide evidence, but vendors can change them and legitimate pages can be short.
import os
import requests
session = requests.Session()
session.headers.update([
("User-Agent", "Mozilla/5.0 AppleWebKit/537.36 Chrome/124 Safari/537.36"),
("Accept", "text/html,application/xhtml+xml"),
("Accept-Language", "en-US,en;q=0.9"),
])
response = session.get(os.environ["TARGET_URL"], timeout=20)
body = response.text.lower()
expected = os.environ.get("EXPECTED_TEXT", "").lower()
markers = (
"just a moment",
"cdn-cgi/challenge-platform",
"datadome",
"_px",
"captcha",
)
has_expected = bool(expected) and expected in body
has_marker = any(marker in body for marker in markers)
suspected = not has_expected and (has_marker or len(response.content) < 1000)
print(response.status_code, len(response.content))
print(dict(response.headers))
print("challenge suspected" if suspected else "real response detected")Node.js can apply the same check without launching a browser.
const headers = new Headers([
["user-agent", "Mozilla/5.0 AppleWebKit/537.36 Chrome/124 Safari/537.36"],
["accept", "text/html,application/xhtml+xml"],
["accept-language", "en-US,en;q=0.9"]
])
const response = await fetch(process.env.TARGET_URL, { headers })
const text = await response.text()
const body = text.toLowerCase()
const expected = (process.env.EXPECTED_TEXT || "").toLowerCase()
const markers = [
"just a moment",
"cdn-cgi/challenge-platform",
"datadome",
"_px",
"captcha"
]
const hasExpected = Boolean(expected) && body.includes(expected)
const hasMarker = markers.some(marker => body.includes(marker))
const suspected = !hasExpected && (hasMarker || Buffer.byteLength(text) < 1000)
console.log(response.status, Buffer.byteLength(text))
console.log(Object.fromEntries(response.headers))
console.log(suspected ? "challenge suspected" : "real response detected")Record the status, final URL, headers, body size, expected-content marker, and a short body sample in production logs. Those fields help distinguish a suspected challenge shell from usable page content before the CSS selector runs.
For teams that do not want to maintain this infrastructure, Context.dev acts as a managed retrieval layer. Our production MCP scraping pipeline guide describes managed browsers, proxies, and CAPTCHA handling without requiring the application to operate those components. This reduces maintenance, but it does not guarantee extraction from every site. Customers still own URL policy, output validation, provenance, and failure handling.
Cause 3: CSS or XPath selectors have drifted from the live DOM
Frontend deployments can invalidate a CSS selector or XPath expression without causing a request error. Build tools may generate new class names, while component updates may move an element under a different parent. Your parser still receives valid HTML, but select() or locator() matches zero elements and returns an empty collection.
Re-inspect the rendered page in browser DevTools before changing request logic. Run the selector against the current DOM, and log the match count before parsing fields. Save the HTML from failed runs so you can compare A/B tests, localized pages, and layouts selected by viewport size or cookies.
BeautifulSoup returns [] when a selector matches nothing. Logging the count separates selector failure from later extraction errors.
import requests
from bs4 import BeautifulSoup
response = requests.get(
"https://example.com/products",
headers={"User-Agent": "Mozilla/5.0"},
timeout=20,
)
soup = BeautifulSoup(response.text, "html.parser")
css_selector = ".product-card__3fK9"
cards = soup.select(css_selector)
print("status", response.status_code)
print("html bytes", len(response.content))
print("matched cards", len(cards))
if not cards:
with open("failed-page.html", "w", encoding="utf-8") as file:
file.write(response.text)Prefer selectors tied to meaning rather than generated presentation classes. Stable data-* attributes often survive design changes. Playwright can also locate visible text or select a parent based on a child element.
import { chromium } from "playwright";
const browser = await chromium.launch();
const page = await browser.newPage({ locale: "en-US" });
await page.goto("https://example.com/products", {
waitUntil: "domcontentloaded",
});
const byAttribute = page.locator('[data-testid="product-card"]');
const byText = page.getByRole("heading", { name: "Featured" });
const byRelationship = page.locator(
'article:has(h2:has-text("Featured"))'
);
console.log("attribute matches", await byAttribute.count());
console.log("text matches", await byText.count());
console.log("relationship matches", await byRelationship.count());
await browser.close();Text-based selectors can break when the site changes language, so use them only when you control or record the locale. If match counts vary across otherwise identical runs, capture the URL, cookies, locale, and returned HTML. Those records can reveal an experiment or regional DOM variant rather than ordinary selector drift. Treat every extraction result as untrusted until required fields and types pass your application’s schema checks. Context.dev can return HTML, Markdown, or structured fields through its Web Scraping API, but customers still own expected-field validation and the decision to reject incomplete records.
Cause 4: The request reached a different URL or page variant
First record the final URL rather than assuming that extraction ran against the URL you requested. A redirect can lead to a login page, consent screen, locale-specific route, canonical host, or another page that legitimately lacks the expected selector. Log the requested URL, final URL, redirect history when your client exposes it, status, content type, response size, and an expected-content marker.
Next inspect your crawler’s robots.txt settings, middleware, and logs to determine whether URL policy prevented a request. Treat robots.txt as a site policy regardless of how a particular client is configured.
Headers, cookies, and locale can also correlate with different page variants. Compare the same diagnostics for your normal request and a controlled request with the context the target expects. Use truthful header values, and do not assume that copying a few browser headers reproduces a browser session.
import os
import requests
url = os.environ["TARGET_URL"]
bare = requests.get(url, timeout=20)
headers = {}
headers["User-Agent"] = os.environ["SCRAPER_USER_AGENT"]
headers["Accept"] = "text/html,application/xhtml+xml"
headers["Accept-Language"] = "en-US,en;q=0.9"
headers["Referer"] = os.environ["EXPECTED_REFERER"]
browser_like = requests.get(url, headers=headers, timeout=20)
print(bare.status_code, len(bare.content))
print(browser_like.status_code, len(browser_like.content))Node.js supports the same comparison through fetch and Headers.
const url = process.env.TARGET_URL
const bare = await fetch(url)
const headers = new Headers()
headers.set("User-Agent", process.env.SCRAPER_USER_AGENT)
headers.set("Accept", "text/html,application/xhtml+xml")
headers.set("Accept-Language", "en-US,en;q=0.9")
headers.set("Referer", process.env.EXPECTED_REFERER)
const browserLike = await fetch(url, { headers })
console.log(bare.status, (await bare.text()).length)
console.log(browserLike.status, (await browserLike.text()).length)Use truthful, stable request metadata and preserve only the cookies and navigation context your workflow legitimately requires. Record every variation you test so that a header experiment does not become an unexplained production dependency.
Cause 5: Rate limiting disguised as empty responses
An empty or truncated 200 response can correlate with higher request frequency, but that pattern is only a diagnostic clue, not a defined rate-limit response. A confirmed 429 Too Many Requests means the client sent too many requests in a given period, and the response may include a Retry-After header. Log both status and body size so you can distinguish protocol-level rate limiting from an unusable 200 response.
Log each request time, status, and response size before parsing. A sequence that starts with normal payloads and becomes empty as request frequency rises points to throttling. Repeat the test with deliberate delays. Treat any minimum size as site-specific because some valid pages are naturally small.
import time
import requests
MIN_BYTES = 1000
session = requests.Session()
for url in urls:
for attempt in range(4):
response = session.get(url, timeout=20)
size = len(response.content)
print(time.time(), response.status_code, size, url)
if size >= MIN_BYTES:
parse(response.text)
break
delay = 2 ** attempt
print(f"Suspicious response. Retrying in {delay}s")
time.sleep(delay)
time.sleep(1)The Python example spaces normal requests by one second and applies exponential backoff when a response falls below the expected size. Production code should also cap retries and record failed URLs for later processing.
A sequential Node.js queue provides similar pacing without sending every request at once.
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms))
const MIN_BYTES = 1000
async function fetchWithBackoff(url) {
for (let attempt = 0; attempt < 4; attempt++) {
const response = await fetch(url)
const body = await response.text()
console.log(Date.now(), response.status, body.length, url)
if (body.length >= MIN_BYTES) return body
await sleep(1000 * 2 ** attempt)
}
throw new Error(`Repeated empty responses for ${url}`)
}
for (const url of urls) {
const html = await fetchWithBackoff(url)
await parse(html)
await sleep(1000)
}If slower requests restore full payloads, keep concurrency bounded and add retry jitter so multiple workers do not retry together. See our guide to diagnosing 429, Retry-After, timeouts, and retry backoff for detailed failure handling.
Context.dev supports managed batch retrieval, which can move retrieval scheduling infrastructure out of application code. Our production MCP pipeline guide also covers shared concurrency limits, request spacing, and retries for transient failures. Customers still decide which URLs are allowed, validate returned records, and define retry limits and terminal failure handling.
A quick diagnostic checklist
- Compare the raw HTML with the rendered DOM, and check whether JavaScript inserts the missing content after page load.
- Inspect the response body for challenge text, unusual scripts, or a nearly empty HTML shell despite a 200 status.
- Run each CSS selector or XPath expression in browser DevTools, and log how many elements it matches.
- Record the requested and final URLs, review robots.txt and crawler policy, and compare controlled variations in headers, cookies, and locale.
- Log response sizes and timestamps across several requests, then add delays to see whether request frequency triggers rate limiting.
When to stop debugging and use a managed scraping API
Production scrapers become expensive when empty results require repeated investigation. JavaScript rendering, anti-bot responses, selector changes, header variations, and silent rate limiting can each reappear whenever a target site changes. You then maintain browser infrastructure, proxies, retries, detection rules, and site-specific extraction logic alongside the data pipeline itself.
DIY tools remain sensible when you need a one-off script or direct control over a browser session. Selenium and Playwright let you manage interactions, cookies, and persistent logins. Requests and BeautifulSoup work well for stable server-rendered pages. Those tools also give you full control when a managed service cannot support a specialized session or interaction.
A managed API may be a better fit when you run scrapers in production and spend recurring engineering time maintaining rendering, proxy, challenge-handling, batch, and retry infrastructure. Context.dev’s Web Scraping API provides dynamic-page rendering, browser actions, wait controls, multiple output formats, and batch retrieval through one API.
Context.dev can return Markdown, HTML, structured fields, images, and screenshots. Structured output can reduce direct dependence on site-specific selectors, but it does not remove the need for application-level checks. Treat output as untrusted until your expected-field and schema validation passes.
For AI workflows, our guide to building a production MCP web scraping pipeline explains how Context.dev supplies JavaScript-rendered pages and structured JSON without requiring the application to maintain browsers, proxies, CAPTCHA handling, or site adapters. The customer’s pipeline still owns URL policy, validation, provenance, retry limits, and terminal failure handling.
Context.dev is a strong fit when the goal is current web data for an application, agent, or LLM pipeline. Playwright and Selenium remain better fits when you need direct control over persistent sessions or specialized browser interactions. Requests and BeautifulSoup remain efficient choices for stable, server-rendered pages.
FAQs
Why does my scraper return 200 but no data?
A 200 OK confirms HTTP success, not extraction success. Check the final URL, content type, response bytes, expected marker, and selector count. If JavaScript creates the missing content, use Playwright or Selenium with a condition-based wait, or use Context.dev’s managed rendering and wait controls. In either case, your application should reject output that is missing required fields.
How do I tell if a site is JavaScript-rendered or blocking me?
Compare the raw response with the rendered DOM. If the target appears only after a browser runs scripts, the page requires rendering. If the response contains challenge markers, an unexpected final URL, or a challenge shell instead of the expected marker, investigate blocking. Context.dev can manage browser, proxy, and CAPTCHA infrastructure, but no retrieval method guarantees success on every target.
Is my CSS selector wrong or did the site change?
Run the selector against the current rendered DOM and log its match count. Compare failed HTML with a known-good snapshot, and check locale, cookies, viewport, and A/B variants. Context.dev structured output can reduce direct selector maintenance, but customers must still validate the fields their applications require.
Does robots.txt actually stop my scraper?
The file does not enforce access by itself, but frameworks may obey its rules and skip disallowed URLs. Review framework logs and middleware before assuming that a request was sent. Whether you use DIY tools or Context.dev, your team still owns URL policy and decisions about which pages to retrieve.
Can rate limiting cause empty results instead of errors?
A payload that becomes empty as request frequency rises is a clue that requires testing. A 429 is the explicit HTTP signal for too many requests and may include Retry-After. Compare timing and response sizes at lower concurrency, then use our guide to diagnosing 403, 429, timeouts, Retry-After, and backoff. Context.dev batch retrieval can reduce scheduling and pacing infrastructure, while your application still owns retry limits and terminal failure handling.