TL;DR
- Define a canonical Pydantic schema before writing site adapters so every record follows the same field contract.
- Parse JSON-LD first. Detect Shopify, WooCommerce, Magento, or BigCommerce before applying platform-specific selector fallbacks.
- Score each field by its source and raise confidence when independent extraction paths agree.
- Normalize prices, currencies, availability, images, and variants before Pydantic validates the record.
- Render JavaScript only when static fetching fails. Add targeted retries and confidence-drift monitoring. A managed API such as Context.dev can handle rendering, proxies, and anti-bot maintenance.
Why product data extraction breaks at multi-site scale
A scraper built for one store usually encodes assumptions about one template. The script may expect a title under .product-title, a price under .price, and a single image in a known container. Those assumptions stop holding when the target list expands across different platforms, themes, locales, and product types.
Site count understates the maintenance problem because one domain can expose several product templates. A retailer may use separate markup for standard products, bundles, subscriptions, and products with variants. Regional storefronts may change currency notation or translate availability labels. A theme update can then alter class names across every template without changing the visible page.
Fixed CSS selectors also produce silent errors. A price selector may still match after a redesign but capture the crossed-out list price instead of the current sale price. A title selector may return a recommendation card when the main product container moves. The scraper completes successfully, yet the stored record becomes wrong.
A layered extractor limits these failures by consulting sources with different failure modes. JSON-LD provides explicit product fields when the merchant publishes valid structured data. Platform detection can select known Shopify or WooCommerce patterns. Generic microdata and heuristic selectors can fill fields that stronger sources omit.
Field-level confidence scores preserve the difference between those paths. A price found in a valid Product offer should receive more trust than a number inferred from visible text. Agreement between both sources can raise the score, while conflicting values can send the record to review. The resulting pipeline treats extraction as evidence gathering rather than a single selector lookup.
Architecture for a multi-site product extraction pipeline
A production pipeline should separate page retrieval, extraction, normalization, and validation. Each stage can then report its own failure type, and you can replace one implementation without rewriting the rest of the pipeline.
[Retry and recovery controller]
| ^
v |
[URL queue] -> [Fetch or render] -> [Platform detector]
|
v
[JSON-LD and microdata parser]
|
v
[Selector fallback chain]
|
v
[Confidence scorer]
|
v
[Normalizer]
|
v
[Pydantic validator]
|
v
[Storage and monitoring]The fetch layer retrieves static HTML first and invokes browser rendering only when the response lacks usable product content. The platform detector then looks for signals associated with Shopify, WooCommerce, Magento, or BigCommerce. A detected platform changes which selectors and embedded data sources the extractor tries.
The structured-data parser inspects JSON-LD and microdata before the selector chain reads presentation markup. Each extraction attempt records the value, source, and supporting evidence. The confidence scorer can therefore prefer a JSON-LD price while retaining a lower-confidence DOM value for comparison.
The normalizer converts accepted candidates into canonical types such as decimal prices, currency codes, absolute image URLs, and standardized availability values. Pydantic checks the resulting object against the product schema. Invalid records can trigger another extraction path, a rendered refetch, or quarantine rather than entering storage.
Retries should wrap retrieval and respond to validation failures without blindly repeating every request. Network errors may justify another fetch, while missing required fields may justify rendering or a different adapter. Monitoring should record extraction success and field confidence by domain so template drift appears before it contaminates a large batch.
The architecture follows one rule. Never trust one extraction path, and score every value you keep. That rule lets you add site-specific adapters without making them the sole source of truth.
Defining a canonical product schema with Pydantic
A canonical model gives every site the same output contract. Your downstream code can compare prices, filter availability, and index products without knowing whether the source used Shopify fields, schema.org properties, or custom HTML.
The model should require fields that identify and price a product. Optional fields can remain absent when a page does not publish them. Pydantic then provides type conversion and rejects records that normalization cannot repair.
import re
from decimal import Decimal, InvalidOperation
from typing import Literal
from pydantic import (
AnyHttpUrl,
BaseModel,
Field,
field_validator,
)
Availability = Literal[
"in_stock",
"out_of_stock",
"preorder",
"backorder",
"unknown",
]
class ProductVariant(BaseModel):
sku: str | None = None
title: str
price: Decimal | None = Field(default=None, ge=0)
availability: Availability = "unknown"
class Product(BaseModel):
source_url: AnyHttpUrl
title: str = Field(min_length=1)
price: Decimal = Field(ge=0)
currency: str = Field(min_length=3, max_length=3)
availability: Availability = "unknown"
images: list[AnyHttpUrl] = Field(default_factory=list)
brand: str | None = None
sku: str | None = None
variants: list[ProductVariant] = Field(default_factory=list)
@field_validator("title", "brand", "sku", mode="before")
@classmethod
def clean_text(cls, value):
if value is None:
return None
cleaned = re.sub(r"\s+", " ", str(value)).strip()
return cleaned or None
@field_validator("price", mode="before")
@classmethod
def parse_price(cls, value):
if isinstance(value, (int, float, Decimal)):
return value
cleaned = re.sub(
r"[^\d.\-]",
"",
str(value).replace(",", ""),
)
try:
return Decimal(cleaned)
except (InvalidOperation, ValueError):
raise ValueError(f"Invalid price value {value!r}")
@field_validator("currency", mode="before")
@classmethod
def normalize_currency(cls, value):
if value is None or not str(value).strip():
raise ValueError("Currency is required")
currency = str(value).strip().upper()
if not re.fullmatch(r"[A-Z]{3}", currency):
raise ValueError("Currency must use a three-letter code")
return currencyThe price validator converts values such as "$1,299.00" into Decimal("1299.00"). The currency validator rejects missing values and ambiguous symbols such as $, since the symbol could mean USD, CAD, AUD, or another currency. Convert symbols only when the page, domain, or offer metadata supplies enough context.
Keep extraction separate from validation. Each adapter should produce a raw dictionary, normalization should map source values into canonical forms, and Product.model_validate(raw_product) should enforce the final contract. Failed records can then enter a quarantine queue with their source URL and extraction evidence instead of silently contaminating storage.
JSON-LD-first extraction and why it should be the primary path
JSON-LD should receive the highest initial confidence because publishers use it to describe products to search engines and other machines. Its field names remain more stable than presentation classes, and an Offer often provides price, currency, availability, SKU, and image data without DOM-specific selectors.
A page may contain several JSON-LD scripts. Each script can hold one object, an array, or an @graph containing multiple entities. The parser therefore needs to inspect every block and select nodes whose @type includes Product.
import json
from collections.abc import Iterator
from typing import Any
from bs4 import BeautifulSoup
def iter_jsonld_nodes(payload: Any) -> Iterator[dict]:
if isinstance(payload, list):
for item in payload:
yield from iter_jsonld_nodes(item)
return
if not isinstance(payload, dict):
return
yield payload
graph = payload.get("@graph")
if graph is not None:
yield from iter_jsonld_nodes(graph)
def has_type(node: dict, expected: str) -> bool:
node_types = node.get("@type", [])
if isinstance(node_types, str):
node_types = [node_types]
return expected in node_types
def first_offer(product: dict) -> dict:
offers = product.get("offers") or {}
if isinstance(offers, list):
return offers[0] if offers else {}
return offers if isinstance(offers, dict) else {}
def image_urls(value: Any) -> list[str]:
values = value if isinstance(value, list) else [value]
urls = []
for item in values:
if isinstance(item, str):
urls.append(item)
elif isinstance(item, dict) and item.get("url"):
urls.append(item["url"])
return urls
def extract_jsonld_products(
html: str,
source_url: str,
) -> list[dict]:
soup = BeautifulSoup(html, "html.parser")
products = []
for script in soup.select('script[type="application/ld+json"]'):
try:
payload = json.loads(script.get_text(strip=True))
except (json.JSONDecodeError, TypeError):
continue
for node in iter_jsonld_nodes(payload):
if not has_type(node, "Product"):
continue
offer = first_offer(node)
brand = node.get("brand")
if isinstance(brand, dict):
brand = brand.get("name")
availability = offer.get("availability", "")
availability = availability.rsplit("/", 1)[-1]
products.append({
"source_url": source_url,
"title": node.get("name"),
"price": offer.get("price")
or offer.get("lowPrice"),
"currency": offer.get("priceCurrency"),
"availability": availability,
"images": image_urls(node.get("image")),
"brand": brand,
"sku": node.get("sku") or offer.get("sku"),
"raw_jsonld": node,
})
return productsJSON-LD coverage varies by market, platform, and page type, so no universal percentage describes how many sites provide usable product records. Measure coverage against your own domain set and distinguish between JSON-LD presence and records that contain the required fields. Pages with missing prices, stale offers, malformed scripts, or no Product node still need platform adapters and selector fallbacks.
Detecting the underlying platform: Shopify, WooCommerce, Magento, BigCommerce
Platform fingerprints let the extractor choose selectors that match a known storefront family. The detector should combine several signals because merchants can remove generator tags, rename CSS classes, or serve assets through a custom domain.
Shopify pages often reference cdn.shopify.com, expose a Shopify JavaScript object, or use product classes such as product__title. A successful probe of /products.json?limit=1 provides another signal, although some stores disable that endpoint. WooCommerce usually leaves WordPress plugin paths, woocommerce body classes, or woocommerce-Price-amount elements. Magento commonly exposes data-mage-init, /static/version asset paths, or Magento_ module names. BigCommerce storefronts often reference stencil-utils, BigCommerce CDN hosts, or productView classes.
import re
from enum import Enum
from bs4 import BeautifulSoup
class Platform(str, Enum):
SHOPIFY = "shopify"
WOOCOMMERCE = "woocommerce"
MAGENTO = "magento"
BIGCOMMERCE = "bigcommerce"
UNKNOWN = "unknown"
SIGNALS = {
Platform.SHOPIFY: [
r"cdn\.shopify\.com",
r"\bShopify\.(theme|routes|currency)\b",
r'class="[^"]*\bproduct__title\b',
],
Platform.WOOCOMMERCE: [
r"wp-content/plugins/woocommerce",
r"\bwoocommerce-Price-amount\b",
r'class="[^"]*\bwoocommerce\b',
],
Platform.MAGENTO: [
r"data-mage-init",
r"/static/version\d+/",
r"\bMagento_[A-Za-z]+",
],
Platform.BIGCOMMERCE: [
r"stencil-utils",
r"cdn\d+\.bigcommerce\.com",
r"\bproductView-(title|price|images)\b",
],
}
def detect_platform(html: str) -> Platform:
soup = BeautifulSoup(html, "html.parser")
generator = soup.select_one('meta[name="generator"]')
generator_text = generator.get("content", "") if generator else ""
searchable = f"{generator_text}\n{html}"
scores = {
platform: sum(
bool(re.search(pattern, searchable, re.I))
for pattern in patterns
)
for platform, patterns in SIGNALS.items()
}
winner = max(scores, key=scores.get)
return winner if scores[winner] > 0 else Platform.UNKNOWNProduction detectors can add a low-cost endpoint probe after passive fingerprinting. For example, a Shopify candidate can receive another vote when /products.json?limit=1 returns product-shaped JSON. Endpoint errors should not overturn strong HTML evidence because store settings and access controls vary.
The detected enum selects the first handler in the fallback chain. An unknown result skips platform selectors and proceeds directly to generic microdata and DOM heuristics.
Building selector fallback chains for inconsistent HTML
A fallback chain gives each field several extraction paths without creating one oversized selector list. Each handler returns candidates with provenance, including failed attempts. Retaining every attempt lets the confidence scorer compare independent sources instead of accepting the first nonempty string.
import re
from dataclasses import dataclass
from typing import Callable
from bs4 import BeautifulSoup, Tag
@dataclass
class Attempt:
field: str
value: str | None
source: str
success: bool
PLATFORM_SELECTORS = {
Platform.SHOPIFY: {
"title": ".product__title",
"price": ".price-item--regular",
},
Platform.WOOCOMMERCE: {
"title": ".product_title",
"price": ".woocommerce-Price-amount",
},
Platform.MAGENTO: {
"title": ".page-title",
"price": ".product-info-main .price",
},
Platform.BIGCOMMERCE: {
"title": ".productView-title",
"price": ".productView-price .price",
},
}
def text_or_content(node: Tag | None) -> str | None:
if not node:
return None
value = node.get("content") or node.get_text(" ", strip=True)
return value.strip() if value and value.strip() else None
def platform_attempts(
soup: BeautifulSoup, platform: Platform
) -> list[Attempt]:
selectors = PLATFORM_SELECTORS.get(platform, {})
results = []
for field in ("title", "price"):
node = soup.select_one(selectors.get(field, "__missing__"))
value = text_or_content(node)
results.append(Attempt(
field=field,
value=value,
source=f"platform.{platform.value}",
success=value is not None,
))
return results
def microdata_attempts(soup: BeautifulSoup) -> list[Attempt]:
selectors = {
"title": '[itemprop="name"]',
"price": '[itemprop="price"]',
}
return [
Attempt(field, text_or_content(soup.select_one(selector)),
"microdata", bool(soup.select_one(selector)))
for field, selector in selectors.items()
]
def inline_font_size(node: Tag) -> int:
match = re.search(
r"font-size\s*:\s*(\d+)", node.get("style", ""), re.I
)
return int(match.group(1)) if match else 0
def heuristic_attempts(soup: BeautifulSoup) -> list[Attempt]:
title = text_or_content(
soup.select_one('meta[property="og:title"]')
or soup.select_one("h1")
)
price_nodes = soup.select('[class*="price" i]')
price_node = max(
price_nodes,
key=lambda node: (
inline_font_size(node),
len(node.get_text(" ", strip=True)),
),
default=None,
)
price = text_or_content(price_node)
return [
Attempt("title", title, "heuristic", title is not None),
Attempt("price", price, "heuristic", price is not None),
]
def run_fallback_chain(
html: str, platform: Platform
) -> list[Attempt]:
soup = BeautifulSoup(html, "html.parser")
handlers: list[Callable[[], list[Attempt]]] = [
lambda: platform_attempts(soup, platform),
lambda: microdata_attempts(soup),
lambda: heuristic_attempts(soup),
]
attempts = []
for handler in handlers:
attempts.extend(handler())
return attemptsThe example ranks price-like elements by inline font size when available. A browser-backed extractor can improve that heuristic with computed styles and element area. Static HTML cannot reliably determine visual prominence when styles come from external CSS.
JSON-LD candidates from the primary extraction path should join this attempt list before scoring. Running all cheap handlers costs slightly more CPU, but the extra evidence exposes disagreements that a stop-on-success chain would hide.
Field-level confidence scoring
Field-level scores let the pipeline accept a reliable title while quarantining an uncertain price from the same page. Source quality supplies the base score, and agreement between independent extractors adds confidence. Conflicts can reduce the score because two plausible but different prices may represent a sale price, a variant price, or an extraction error.
A practical starting scale assigns 0.95 to JSON-LD, 0.85 to platform selectors, 0.75 to microdata, and 0.45 to heuristics. You should tune those values against labeled samples rather than treating them as universal probabilities.
import re
from collections import defaultdict
from typing import Any
def base_score(source: str) -> float:
if source == "json_ld":
return 0.95
if source.startswith("platform."):
return 0.85
if source == "microdata":
return 0.75
return 0.45
def comparable(field: str, value: str) -> str:
compact = " ".join(value.lower().split())
if field == "price":
match = re.search(r"\d[\d,.]*", compact)
return match.group(0).replace(",", "") if match else compact
return compact
def assemble_raw(attempts: list[Attempt]) -> dict[str, Any]:
by_field: dict[str, list[Attempt]] = defaultdict(list)
for attempt in attempts:
if attempt.success and attempt.value:
by_field[attempt.field].append(attempt)
raw: dict[str, str] = {}
confidence: dict[str, float] = {}
provenance: dict[str, str] = {}
for field, candidates in by_field.items():
winner = max(candidates, key=lambda item: base_score(item.source))
winner_value = comparable(field, winner.value)
peers = [item for item in candidates if item is not winner]
agreements = sum(
comparable(field, item.value) == winner_value
for item in peers
)
conflicts = len(peers) - agreements
score = base_score(winner.source)
score += min(0.10, agreements * 0.05)
score -= min(0.10, conflicts * 0.03)
raw[field] = winner.value
confidence[field] = round(max(0.0, min(1.0, score)), 2)
provenance[field] = winner.source
return {
"raw": raw,
"confidence": confidence,
"provenance": provenance,
}The comparison function performs only enough cleanup to detect agreement before formal normalization. The later normalization stage still owns currency parsing, decimal conversion, availability mapping, and schema validation.
Acceptance thresholds should reflect field risk. A low-confidence description may remain useful, while a low-confidence price should usually enter a review or retry queue. Storing provenance beside each score also lets monitoring identify whether a site has fallen from a platform selector to a heuristic after a redesign.
Normalization and validation before data leaves the pipeline
Normalization converts extracted strings into the types and vocabulary required by the canonical schema. Prices need decimal values and ISO currency codes, while availability values need a small shared vocabulary such as in_stock, out_of_stock, and preorder. Unit labels such as kilograms, kg, and KG should resolve to one canonical value.
import re
from decimal import Decimal, InvalidOperation
from typing import Any
CURRENCY_SYMBOLS = {
"$": "USD",
"€": "EUR",
"£": "GBP",
"¥": "JPY",
}
AVAILABILITY = {
"instock": "in_stock",
"in stock": "in_stock",
"outofstock": "out_of_stock",
"out of stock": "out_of_stock",
"preorder": "preorder",
"pre-order": "preorder",
}
UNITS = {
"kilogram": "kg",
"kilograms": "kg",
"kg": "kg",
"gram": "g",
"grams": "g",
"g": "g",
}
def parse_price(value: Any) -> Decimal | None:
if value is None:
return None
cleaned = re.sub(r"[^\d,.\-]", "", str(value))
if "," in cleaned and "." in cleaned:
cleaned = cleaned.replace(",", "")
elif cleaned.count(",") == 1:
left, right = cleaned.split(",")
cleaned = f"{left}.{right}" if len(right) == 2 else left + right
try:
return Decimal(cleaned)
except InvalidOperation:
return None
def normalize_availability(value: Any) -> str | None:
if not value:
return None
token = str(value).rsplit("/", 1)[-1].replace("_", " ").lower()
return AVAILABILITY.get(token.replace(" ", ""), AVAILABILITY.get(token))
def normalize_unit(value: Any) -> str | None:
return UNITS.get(str(value).strip().lower()) if value else NoneA normalization failure should reduce trust in the affected field rather than terminate the whole job. The normalizer can set an invalid price to None, reduce its confidence to zero, and preserve the raw value for debugging. Currency symbols also require care because $ can represent several currencies. Domain configuration, locale metadata, or an explicit JSON-LD currency should outrank symbol-based inference.
Pydantic performs the final contract check after normalization. A production pipeline should retain the candidate, confidence map, source URL, and validation errors when validation fails.
from pydantic import ValidationError
def validate_record(raw: dict) -> tuple[Product | None, dict]:
confidence = raw["confidence"].copy()
price = parse_price(raw["fields"].get("price"))
if price is None:
confidence["price"] = 0.0
candidate = {
**raw["fields"],
"price": price,
"availability": normalize_availability(
raw["fields"].get("availability")
),
}
try:
product = Product.model_validate(candidate)
return product, confidence
except ValidationError as exc:
quarantine = {
"url": raw["url"],
"candidate": candidate,
"raw_fields": raw["fields"],
"confidence": confidence,
"errors": exc.errors(),
}
save_quarantine(quarantine)
return None, confidenceQuarantine is the safest default for records missing required fields such as title or currency. Dropping records suits known duplicates or clearly irrelevant pages. Partial acceptance works when required fields validate and optional fields can be removed without changing the product’s identity, but the stored record should identify every omitted field.
Handling JavaScript-rendered product pages
Headless rendering belongs behind the static fetch path because most usable product pages do not require a browser. A normal HTTP request may already expose JSON-LD, microdata, server-rendered HTML, or an embedded application state object. Starting Playwright for those pages adds latency and resource use without improving extraction.
Browser rendering becomes appropriate when the static response contains only an application shell, when product data appears after JavaScript executes, or when interaction triggers the required content. The pipeline should make that decision using concrete signals. Examples include an empty product container, missing fields across every extraction path, or scripts that reference client-side product APIs.
Playwright also needs an explicit stopping condition. Waiting for networkidle can hang on pages with analytics, chat widgets, or continuous requests. Waiting for a known product selector or response endpoint usually gives more predictable behavior, followed by a bounded timeout.
Browser infrastructure becomes expensive to operate across hundreds of sites. You need browser pools, concurrency limits, process recycling, timeout handling, and memory controls. Browser updates can change page behavior, while blocks may require proxy rotation and fingerprint maintenance. A global semaphore should cap browser sessions, and a per-domain limit should prevent one site from consuming the pool.
A managed rendering layer can own those browser and network concerns while your code retains the canonical schema, confidence rules, and storage policy. Context.dev fits at the fetch and render boundary when you want rendered pages or schema-shaped output without maintaining browsers, proxies, and anti-bot components. Custom Playwright remains useful when extraction depends on logged-in sessions, unusual interactions, or precise control over browser state.
Retries, concurrency, and failure monitoring in production
Production retries should respond to the failure category instead of repeating every request with the same settings. Transient network errors support exponential backoff with jitter. A block response should switch fetch strategy, proxy route, or rendering mode. Validation failures should enter quarantine because downloading the same HTML again rarely repairs malformed data.
import asyncio
import random
async def fetch_with_retry(url: str, client, attempts: int = 4):
mode = "http"
for attempt in range(attempts):
try:
response = await client.fetch(url, mode=mode)
if response.status in {403, 429}:
raise BlockedError(response.status)
if response.status >= 500:
raise NetworkError(response.status)
return response
except BlockedError:
if mode == "http":
mode = "rendered"
continue
raise
except NetworkError:
if attempt == attempts - 1:
raise
delay = min(30, 2 ** attempt) + random.random()
await asyncio.sleep(delay)
raise RuntimeError(f"Fetch attempts exhausted for {url}")Concurrency needs both global and per-site limits. The global limit protects sockets, memory, and browser capacity, while the per-site limit reduces rate spikes against one domain. Separate queues for static requests and rendered requests prevent slow browser jobs from blocking inexpensive HTTP fetches.
Monitoring should track extraction quality as well as request outcomes. Useful per-site measurements include fetch success rate, validation success rate, field completeness, and median confidence for important fields. Store those measurements in time windows so the current period can be compared with the site’s recent baseline.
from collections import defaultdict
from statistics import median
history = defaultdict(list)
def record_run(site: str, ok: bool, confidence: dict[str, float]) -> None:
history[site].append({
"ok": ok,
"confidence": confidence,
})
def site_health(site: str) -> dict:
runs = history[site][-100:]
success_rate = sum(run["ok"] for run in runs) / max(len(runs), 1)
price_scores = [
run["confidence"].get("price", 0.0)
for run in runs
]
return {
"success_rate": success_rate,
"median_price_confidence": median(price_scores) if price_scores else 0.0,
}Confidence drift catches failures that HTTP monitoring misses. A redesigned site may continue returning status 200 while the title falls back to og:title, the price disappears, and validation still accepts a partial record. Alerts should compare each field against its own historical baseline and route affected samples to quarantine for inspection.
Where a managed extraction layer like Context.dev fits
A managed extraction layer replaces the parts of this architecture that create the most operational work. Context.dev handles page retrieval, JavaScript rendering, crawling, and structured data delivery through one API. You can send product URLs and receive clean JSON without running Playwright workers, rotating proxies, or maintaining browser fingerprints.
Schema-shaped output also reduces the number of site-specific adapters you need to maintain. Context.dev absorbs differences in rendering and page structure before returning data to your Python pipeline. Your code can focus on mapping the response into the canonical product model, applying business-specific normalization, and validating records with Pydantic.
Managed extraction does not remove the need for validation. Product fields can still be absent, ambiguous, or unsuitable for your schema. You should preserve field-level confidence scores, quarantine invalid records, and monitor changes in completeness. Consistent API output makes those controls easier to operate because fewer site-specific parsing rules sit between retrieval and validation.
An in-house pipeline gives you full control over selectors, request behavior, source precedence, and unusual product models. That control can justify the maintenance cost when extraction logic provides a competitive advantage or when compliance rules require direct infrastructure ownership. Context.dev fits teams that need faster production deployment and prefer not to own rendering, proxy rotation, anti-bot handling, retries, concurrency, and adapter maintenance.
Conclusion
Durable multi-site extraction depends on layered evidence and field-level confidence. Adding more CSS selectors may repair individual sites, but it cannot detect conflicting values or silent degradation across hundreds of domains.
Build the pipeline in-house when you need complete control and can support its ongoing maintenance. Choose a managed layer when structured product data matters more than owning crawler infrastructure. Readers still choosing foundational tools should consult our broad Python scraping library guide. Scrapfly’s multi-source product pipeline article provides another technical deep dive into this architecture.
FAQs
How should I handle a site with no JSON-LD and no recognizable platform?
Run generic microdata and DOM heuristics, then assign lower confidence to every inferred field. Require corroboration for sensitive values such as price and availability. Send records below your acceptance threshold to a review queue instead of silently publishing them.
How often should I re-check confidence scores after a site redesign?
Recalculate scores on every extraction. Compare rolling field scores and completeness rates against each site’s recent baseline. A sudden drop after a redesign should trigger an alert and a saved-page investigation, even when Pydantic still accepts the record.
Should I store raw HTML alongside parsed product output?
Store raw HTML or a compressed snapshot when storage rules permit it. Snapshots let you reproduce parsing failures, test new selectors, and audit the source behind a value. Apply a retention limit because product pages can contain personalization, location signals, or other data you do not need indefinitely.
How does this guide differ from a general Python scraping guide?
A general guide helps you choose tools such as Requests, BeautifulSoup, Parsel, Scrapy, or Playwright. This architecture addresses a narrower production problem. It explains how to turn inconsistent product pages into one validated schema while tracking confidence and extraction failures.
When should I choose a managed API instead of this in-house pipeline?
Choose a managed API when browser operations, blocking, proxy rotation, and site-specific maintenance consume more engineering time than your extraction rules justify. Context.dev suits pipelines that need rendered pages and consistent structured output without owning that infrastructure. Keep the pipeline in-house when custom request behavior, persistent sessions, or specialized parsing logic outweigh the operational savings.