A lists crawler collects repeated records from listing pages, follows pagination, and saves the results as structured data. Also called a list crawler, it turns product grids, job boards, directories, and search results into usable JSON or CSV for downstream application workflows. A reliable implementation tracks both the pages it visited and the items it extracted.
Use Beautiful Soup for a small static list, Scrapy for scheduled crawls with queues and pipelines, and Context.dev when you want managed page retrieval and schema-based extraction. Add browser interaction only when the records require JavaScript. Across all three approaches, your application still owns completeness checks, record identity, and downstream validation.
This guide builds those approaches around the same public practice catalog, then explains the production decisions that introductory scraping examples often skip: stopping pagination, recovering interrupted runs, preserving evidence, and detecting incomplete results.
What is list crawling?
List crawling is the process of traversing listing pages and extracting the repeated records they contain. A crawler discovers or schedules pages; an extractor reads the items on each page. Combining them produces a dataset such as products with prices, jobs with locations, or companies with profile URLs.
Consider a bookstore category containing twenty cards and a Next link. Reading those cards is list extraction. Following Next, extracting another batch, and merging the records is list crawling. Visiting each book's detail page for an ISBN is a separate enrichment stage.
| Technique | Main job | Typical output |
|---|---|---|
| General web crawling | Discover pages by following links | URLs or a page corpus |
| Single-page scraping | Read fields or content from one URL | One page's JSON, HTML, or Markdown |
| List crawling | Traverse listings and collect repeated entities | Products, jobs, events, or directory records |
| URL-list scraping | Process an existing collection of URLs | One result per supplied URL |
These distinctions affect architecture. If you already have ten thousand product URLs, you need a bounded batch job, not a spider that rediscovers the entire site. If you only have a category URL, you need a pagination strategy before you can claim coverage. Our URL discovery guide covers the discovery step separately.
Which websites are suitable for a lists crawler?
Good targets have repeated item containers, stable identifiers, and an understandable way to request more records. Product categories, public procurement notices, event calendars, and company directories often fit. The repeated element can be an HTML card, a table row, or an object returned by a documented API.
Start with the fields your application needs. A job index might provide title, company, location, and detail URL; salary may appear only on the detail page. Extract the index once, deduplicate the URLs, and enrich only the records requiring additional fields. This reduces redundant work without pretending the index contains information it never displayed.
Beautiful Soup vs. Scrapy vs. Context.dev
Choose based on page behavior and operational requirements, rather than assuming one tool produces better data in every situation. Deterministic selectors can extract exact values. Schema-based extraction can reduce site-specific parsing work, but valid JSON still needs factual and completeness checks.
| Approach | Best fit | What you operate | Main limitation |
|---|---|---|---|
| Requests + Beautiful Soup | A few stable, server-rendered lists | Fetch loop, selectors, pagination, retries | Does not execute JavaScript |
| Scrapy | Recurring crawls with scheduling and item processing | Spider rules, pipelines, deployment | Browser rendering needs additional integration |
| Playwright | Lists requiring browser interaction | Browser sessions, waits, resource limits | More moving parts than direct HTTP |
| Context.dev | Managed retrieval and structured extraction across sites | Schemas, page queue, validation, storage | API usage costs; extraction needs review |
Beautiful Soup parses markup; it does not fetch URLs or schedule a crawl. Scrapy supplies a crawler framework, while Context.dev supplies managed web-data APIs. They can also work together: a queue or Scrapy spider can schedule URLs, and a managed API can retrieve pages that require rendering. For broader tool selection, see our Python scraping libraries comparison.
Inspect the list before writing a crawler
Open the page, inspect one repeated item, and compare the browser DOM with the original HTTP response. If the item exists in both, direct HTTP is a reasonable starting point. If it appears only after interaction, inspect the network requests or use browser rendering. The Playwright network guide explains how to observe requests and responses.
Record four things before implementation: the item container, each field's location, the pagination mechanism, and the condition that proves you reached the end. Inspect both the first and second page. A selector that works only on page one is not a working list crawler.
Prefer an official API or export when available. Check the site's access policies and robots.txt, identify your client, and choose a conservative request rate. For authenticated sources, establish permission and session handling before scheduling a recurring job. Keep the initial target narrow enough that you can manually compare its records with the output.
Build a lists crawler with Python and Beautiful Soup
We will use Books to Scrape, a demonstration site designed for scraping practice. Its prices and ratings are synthetic. The sample starts at the catalog and follows Next links for at most three listing pages. It is deliberately a bounded demonstration, not a claim to extract the entire catalog.
Create an isolated Python environment and install the dependencies:
python -m venv .venv
source .venv/bin/activate
python -m pip install requests beautifulsoup4Save this as lists_crawler.py. On Windows, activate the environment with .venv\Scripts\activate instead.
import json
import time
from urllib.parse import urldefrag, urljoin, urlsplit
import requests
from bs4 import BeautifulSoup
START_URL = "https://books.toscrape.com/"
MAX_PAGES = 3
visited_pages = set()
records = {}
next_url = START_URL
with requests.Session() as session:
session.headers["User-Agent"] = "ListsCrawlerTutorial/1.0"
while next_url and len(visited_pages) < MAX_PAGES:
if urlsplit(next_url).netloc != urlsplit(START_URL).netloc:
raise ValueError("Pagination left the expected host")
if next_url in visited_pages:
raise RuntimeError("Pagination repeated a visited page")
response = session.get(next_url, timeout=(5, 30))
response.raise_for_status()
soup = BeautifulSoup(response.content, "html.parser")
cards = soup.select("article.product_pod")
if not cards:
raise RuntimeError(f"No book cards at {response.url}")
for card in cards:
link = card.select_one("h3 a")
price = card.select_one(".price_color")
if link is None or price is None:
raise ValueError("A book card is missing required fields")
item_url = urldefrag(urljoin(response.url, link["href"]))[0]
records[item_url] = {
"title": link["title"],
"url": item_url,
"price_text": price.get_text(strip=True),
"source_url": response.url,
}
visited_pages.add(next_url)
next_link = soup.select_one("li.next a")
next_url = urljoin(response.url, next_link["href"]) if next_link else None
if next_url:
time.sleep(1)
with open("books.json", "w", encoding="utf-8") as output:
json.dump(list(records.values()), output, ensure_ascii=False, indent=2)
print(f"Pages: {len(visited_pages)}; unique books: {len(records)}")
print("Stopped at page budget" if next_url else "Reached final page")Run python lists_crawler.py. The output is an array of book records with their source listing URLs. The script reads each link's full title attribute because the visible heading can be shortened. That is a source-specific extraction choice, not a requirement to repair Beautiful Soup output with an AI model.
The Beautiful Soup documentation describes CSS selection and attribute access. Requests handles retrieval; its timeout and status-error behavior is separate from parsing. Passing response bytes lets the parser inspect the document's encoding information instead of depending solely on an HTTP encoding guess.
What this example handles, and what to add next
The script resolves relative links against the current page, removes URL fragments for item identity, rejects pagination to a different host, detects repeated page URLs, and deduplicates books by detail URL. It retains price text exactly as displayed so normalization can happen explicitly later.
It does not implement automatic policy checks, retries, persistent checkpoints, or browser rendering. The host check applies to scheduled pagination links; Requests follows HTTP redirects automatically. A production crawler must validate redirected destinations too. These boundaries matter when adapting a trusted demonstration to sources your application does not control.
Keep page URLs and record identifiers in separate collections. Multiple pages can contain the same item, and several different URLs can return the same listing. A visited-page set alone cannot guarantee unique products; an item dictionary alone cannot detect every pagination loop.
Build a list crawler with Scrapy
Use Scrapy when you need a framework around scheduling, downloading, request filtering, and exported items. It is particularly useful when the task runs repeatedly and the work needs clear configuration rather than an expanding collection of custom loops.
Install Scrapy with python -m pip install scrapy, then save the following as lists_spider.py. Like the previous example, it intentionally schedules only three listing pages.
import scrapy
class BookListsSpider(scrapy.Spider):
name = "book_lists"
allowed_domains = ["books.toscrape.com"]
custom_settings = {
"ROBOTSTXT_OBEY": True,
"USER_AGENT": "ListsCrawlerTutorial/1.0",
"AUTOTHROTTLE_ENABLED": True,
"CONCURRENT_REQUESTS_PER_DOMAIN": 2,
"DOWNLOAD_DELAY": 1,
}
async def start(self):
yield scrapy.Request("https://books.toscrape.com/")
def parse(self, response, page_number=1):
cards = response.css("article.product_pod")
if not cards:
raise ValueError(f"No book cards at {response.url}")
for card in cards:
yield {
"title": card.css("h3 a::attr(title)").get(),
"url": response.urljoin(card.css("h3 a::attr(href)").get()),
"price_text": card.css(".price_color::text").get(),
"source_url": response.url,
}
next_page = response.css("li.next a::attr(href)").get()
if next_page and page_number < 3:
yield response.follow(
next_page,
callback=self.parse,
cb_kwargs={"page_number": page_number + 1},
)Run the fresh demonstration with:
scrapy runspider lists_spider.py -O books-scrapy.jsonl-O overwrites that output file. JSON Lines stores one object per line, which makes the export convenient to stream into a downstream validation or database stage. Review the spider's error statistics as well as the file: a crawl producing some items can still have failed pages.
The example uses the current asynchronous start() interface documented in Scrapy's spider reference. Its request duplicate filter prevents repeated requests, not duplicate business entities. Add an item pipeline or database uniqueness constraint when different listings can contain the same book.
Resume a larger crawl without starting over
For longer runs, Scrapy's JOBDIR persistence can retain scheduled requests, the duplicate filter, and supported spider state. Assign a separate directory to each crawl job, stop cleanly, and resume with the same Scrapy version and job configuration.
Coordinate persistence with your export strategy. Reusing -O while resuming overwrites previously exported items even though the scheduler remembers completed requests. Append to JSON Lines with -o, or write idempotently to a database keyed by record identity. A durable request queue and durable item storage solve different problems.
Enable AutoThrottle to adapt download delays, then monitor behavior per host. A concurrency setting limits simultaneous requests; it is not a promise about requests per second. Keep explicit budgets for pages, elapsed time, retries, and records so a malformed pagination rule cannot run indefinitely.
Extract list data with Context.dev
Context.dev is an alternative when you want managed retrieval and a consistent extraction contract across changing websites. The Scrape API reads one URL per request. Request structured output with formats.json, provide an object-root JSON Schema in jsonParams.schema, and inspect the returned json.success before reading json.data.
Use the same practice catalog for a direct comparison. Install pydantic alongside Requests, then set CONTEXT_DEV_API_KEY in your local environment or secret manager. Keep the real key out of source files. Save this example as managed_list.py.
import json
import os
import requests
from pydantic import BaseModel, Field, HttpUrl
class Book(BaseModel):
title: str = Field(min_length=1)
url: HttpUrl
price_text: str | None = None
class BookListing(BaseModel):
books: list[Book]
next_page_url: HttpUrl | None = None
response = requests.post(
"https://api.context.dev/v1/web/scrape",
headers={"Authorization": f"Bearer {os.environ['CONTEXT_DEV_API_KEY']}"},
json={
"url": "https://books.toscrape.com/",
"formats": {"json": True},
"jsonParams": {
"schema": BookListing.model_json_schema(),
"instructions": (
"Extract only book cards on this listing page. "
"Keep titles as stated; do not expand shortened text. "
"Preserve displayed price text; use null if absent. "
"Return absolute book URLs and the Next link, or null. "
"Do not invent records or follow links."
),
},
"maxAgeMs": 0,
},
timeout=(5, 120),
)
response.raise_for_status()
result = response.json()
extraction = result["json"]
if extraction.get("success") is not True:
raise RuntimeError(f"Extraction failed: {result.get('request_id')}")
if result.get("isPartial") or extraction.get("isTruncated"):
raise RuntimeError("Review this page: capture or extraction was incomplete")
listing = BookListing.model_validate(extraction["data"])
if not listing.books:
raise RuntimeError("No books extracted; inspect the source page")
output = listing.model_dump(mode="json")
output["source_url"] = result["url"]
output["request_id"] = result["request_id"]
print(json.dumps(output, ensure_ascii=False, indent=2))This is a single-page extraction example. To crawl the list, place validated next-page URLs into your own bounded queue, enforce the allowed host and path, and stop on repeated URLs or exhausted budgets. A returned Next URL is a candidate to verify against the source, not proof that every listing page was discovered.
The JSON extraction documentation explains missing-value behavior and input limits. Nullable fields represent absent facts without forcing an invented value. The response's top-level isPartial flag also marks incomplete output. Schema validation checks structure and types; it cannot establish that every product was found or every extracted price matches the page. Compare representative records with source evidence before production use.
For stable markup, deterministic CSS extraction may be simpler than model-based extraction. For known URL lists, asynchronous batches provide a managed scheduling option. Choose the output and workflow you need, then inspect their current limits and costs rather than assuming every API mode has identical behavior.
How does this compare with Firecrawl?
Firecrawl also supports page scraping and structured extraction through its documented Scrape interface. Evaluate both services with the same pages, requested fields, freshness requirements, and acceptance criteria. Do not compare a two-field selector demo against a seven-field managed extraction and attribute the difference to inherent data quality.
For Context.dev, keep retrieval, extraction success, and dataset validation as separate checks. For any provider, measure missing records, wrong values, latency, and total usage on your workload. Our Firecrawl comparison covers the broader product decision; this tutorial focuses on implementing the list workflow correctly.
Handle pagination, load-more buttons, and infinite scroll
Pagination is where list crawling most often becomes incomplete. The page mechanism determines what you schedule and what evidence counts as completion. Rendering a page once does not automatically traverse all of its hidden results.
| Listing mechanism | Retrieval strategy | Useful stopping evidence |
|---|---|---|
| Numbered pages | Follow verified Next links or documented page parameters | Final page indicator and no next link |
| Offset API | Request the next permitted offset | Documented terminal response |
| Cursor API | Pass the exact returned cursor | No next cursor |
| Load more | Observe its permitted request or click the button | End indicator and no new stable item IDs |
| Infinite scroll | Observe its data requests or scroll in bounded steps | Terminal signal, plus unchanged item IDs |
Numbered pages and cursors
Follow the Next link where possible instead of guessing that every site accepts ?page=2. Keep filter, locale, and sort parameters that define the dataset. If page two changes the sort order or drops a category filter, you may extract plausible records from the wrong collection.
Treat cursors as opaque values. Do not increment them, decode them into guessed offsets, or discard other parameters accompanying the cursor. Save the last acknowledged cursor with your crawl state. If it expires, follow the source's restart semantics and deduplicate records already committed.
Guard against soft pagination loops. Some sites return page one for any out-of-range page number, producing different requested URLs with identical items. Track a signature of stable item IDs as well as visited URLs, then flag repeated pages for review. An empty result is not automatically the end: it can also mean an expired session, block page, or changed selector.
Load more and infinite scroll
Prefer a permitted, documented data endpoint when it exists. Otherwise, use browser interaction and wait for a meaningful state change, such as new item IDs or the appearance of an end marker. Fixed sleeps alone do not demonstrate that the next batch loaded successfully.
With Context.dev, browser actions can interact before capture. These actions require a paid plan, bypass cache, and have documented action limits. Bound the interaction sequence. Capturing the rendered page after a few scrolls should be described as a partial capture unless you verified the list's end.
Watch for virtualized lists, which remove earlier cards from the DOM as you scroll. A final DOM snapshot may contain only the current viewport even after many items loaded. Capture each batch during traversal or use an authorized data endpoint, rather than treating a long scroll as proof of a complete dataset.
Deduplicate, normalize, and validate extracted records
Give every record a stable identity before loading it into a database. A source-provided product ID, requisition ID, or listing ID is usually preferable to a title. Where no ID exists, a carefully normalized detail URL can work. Preserve query parameters that distinguish product variants; remove tracking parameters only when you understand their meaning.
Separate the entity key from observations about that entity. A product's identity should stay stable when its price changes. Store the price, availability, source page, and observation timestamp as a new observation or versioned update. Otherwise, deduplication can accidentally erase the history your monitoring workflow needs.
Normalize deliberately. Keep currency beside the amount, use decimal arithmetic for prices, and retain the original displayed value. A comma can be a decimal separator or a thousands separator depending on locale. Removing every nondigit character turns both formatting ambiguity and missing values into incorrect numbers.
Apply required-field, URL, enum, and numeric constraints at the ingestion boundary. Pydantic strict mode can help where automatic coercion would hide bad inputs. In TypeScript, validate the response with Zod before persistence. Route rejected records into a review queue with their source and extraction version instead of quietly dropping them.
Measure quality with explicit denominators. If you reviewed fifty sampled records and two prices were wrong, report that sample result, not a universal accuracy claim. Track missing-field rates, duplicate rates, and records rejected per page. Preserve enough raw evidence to reproduce surprising values when a source or selector changes.
Make your lists crawler reliable in production
A production list crawl needs independent controls for completeness, resource use, and recovery. Start by separating listing discovery, listing extraction, detail enrichment, validation, and storage. Then give each stage an observable outcome. The production Python pipeline guide expands on this architecture.
Define completeness before scaling
A successful HTTP response is a transport outcome. A successful extraction is a parsing outcome. Neither proves that you collected every expected record. Define coverage using the source's displayed total when reliable, the number of acknowledged pages, terminal pagination evidence, and sampled comparisons with the source.
Store a run manifest containing the seed URL, filters, start and finish times, visited pages, failed pages, extracted records, unique entities, and stopping reason. Label a run partial when it reaches its budget or retains unresolved failures. Reconcile totals after deduplication, since overlapping pages can inflate raw extraction counts.
Retry fetch failures without hiding extraction failures
Retry transient network failures and eligible server errors with a capped policy. Honor Retry-After when provided, apply backoff with jitter, and respect explicit access blocks. A timeout, a rate limit, and a selector mismatch require different responses. Blindly retrying all three wastes requests and obscures the actual failure.
Checkpoint after durable writes. If a worker stores records and crashes before marking the page complete, replay should upsert the same entities. If it marks the page complete first, a crash can lose records permanently. A transactional database or equivalent acknowledgment design should connect the page outcome to committed output.
Set budgets around useful output
Budget listing pages, detail pages, browser time, API usage, and maximum retries independently. Estimate retrieval volume before enabling enrichment. If a hypothetical directory has one hundred listing pages and ten thousand unique detail URLs, visiting every detail page adds ten thousand fetches; it does not remain a hundred-request job.
Evaluate cost per usable record, including failed pages, duplicate work, extraction, and engineering maintenance. Cache stable content and refresh volatile facts on an appropriate schedule. Set maxAgeMs: 0 when a Context.dev request needs fresh retrieval; avoid forcing freshness for unchanged reference material without a reason. Verify current pricing for the outputs you enable.
Detect regressions before downstream data breaks
Keep saved page fixtures from different pagination positions and layouts. Assert that expected records and required fields still extract, then run a small controlled live check separately. Alert on sudden changes in item counts, null rates, source hosts, or price distributions. A syntactically valid dataset can still be the wrong dataset.
Retain the exact extraction schema or selectors with each run version. When counts change, compare source snapshots before changing business logic. This helps distinguish a legitimate catalog change from a rendering failure. Our guide to scrapers that survive layout changes covers maintaining those adapters.
Troubleshoot common list crawling failures
| Symptom | Likely cause | First check |
|---|---|---|
| Empty list with HTTP 200 | JavaScript shell, access page, or selector drift | Inspect returned content and expected item containers |
| Same records on every page | Ignored page parameter or soft loop | Compare final URLs and item-ID signatures |
| Fewer records than the site total | Missed pagination, filters, or virtualized content | Check the run manifest and terminal evidence |
| Duplicate records | Overlapping listings or unstable identity | Deduplicate by source ID or verified detail URL |
| Scrapy export loses earlier items | Resume used an overwrite export | Check -O versus append or database writes |
| Valid JSON contains incorrect facts | Extraction mapped the wrong fields | Compare records with saved source evidence |
| Costs rise unexpectedly | Unbounded enrichment or retry loops | Inspect per-stage request and usage budgets |
FAQs
What is a lists crawler?
A lists crawler is a program that collects repeated records from listing pages and follows pagination to produce structured data. Typical outputs include product catalogs, job listings, directories, and event datasets. The implementation combines page scheduling, field extraction, duplicate handling, validation, and storage.
Is list crawling the same as scraping a list of URLs?
They are related but different. List crawling discovers records across listing pages and their pagination. Scraping a list of URLs starts with already-known addresses. The latter usually needs bounded batch processing rather than link discovery, although both require validation and failure handling.
Can Beautiful Soup crawl JavaScript lists?
Beautiful Soup can parse HTML returned by a request or supplied by another tool, but it does not execute JavaScript. If records only appear after rendering, retrieve them through an authorized endpoint, a browser such as Playwright, or a managed rendering API before parsing.
Does Scrapy automatically remove duplicate items?
Scrapy normally filters duplicate requests using request fingerprints. That does not guarantee unique business records when different pages contain the same entity. Use an item pipeline or persistent uniqueness constraint keyed by a stable source identifier or canonical detail URL.
Does one Context.dev Scrape request crawl every page?
No. A Scrape request reads one URL. Your application must schedule additional listing and detail URLs, or choose a suitable managed crawl or batch workflow. Browser actions can reveal more content on that page, but a rendered capture does not establish complete pagination coverage.
How do I know a list crawl is complete?
Verify the source's terminal pagination signal, reconcile expected and unique record counts where available, and resolve failed pages. Record the stopping reason. Treat a budget-limited run, truncated extraction, or unexplained shortfall as partial rather than marking it complete because requests succeeded.
Build the smallest crawler that proves coverage
Start with one listing page, a modest field schema, and a manual comparison against the source. Add pagination with explicit stopping evidence, then persistence and enrichment. Use Beautiful Soup for a transparent static prototype, Scrapy for crawler orchestration, or Context.dev's scraping API for managed retrieval and extraction. Keep the same acceptance criteria whichever tool you choose.
For a quick retrieval check before designing a full dataset, try the free URL-to-Markdown tool. For structured records, continue with the JSON extraction guide, validate the output, and retain the source evidence. The useful result is a dataset you can explain and reproduce.