Build a bulk product-image pipeline
- Discover and queue pages. Start with your catalog URLs or the Sitemap API. Deduplicate the URL list, limit concurrent requests, and save progress so a failed page does not restart the whole collection.
- Extract and select images. Request dimensions and asset classification to filter candidates. Use wait controls or browser actions for dynamic galleries. Set
maxAgeMs: 0when a catalog refresh needs a fresh scrape. - Prepare the AI dataset. Download and validate selected images, retain the source page and collection date, and remove duplicates across stores. The API's
dedupeoption operates within each page. Follow the computer vision example to filter images before ingestion.













