Updated

Updated · KDnuggets · Nov 5

Abid Ali Awan Scrapes 9,938 of 10,000 Pages in 22 Minutes With Python and Olostep

Updated

Updated · KDnuggets · Nov 5

Abid Ali Awan Scrapes 9,938 of 10,000 Pages in 22 Minutes With Python and Olostep

Q: Why did a web scraping job that processed thousands of pages in minutes take over 22 minutes to actually finish?

It took more than 22 minutes because the fast headline number measured only the scraping phase, not the full end-to-end workflow. In the test, Olostep’s Batch API processed 7,225 MDN pages in under three minutes, showing strong early throughput. But that was only the point at which the remote batch job had moved through much of its queue. The developer still had to wait for the batch to fully complete, then download thousands of returned pages, save them as individual Markdown files, and verify that the saved files matched the original 10,000-URL request list. Those downstream steps added most of the elapsed time. Fetching and writing nearly 10,000 results creates network, disk, and concurrency overhead that is separate from the scraper’s own processing speed. Verification also matters because a page counted as “processed” is not automatically a usable output. The final audit found 9,938 Markdown files saved and 62 missing, which shows why post-scrape validation cannot be skipped. The broader lesson is that large-scale scraping performance should be judged by end-to-end delivery, not by early progress metrics alone. Olostep appears to have performed in line with its design for high-volume asynchronous jobs, but the benchmark showed that real completion in an enterprise pipeline includes collection, retrieval, storage, and quality checks before the job is actually finished.

1 articles · Updated · KDnuggets · Nov 5

9,938 of 10,000 MDN documentation pages were saved as Markdown in Abid Ali Awan’s Python test using Olostep’s Batch API, with the full scrape-download-verify workflow taking 22 minutes 26 seconds.
7,225 pages were processed in about 2 minutes 52 seconds, but the remaining pages took far longer, showing that early throughput overstated end-to-end completion speed.
62 pages were missing at the end, leaving 99.38% file coverage and underscoring Awan’s point that a completed batch does not guarantee every requested page was actually saved.
Olostep’s batch workflow let Awan submit 10,000 URLs in one job, monitor progress, then download results concurrently; the Markdown download stage alone took about 2 minutes 25 seconds.
The test framed batch scraping as useful for enterprise data pipelines, but concluded that reliability checks and total workflow timing matter as much as headline scraping speed.

Sources

KDnuggets1h ago

Abid Ali Awan Scrapes 9,938 Web Pages in 22 Minutes with Python, Olostep API

Why did a web scraping job that processed thousands of pages in minutes take over 22 minutes to actually finish?

If an API scrapes 7,000 pages in three minutes, what hidden bottlenecks are secretly slowing down enterprise data pipelines?

Sources

1 total

KDnuggets1h ago

Abid Ali Awan Scrapes 9,938 Web Pages in 22 Minutes with Python, Olostep API

Abid Ali Awan Scrapes 9,938 of 10,000 Pages in 22 Minutes With Python and Olostep

Summary

Sources

Insights

Sources