I Built a Wikipedia Scraper That Runs on Apify Without Proxies

Most Wikipedia scrapers on Apify use Playwright or headless browsers. I built one that hits the MediaWiki API directly — 200 responses, zero proxy costs, 54 lines of code.

Why Wikipedia Data?

Wikipedia is the largest open knowledge base on the internet. 60M+ articles across 300+ languages. Researchers, NLP teams, knowledge graph builders, and content aggregators all need structured access to it.

The problem: most tools overcomplicate it. They spin up headless browsers to render pages, manage proxy pools to avoid rate limits, and charge you for the overhead.

What This Actor Does

Search any keyword across any Wikipedia language. Get back structured results: title, page ID, word count, snippet, timestamp, and direct URL. Paginate up to 500 results per keyword with automatic offset handling.

Input:

keywords — array of search terms
lang — language code (default: en)
maxResults — up to 500 per keyword

Output per result:

{
  "keyword": "quantum computing",
  "lang": "en",
  "title": "Quantum computing",
  "pageid": 22967,
  "size": 85432,
  "wordcount": 12400,
  "snippet": "Quantum computing is a type of computation...",
  "timestamp": "2024-12-15T10:30:00Z",
  "url": "https://en.wikipedia.org/wiki/Quantum_computing"
}

The No-Proxy Advantage

Wikipedia's MediaWiki API is open. No authentication required. No rate limit tricks needed if you respect their guidelines.

This means:

Zero proxy costs — runs clean on Apify's AWS infrastructure
No browser overhead — plain HTTP GET, ~250ms per request
Reliable — no CAPTCHA challenges, no IP blocks, no flaky browser sessions

Most competing actors charge $3-5 per 1k results and burn proxy credits. This one runs at the Apify free tier.

Use Cases

NLP training data — bulk collect article metadata for text classification
Knowledge graph construction — page IDs + titles + timestamps for entity linking
Content gap analysis — compare coverage across languages
Research automation — systematic literature discovery on any topic
SEO research — find what Wikipedia covers about your niche

Running It

Search for "Wikipedia Search Scraper" on Apify Store, or use the API directly:

curl -X POST "https://api.apify.com/v2/acts/primesievecoder~wikipedia-search-scraper/runs" \
  -H "Content-Type: application/json" \
  -d '{"keywords": ["quantum computing", "machine learning"], "lang": "en", "maxResults": 100}'