You finally set up that slick AI-powered web scraping pipeline. It pulls in articles, summarizes them, and dumps a beautiful daily digest into your inbox every morning. Life is good.
Then you try to use the same pipeline for anything else — extracting product specs, monitoring competitor pricing, pulling structured data from documentation sites — and it falls apart spectacularly. Broken selectors, hallucinated fields, timeouts, and outputs that look nothing like what you expected.
I've been there. After spending way too many hours debugging these pipelines, I finally understand why news digests work so reliably while everything else feels like building on quicksand.
The root cause: news articles are the easy mode of web scraping
Here's the thing most people don't realize. News articles have a set of properties that make them uniquely scrapable:
- Consistent DOM structure — most news sites use article tags, semantic HTML, and predictable layouts
- Plain text heavy — the content you want is mostly paragraphs, not nested tables or interactive widgets
- RSS/Atom feeds — many news sources literally give you structured data for free
- Standardized metadata — OpenGraph tags, JSON-LD, Dublin Core — news sites are SEO-obsessed and mark everything up properly
- Low JavaScript dependency — article content usually renders server-side
When you point an LLM-based scraper at a news site and say "summarize this," you're basically asking it to read clean text and produce clean text. That's LLM home turf.
But when you point the same pipeline at a SaaS pricing page, an e-commerce product listing, or a dynamically-rendered dashboard, you're asking it to parse JavaScript-rendered content through a straw.
The three failure modes I keep hitting
1. JavaScript-rendered content that never arrives
The most common failure. Your scraper fetches the HTML, but the actual content loads via JavaScript after the initial page render.
import requests
from bs4 import BeautifulSoup
# This gets you an empty shell on most modern sites
response = requests.get("https://example.com/products")
soup = BeautifulSoup(response.text, "html.parser")
products = soup.find_all(".product-card") # Returns empty list
The fix isn't complicated, but it does add overhead. You need a headless browser:
from playwright.sync_api import sync_playwright
def scrape_with_js(url, wait_selector=None, timeout=15000):
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url, timeout=timeout)
if wait_selector:
# Don't just wait for load — wait for the actual content
page.wait_for_selector(wait_selector, timeout=timeout)
content = page.content()
browser.close()
return content
# Now you actually get the rendered DOM
html = scrape_with_js(
"https://example.com/products",
wait_selector=".product-card" # wait until products render
)
The key insight: always specify a wait selector for the content you actually care about. Waiting for load or networkidle is unreliable because SPAs never truly stop making network requests.
2. LLM extraction that hallucinates structure
This one burned me hard. I had a pipeline that used an LLM to extract structured JSON from scraped pages. It worked great in testing, then started returning confidently wrong data in production.
The problem: when the page structure changes slightly (A/B tests, layout updates, seasonal redesigns), the LLM doesn't fail gracefully — it guesses. And LLM guesses look exactly like real data.
import json
def extract_with_validation(llm_output, schema):
"""Don't trust LLM extraction blindly — validate against a schema."""
try:
data = json.loads(llm_output)
except json.JSONDecodeError:
return None, "LLM returned invalid JSON"
errors = []
for field, rules in schema.items():
if rules.get("required") and field not in data:
errors.append(f"Missing required field: {field}")
if field in data and "type" in rules:
if not isinstance(data[field], rules["type"]):
errors.append(f"Wrong type for {field}")
if field in data and "pattern" in rules:
import re
if not re.match(rules["pattern"], str(data[field])):
errors.append(f"Field {field} doesn't match expected pattern")
if errors:
return None, errors
return data, None
# Define what valid output looks like
product_schema = {
"name": {"required": True, "type": str},
"price": {"required": True, "type": (int, float)},
"currency": {"required": True, "type": str, "pattern": r"^[A-Z]{3}$"},
}
result, errors = extract_with_validation(llm_response, product_schema)
if errors:
# Log it, alert, fall back to CSS selectors — but don't use bad data
print(f"Extraction failed validation: {errors}")
The rule I follow now: LLM extraction should always have a validation layer. If you wouldn't trust user input without validation, don't trust LLM output without it either.
3. Rate limiting and anti-bot detection
News sites generally want to be scraped (hello, SEO traffic). Other sites actively fight it. If your scraper works perfectly in development but fails in production, you're probably getting blocked.
Signs you're being blocked without knowing it:
- You get 200 responses but the content is a CAPTCHA page
- Responses get progressively slower until they timeout
- You get valid-looking but outdated/cached content
Building a scraping pipeline that actually works
After many failed attempts, here's the architecture I've settled on for non-news scraping:
Layer 1: Prefer APIs and structured feeds. Before scraping anything, check if the site offers an API, RSS feed, sitemap, or structured data endpoints. You'd be surprised how often this exists.
Layer 2: Use CSS/XPath selectors as your primary extraction method. They're deterministic, fast, and they either work or they don't — no hallucination risk.
def extract_with_fallback(html, css_selectors, llm_prompt):
"""Try deterministic extraction first, fall back to LLM only if needed."""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
results = {}
for field, selector in css_selectors.items():
element = soup.select_one(selector)
if element:
results[field] = element.get_text(strip=True)
# Only invoke the LLM if selectors missed fields
missing = [f for f in css_selectors if f not in results]
if missing:
# LLM handles the messy edge cases
llm_result = call_llm(html, llm_prompt, fields=missing)
results.update(llm_result)
return results
Layer 3: LLM extraction as fallback, not primary. Use the LLM for the messy stuff that selectors can't handle — but always validate its output.
Layer 4: Monitor for drift. Set up checks that alert you when extraction results change shape or volume unexpectedly. A 30% drop in successful extractions usually means the site changed its layout.
Prevention: design for failure from the start
A few habits that have saved me from 3am debugging sessions:
- Store raw HTML alongside extracted data. When extraction breaks, you can re-run against the saved HTML without re-scraping. This is huge for debugging.
- Version your extraction configs. When you update selectors, keep the old ones. Sites sometimes revert changes.
- Set up dead-letter queues. Pages that fail extraction shouldn't just disappear. Route them somewhere you can inspect and reprocess.
- Test against cached snapshots, not live sites. Your CI should not depend on external websites being up and unchanged.
The uncomfortable truth is that web scraping beyond news digests requires ongoing maintenance. It's not a set-and-forget pipeline — it's a living system that needs monitoring, validation, and regular adjustment. The sooner you accept that and build accordingly, the less time you'll spend wondering why your "AI-powered" scraper is confidently returning garbage.
The LLM isn't the problem. The web is the problem. The LLM just makes it easier to not notice when things break.
