Competitor pricing sounds easy until you try to automate it. A rival drops delivery fees for lunch, adds a banner-only promo code, changes menu item names, and your dashboard still says nothing changed because your scraper matched the wrong DOM node or cached yesterday's page.
Food delivery data moves fast, but the harder problem is not speed. It is trust. If the data is late, duplicated, partially scraped, or normalized badly, teams stop using it. A useful web data strategy starts with a narrow decision and works backward from there.
Start with the decision, not the data source
Do not begin with “we should scrape competitors.” Begin with the action you want to take.
For example:
- If a competitor undercuts a basket by more than 15%, notify pricing.
- If three competitors run free delivery in the same zone, consider a short promo.
- If reviews mention late delivery more than usual, check staffing and routing.
That gives you a data contract. For competitor pricing, you probably need:
create table competitor_offers (
observed_at timestamp not null,
source text not null,
competitor text not null,
city text not null,
restaurant_name text not null,
item_name_raw text not null,
item_name_normalized text,
item_size text,
price_cents integer,
delivery_fee_cents integer,
promo_text text,
promo_type text,
url text not null,
scrape_run_id uuid not null
);
The observed_at and scrape_run_id fields matter more than they look. Without them, you cannot tell whether a price changed or whether your scraper simply ran twice against the same HTML.
Collect less, but collect it consistently
A small, reliable scrape beats a broad scrape that fails silently. Pick a few high-value markets, restaurants, and basket types first. For example, track the same 20 restaurants across three delivery zones every 30 minutes during business hours.
A minimal scraper might look like this:
import uuid
import requests
from bs4 import BeautifulSoup
from datetime import datetime, timezone
RUN_ID = str(uuid.uuid4())
URLS = [
{
"competitor": "example-delivery",
"city": "chicago",
"url": "https://example.com/chicago/restaurants/burger-place"
}
]
def fetch(url):
res = requests.get(
url,
headers={"User-Agent": "pricing-monitor/1.0"},
timeout=15
)
if res.status_code != 200:
raise RuntimeError(f"fetch failed: {res.status_code} for {url}")
return res.text
def parse_listing(html, meta):
soup = BeautifulSoup(html, "html.parser")
rows = []
items = soup.select("[data-testid='menu-item']")
if not items:
raise RuntimeError("no menu items found, selector may be stale")
for item in items:
name = item.select_one("[data-testid='item-name']")
price = item.select_one("[data-testid='item-price']")
promo = item.select_one("[data-testid='promo-banner']")
if not name or not price:
continue
rows.append({
"observed_at": datetime.now(timezone.utc).isoformat(),
"source": "web",
"competitor": meta["competitor"],
"city": meta["city"],
"item_name_raw": name.get_text(strip=True),
"price_raw": price.get_text(strip=True),
"promo_text": promo.get_text(" ", strip=True) if promo else None,
"url": meta["url"],
"scrape_run_id": RUN_ID
})
return rows
for meta in URLS:
html = fetch(meta["url"])
records = parse_listing(html, meta)
print(records)
This is not production code. It is the shape of the pattern: fetch, fail loudly, parse, attach run metadata, then load raw records before cleaning them.
The common failure mode is an empty selector after a frontend release. If your code returns zero rows and treats that as success, your dashboard will show “no offers” instead of “data collection failed.” Those are very different business signals.
When selector churn, geo-specific pages, login flows, or rate limits become the main work instead of the analysis, Wire fits in this extraction layer as a way to handle reliable web data collection while keeping your warehouse and alerting logic under your control.
Normalize before you compare
Food delivery data has annoying naming problems. “Cheeseburger,” “Classic Cheeseburger,” and “Cheeseburger - Single” may refer to the same item. “Small,” “sm,” and “S” may be the same size. Promo text is worse: “$5 off $25,” “Save five when you spend 25,” and “FIVEON25” can describe the same mechanic.
Keep raw fields and normalized fields. Do not overwrite the source text.
import re
def price_to_cents(value):
match = re.search(r"\$?([0-9]+(?:\.[0-9]{2})?)", value)
if not match:
return None
return int(float(match.group(1)) * 100)
def normalize_size(value):
if not value:
return None
v = value.strip().lower()
return {
"s": "small",
"sm": "small",
"small": "small",
"m": "medium",
"med": "medium",
"medium": "medium",
"l": "large",
"lg": "large",
"large": "large"
}.get(v, v)
def classify_promo(text):
if not text:
return None
t = text.lower()
if "free delivery" in t:
return "free_delivery"
if re.search(r"\$\d+\s*off", t):
return "fixed_discount"
if re.search(r"\d+%\s*off", t):
return "percent_discount"
return "other"
Expect edge cases. A scraped price of $0.00 might mean a free add-on, not a broken record. A missing delivery fee might mean pickup-only, hidden fee calculation, or blocked location context. Add a data_quality_status field if you need humans to review uncertain rows instead of dropping them.
Alert on business changes and data failures
You need two kinds of alerts: market alerts and pipeline alerts.
A market alert tells someone a competitor changed something meaningful:
select
competitor,
city,
item_name_normalized,
min(price_cents) as current_lowest_price
from competitor_offers
where observed_at >= now() - interval '30 minutes'
group by 1, 2, 3
having min(price_cents) < 0.85 * (
select avg(price_cents)
from competitor_offers history
where history.item_name_normalized = competitor_offers.item_name_normalized
and history.city = competitor_offers.city
and history.observed_at between now() - interval '7 days' and now() - interval '1 day'
);
A pipeline alert tells you not to trust the data yet:
select scrape_run_id, competitor, city, count(*) as rows_collected
from competitor_offers
where observed_at >= now() - interval '2 hours'
group by 1, 2, 3
having count(*) < 10;
That second query catches selector breaks, blocked pages, bad location settings, and partial loads. It will not tell you exactly what failed, but it prevents a quiet failure from becoming a pricing decision.
Measure whether the actions helped
A dashboard full of competitor prices is not the goal. Tie each response to an outcome.
If pricing matches competitor discounts, measure gross margin, order volume, and lost sales before and after the change. If operations responds to delivery complaints, track review rating, late delivery rate, and refund volume. If marketing launches a promo because competitors did, compare redemption rate against the discount cost.
Keep the loop short. For food delivery, a weekly report may be too slow for lunch and dinner promotions, but a real-time alert for every one-cent change will create noise. Start with thresholds that match an actual decision, then tune them after a few weeks of false positives and missed events.
A good next step is to pick one city, one competitor set, and one basket definition, then build the raw table, normalization pass, and failure alert before adding more sources.
