A pricing team asks for competitor intelligence, and the first version usually looks simple: request a few quotes, put the premiums in a spreadsheet, compare numbers. Then the quote form changes, one site starts returning captchas, another hides fees behind a dropdown, and your dataset quietly fills with nulls.
Insurance pricing is not fully hidden. Every quote flow leaks signals about how inputs affect premiums: age, postcode, deductible, vehicle type, coverage limits, policy duration, and so on. The hard part is not noticing one quote. The hard part is collecting comparable observations often enough that the data can support a decision.
Treat quotes as observations, not facts
A quote is an observation taken at a specific time with a specific input profile. Store it that way. Do not just keep insurer name and premium.
A useful quote row usually needs fields like these:
CREATE TABLE quote_observations (
id INTEGER PRIMARY KEY,
captured_at TEXT NOT NULL,
source TEXT NOT NULL,
product TEXT NOT NULL,
insurer TEXT NOT NULL,
premium_cents INTEGER NOT NULL,
currency TEXT NOT NULL,
deductible_cents INTEGER,
coverage_limit_cents INTEGER,
policy_duration_days INTEGER,
geography TEXT,
profile_hash TEXT NOT NULL,
raw_payload TEXT NOT NULL
);
The profile_hash matters because you need to compare like with like. If yesterday's driver age was 35 and today's is 36, the delta may be age-related rather than a competitor repricing. The raw_payload matters because parsers break. When they do, you need to reprocess old captures without rerunning the quote journey.
In this kind of workflow, Wire fits as the extraction layer for collecting live insurance quote fields across regions, demographics, and time windows.
Vary one input at a time
The common mistake is generating a large random sample and hoping analysis will explain it later. That gives you coverage, but it makes causality messy. For pricing intelligence, a controlled sampling matrix is easier to debug.
Start with a baseline profile, then change one input at a time:
import hashlib
import json
from copy import deepcopy
baseline = {
'product': 'auto',
'age': 35,
'postcode': '94110',
'deductible_cents': 50000,
'coverage_limit_cents': 10000000,
'vehicle': '2019 Toyota Camry',
'policy_duration_days': 365,
}
variants = []
for age in [25, 35, 45, 60]:
p = deepcopy(baseline)
p['age'] = age
variants.append(p)
for deductible in [25000, 50000, 100000]:
p = deepcopy(baseline)
p['deductible_cents'] = deductible
variants.append(p)
for postcode in ['94110', '90011', '10001']:
p = deepcopy(baseline)
p['postcode'] = postcode
variants.append(p)
def profile_hash(profile):
stable = json.dumps(profile, sort_keys=True)
return hashlib.sha256(stable.encode()).hexdigest()
for profile in variants:
print(profile_hash(profile), profile)
This is not statistically complete, but it is understandable. If a premium jumps when only the postcode changes, you know where to investigate. You can add a larger sample later once the pipeline proves it can collect clean data.
Fail loudly when extraction breaks
Silent failure is worse than no data. If your parser cannot find the premium, raise an error and mark the capture as failed. Do not insert a zero, an empty string, or the previous value.
from bs4 import BeautifulSoup
import re
class QuoteParseError(Exception):
pass
def parse_premium(html):
soup = BeautifulSoup(html, 'html.parser')
node = soup.select_one('[data-testid=monthly-premium]')
if node is None:
raise QuoteParseError('premium selector not found')
text = node.get_text(' ', strip=True)
match = re.search(r'\$([0-9,]+)(?:\.([0-9]{2}))?', text)
if not match:
raise QuoteParseError(f'premium text did not match currency pattern: {text}')
dollars = int(match.group(1).replace(',', ''))
cents = int(match.group(2) or '00')
return dollars * 100 + cents
This will fail when the site changes markup, when the quote is hidden behind a login, when a bot wall returns a page that still has HTTP 200, or when the premium appears as an annual total instead of monthly. Those are good failures. They tell you the observation is not comparable.
Also record the response status, final URL, screenshot path if you use a browser, and parser version. When someone asks why Tuesday's premiums disappeared, you want an answer better than maybe the site was weird.
Normalize before comparing
Competitors rarely present prices the same way. One shows monthly premium. Another shows annual premium. One includes taxes. Another adds fees at checkout. If you compare raw displayed values, you will report false deltas.
Normalize into a common unit before analysis:
def normalize_to_annual_cents(premium_cents, billing_period, fees_cents=0):
if billing_period == 'monthly':
return premium_cents * 12 + fees_cents
if billing_period == 'annual':
return premium_cents + fees_cents
raise ValueError(f'unknown billing period: {billing_period}')
previous = 118000
current = 124500
change_pct = (current - previous) / previous * 100
if abs(change_pct) >= 3:
print(f'premium moved {change_pct:.1f}%')
That 3 percent threshold is arbitrary, but having a threshold is important. Without one, every tiny movement becomes an alert. With one, you can separate noise from changes worth reviewing.
Feed a decision, not just a dashboard
Decision Intelligence sounds abstract, but in practice it is a feedback loop:
- Define quote profiles you care about.
- Collect competitor quotes on a schedule.
- Normalize the results.
- Detect meaningful changes.
- Send exceptions to pricing, actuarial, or underwriting teams.
- Record what action they took.
The last step is easy to skip. Do not skip it. If a competitor drops premiums in California after wildfire season, your team might choose to hold price because your loss model disagrees. That decision is data. Capture it so future analysis can distinguish between ignored alerts and deliberate pricing choices.
Wire can also sit in the monitoring part of this loop, where repeated quote captures need run-level tracking instead of one mixed pile of scraped rows.
The edge cases are real. Some quote flows personalize based on cookies. Some aggregators reorder insurers based on commercial agreements. Some insurers return different prices after multiple requests from the same session. Treat those as experimental conditions, not annoyances to hide.
A practical next step: pick one insurance product, define 20 stable customer profiles, collect quotes twice a day for a week, and reject any row that does not match your schema. You will learn more from the failures in that small pipeline than from a large dataset you cannot trust.
