Stop Paying Your LLM Twice: The Dedup Pattern Every Ingestion Pipeline Needs

Every product team drowns in feedback the same way.

A review lands on the App Store. Someone vents on Reddit. A G2 comparison goes up. A support ticket gets resolved, a tweet gets posted, a Play Store rating drops from four stars to two. Each one of those is a signal about your product, and each one lives on a different platform, in a different format, behind a different wall.

Reading them all is impossible. Acting on them is harder.

So I built FountainData VOC: a multi-tenant Voice-of-Customer engine that connects to roughly 30 feedback sources, pulls everything into one pipeline, strips out the noise, runs AI extraction over the clean signal, clusters it into themes, ranks those themes by business impact, and pushes the results straight into Jira, Linear, GitHub, or Trello where work actually happens.

This is the story of what building it taught me. Not the marketing version. The part about scrapers, retries, deduplication, and why my cloud bill looked the way it did.

The Shape of the System

Before the lessons, here is the pipeline in one pass:

The backend is FastAPI with async SQLAlchemy over Postgres, pgvector for similarity search, Celery backed by Redis for background work, and custom Playwright scrapers for the platforms that fight back. Embeddings are 768 dimensions from Gemini. Everything is multi-tenant, with plans, quotas, and role-based access enforced end to end.

That is the skeleton. The interesting parts are the joints.

Lesson 1: Scraping Is an Optimization Problem, Not a Coding Problem

My first working scraper took more than 20 minutes to pull a single source. Same platform, same review count, every time. For a while I accepted it because it worked, and "it works" is a dangerous sentence in a pipeline whose whole job is to run on a schedule.

When I finally profiled it, the problem was not the scraping logic. It was everything around it: a fresh browser context per page, images and fonts downloaded for pages I only read text from, and no session reuse between pagination calls.

The fix was boring, which is usually how you know it is correct:

• Reuse the browser instance across pages instead of spawning per navigation.
• Block images, fonts, stylesheets, and media requests at the route level. Reviews are text.
• Run Playwright in stealth mode with sensible anti-automation arguments so the big platforms stop serving challenge pages.
• Normalize URL variants before fetching so Trustpilot slugs and app IDs resolve to one canonical target instead of three near-duplicate crawls.

Same reviews, under 10 minutes. One commit, two files changed. The lesson stuck: when a data job is slow, the fix is almost never "more parallelism." It is "stop doing work you never needed."

Lesson 2: Deduplication Is Your LLM Budget's Bodyguard

Here is the mistake that quietly defines most ingestion systems: they treat every incoming record as new.

Mine did too, at first. And in this domain, duplicates are not an edge case. They are the default state of the world:

• Stores surface the same edited review again on the next crawl.
• A user posts the same complaint on Reddit and then emails support about it.
• A scheduled sync overlaps with a manual trigger and pulls the same window twice.
• A batch fails halfway through, gets retried, and arrives as a "fresh" set where half the rows already exist.

If every one of those rows triggers an LLM extraction call, you are paying, repeatedly, to learn things you already know. Worse, your clusters fill with double-counted sentiment and your impact rankings start lying to you.

So deduplication became the most economically important code in the system, and it works in two layers.

Layer one: hash before you store. Every incoming review gets stripped and hashed with SHA-256. The hash is computed on normalized content, not raw bytes, so trailing whitespace and formatting differences do not sneak past. If the hash exists, the row is skipped before it ever touches the database write path.

Layer two: reuse processed intelligence. This is the part that actually saves money. When a raw review passes dedup and reaches the AI stage, the worker hashes its content again and checks whether an identical review has already been processed. If it has, the existing sentiment, topics, feature-request classification, and embedding are copied forward instead of calling the model again. Identical text produces identical analysis. There is no reason to pay for it twice.
Here is what that reuse check looks like in the worker, trimmed to the essentials:

import hashlib

h = hashlib.sha256((raw_review.content or "").strip().encode("utf-8")).hexdigest()

reused = await db.execute(
    select(ProcessedReview)
    .join(ProcessedReview.raw_review)
    .where(RawReview.content_hash == h)
)
reused_pr = reused.scalars().first()

if reused_pr:
    # Identical text produces identical analysis. Copy it forward
    # and skip the model call entirely.
    pr = ProcessedReview(
        raw_review_id=raw_review.id,
        embedding=reused_pr.embedding,
        sentiment=reused_pr.sentiment,
        topics=reused_pr.topics,
        issues=reused_pr.issues,
        feature_requests=reused_pr.feature_requests,
        summary=reused_pr.summary,
        confidence_score=reused_pr.confidence_score,
        evidence_quotes=reused_pr.evidence_quotes,
        # ...remaining metadata fields copied the same way
    )
    db.add(pr)

The same discipline applies to failures. Retries are necessary; scrapers fail on rate limits, timeouts, and transient 5xx constantly. Every network-facing scraper runs bounded retries with backoff delays and explicit handling of retryable status codes. But a retry without an idempotency guarantee just re-ingests half-completed work as new records, which multiplies both storage and AI cost. With the hash check in front, a retry becomes cheap: the pipeline redoes only the rows that genuinely never landed.

The pattern generalizes beyond this project. In any system where an external event triggers expensive processing, ask yourself: what happens when the same event shows up twice? If the answer involves paying twice, you do not have a throughput problem. You have a deduplication problem wearing a throughput costume.
One of the scrapers, trimmed to the skeleton:

max_retries = 3
result = []

for attempt in range(max_retries):
    try:
        result, _ = reviews(app_id, lang=lang, country=country,
                            sort=Sort.NEWEST, count=limit)
        break
    except Exception as e:
        print(f"Play Store sync failed "
              f"(attempt {attempt+1}/{max_retries}): {e}")
        if attempt == max_retries - 1:
            raise e
        time.sleep(2 * (attempt + 1))

The hash check upstream is what makes this safe to run. If attempt one stored half a source before dying, attempts two and three simply skip everything that already landed and fill in only the gap.

Lesson 3: The Database Is Better at This Than Your Application Code

Once volume grew, the naive patterns started showing up in profiling.

Bulk inserts became chunked batches of 500 records so large syncs would not balloon worker memory. External ID existence checks, which started life as one query per review, became chunked lookups of 1,000 IDs at a time. A hard safety cap plus recursive backfill keeps any single run bounded no matter how far behind a source has fallen.

The biggest win came from moving vector search out of application memory entirely. The original RAG path loaded embeddings into the process to compute similarity, which works fine in a demo and falls over in production. Refactoring the search to execute directly inside pgvector cut memory usage during RAG operations by roughly 90 percent, because Postgres was already sitting right there with an index.

Targeted indexes on the raw and processed review tables removed the remaining full-table scans during high-volume ingestion windows.

None of these changes are clever. That is the point. Chunk your writes, batch your lookups, index what you filter on, and let the database do database work. The fancy parts of the system, the agents and the clustering and the ROI scoring, only perform because the boring foundation underneath them does.

The Chapter About Money

Now the part nobody puts in the launch post.

The production deployment ran on AWS: RDS for Postgres, Redis for queues, Secrets Manager for configuration, containers deployed through Cloud Build onto Cloud Run, with a Celery worker service alongside the API. A properly enterprise-shaped setup.

And it ate credits relentlessly.

Here is what I learned watching that bill. Always-on infrastructure charges you for existing, not for working. An idle Celery worker still costs. A small RDS instance still costs. NAT egress from a VPC still costs. The architecture was correct for a company at scale and expensive for a product finding its users, and those two facts are not in conflict, but they do compete for the same budget.

The AI side compounded it. Extraction ran one model call per review, sequentially. Correct and simple, but the cost curve was linear with ingestion volume and there was no ceiling on it until I added plan-based sampling rates and a hard per-sync cap on AI jobs. Those two controls turned an unpredictable line item into a bounded one, which matters more than any amount of prompt tuning.

The honest takeaway: I bought enterprise infrastructure before I had enterprise problems. Some of that was deliberate learning, and I do not regret the education. But if I were coaching someone building version one today, the bill conversation happens on day one, not after the first invoice.

What I Would Do Differently Starting Fresh

Rebuilding this pipeline today, with the same requirements and scarcer resources, four things change.

1. I would not reach for Celery by default.

Celery is powerful, battle-tested, and completely capable. It is also heavy: a broker, result backends, beat scheduling, worker fleet management, and serialization semantics you must respect across versions. Several of my silent failure modes lived in that layer. For a workload shaped like this one, fetch, process, store, notify, I would evaluate a lighter task layer first: arq or TaskIQ for async-native Python, or even Postgres with SKIP LOCKED as the queue itself. Fewer moving pieces means fewer places for a job to disappear quietly. Celery earns its weight at real scale; version one rarely starts there.

2. Concurrency at the LLM boundary.

Extraction called the model once per review, in sequence. The moment I added sampling rates and caps I was managing a symptom. The actual fix is bounded concurrency against the provider, with batching where the API allows it, turning a linear cost-and-time curve into something much flatter. Ten reviews should not take ten times longer than one.

3. One process, not a fleet, until metrics say otherwise.

I provisioned the shape of a company before I had the workload of one. A single well-configured service on modest hardware would have carried version one fine, and every component I split out early was a component I had to monitor, secure, and pay for before it earned its keep. Split when the data forces you to, not when the architecture diagram looks impressive.

4. Instrument cost as a first-class metric from day one.

Requests per second and p95 latency are table stakes. But in an AI pipeline, the metric that decides whether the product survives is cost per record processed. I added usage tracking and AI cost monitoring eventually; I should have had them before the first sync ever ran. When cost is visible per workspace, per source, and per run, bad patterns announce themselves immediately instead of showing up on an invoice.

What Came Out of It

Despite all of that, the system works. It reads from around 30 source types, from app stores and review platforms to Reddit, Twitter, support desks, CRMs, spreadsheets, and raw S3 drops. It deduplicates, extracts, clusters, ranks by impact, answers questions through RAG agents with evidence quotes attached, detects volume spikes with z-score anomaly detection, and pushes prioritized work into the tools teams already use.

More importantly, building it changed how I think about data pipelines. Throughput is not about running fast. It is about refusing to do the same work twice, letting the database do what databases are good at, bounding everything that can spiral, and treating your cloud bill as a feature requirement rather than an afterthought.

A Smaller Way In

The full platform carries all the enterprise machinery: multi-tenancy, plan gating, billing, integrations, agents. That machinery is necessary at scale, but it is also a lot if all you want is the intelligence.

So we distilled the core of it into PHOS, a lighter product that keeps the part that matters most: scraping real customer reviews from app stores, review sites, and the Chrome Web Store, and turning them into structured, ranked product insight.

If you want to see this pipeline applied to your own product without standing up any of the infrastructure above, that is what it is for:

PHOS

If you have built something similar, I would love to hear how you handled deduplication and retry safety. Drop a comment and let me compare war stories.