CitePulse: Auditing the Answer Layer

CitePulse is a local-first, open-source AEO audit instrument (MIT): it runs on your own machine against a free local model (llama3.1:8b via Ollama) and assesses a site's AI answer readiness against five derived principles — a machine must be able to read the site, cite it truthfully, share it against competitors in real prompts, act on it when an autonomous agent attempts a task, and be told honestly when a score cannot be measured ("not determined" rather than a guessed number). It drives a real browser as an autonomous agent instead of just parsing an API. This case study runs it against three real commercial targets on the same day — an AI search-monitoring SaaS, a European staffing and recruitment firm, and a cooperative bank — and walks the reasoning behind every value in the resulting audit reports (run 2026-09-24, CitePulse v1.7.0).

Anonymization note. The three targets are real, named companies that were audited without prior arrangement from their public sites. Their identities are withheld here; each is described only by category. Every figure is exactly as the tool reported it.

Disclosure. The author is the maintainer of CitePulse, the tool used for these audits.

1. Three tests, three verdicts

One instrument, one model, one prompt-scheme version, one day. Only the targets differ — and so do the verdicts, each for a different reason.

Three failure profiles, visible before reading a single KPI:

• Target A fails where agents act. Fully measured (9/9) — best-in-class readability, 100% citation correctness, 91.3% share — and then collapses the moment an autonomous agent tries to do something.
• Target B is invisible. Crawl infrastructure is best-in-class, yet the model never quotes it at all — a retrieval/content failure, not a technical one.
• Target C is refused. Nearly half its metrics (4 of 9) are not determined: its crawler-facing surface blocks measurement outright and every agent task hits an authentication wall. The tool declines to score what it cannot verify.

The verdict band is never the average of nine numbers; it is the report's statement of the weakest load-bearing principle. That is the framing all three scorecards below sit on.

2. The scorecards: nine KPIs across the three targets

Read the rows twice: once for the values, once for what the empty cells are saying. The not-determined cells are not voids — they are the instrument refusing to guess. Target B's citation and SoV at 0% with crawl at 3.0 is the purest demonstration in this dataset that readability and retrievability are different things. Target C's four not-determined cells are the instrument correctly distinguishing "we can't verify this" from "this failed."

3. The underpinning logic: from principle to the audit values

A score is only as good as the chain between the readiness claim and the number. This section walks that chain principle by principle, across all three targets.

3.1 Principle 1 — the read layer

Claim operationalized: a machine must traverse every layer of the estate — robots policy, structure index, machine-readable summary — before any other readiness question is addressable. If principle 1 fails, every later metric is uninterpretable because you can't tell "missing" from "unread."

• Target A: 3.0 / 3.0 / 3.0. All three gates fully pass — robots allows known AI crawlers, valid schema is present, llms.txt has section headers and live links. Analytically this is the most valuable row: because the read layer is perfect, A's later failure cannot be blamed on blocking. Attribution, secured.
• Target B: crawl 3.0, llms.txt not determined. The crawler can traverse the site and a sitemap exists, but the documented llms.txt paths return 404 and 403 — the probe cannot conclude whether an index is present or absent, so it does not invent an answer. A partial read of the intended index layer.
• Target C: none of it. robots.txt and the schema probe both come back blocked. The audit cannot conclude whether AI crawlers are disallowed or whether structured data exists — it reports exactly that.

3.2 Principle 2 — correctness vs. rate

Claim operationalized: readability ≠ citability. Two separate things must be true: a fetched page must contain claims a model can quote accurately (correctness), and those pages must actually be surfaced in answers (rate).

• Target A: 100% correctness, 55.6% rate. Ten of ten judgeable citations were supported by the cited pages' own content — the pages, once found, are truthful sources. But fewer than two-thirds of answers cite A at all, and the segment split makes the mechanism explicit: ~100% citation on capability-level queries, ~0% on category-discovery queries. The pages are answer-shaped but not question-matched. Because correctness is 100%, this is a placement problem, not a truthfulness problem — and the remediation differs accordingly.
• Target B: 0% rate, correctness not determined. There are no citations to judge — the correctness probe literally has an empty input. The model prefers a competing professional network and a job board on the queries tested. This is the signature of content that exists but is not retrievable as quotable facts: "indexed but never quoted."
• Target C: 100% correctness, 33.3% rate. Five of five judgeable citations check out, but only 6 of 18 answers cite the bank. Notably, it is not cited on the most basic question in the set — "What is ?" — while specialty topics are well covered. Uneven topical coverage, not weak content.

3.3 Principle 3 — share

Claim operationalized: presence isn't share; share must be competitive. Being citable in principle isn't being chosen against whoever surfaces in the same prompts.

• Target A: 91.3% raw, 89.1% weighted. Dominance in the retrieval layer, with a two-point weighted gap pointing at where a competitor edges into comparison queries — the place where buying decisions concentrate.
• Target B: 0% share. Not cited, not chosen, across all five tested query stages. This KPI is the numeric cost of the principle-2 failure.
• Target C: 86.5% raw, 91.2% weighted. High and broad — a peer bank still beats it on brand-identity queries ("What is …?"), a narrow authority gap rather than a systemic one.

One anomaly worth reading against principle 5: Target B shows 0% unweighted SoV yet a 91.7% weighted SoV banded best-in-class. Both numbers come from the same N=18 sample; the apparent contradiction is a reminder that a business-impact-weighted score can offset a tiny, dominated share — precisely the situation where a confidence flag earns its keep.

3.4 Principle 4 — action (the layer almost no tool tests)

Claim operationalized: being mentioned isn't being useful. When an autonomous agent tries a real task, can it finish? This requires actually driving a browser, which is why nearly every other tool in the category stops at principle 3.

• Target A: 33.3% completion, 74.3% interaction readiness. Of the three site-attributable tasks, one succeeded and two were site failures (an API task was blocked as a policy restriction and is counted separately, not as a failure). The interaction reading is a transcript — 9 of 35 low-level actions failed:

Page.click: Element is not visible
Call log:
  - waiting for locator("[data-aeo-idx=\"7\"]")
    - locator resolved to 2 elements. Proceeding with the first one:
      <a tabindex="0" data-aeo-idx="7" href="/features/ads-tracking"
         class="navbar2_dropdown-link w-dropdown-link">Ads Tracking</a>
  - attempting click action
    - scrolling into view if needed

Three engineering-grade signals in one trace: a non-unique selector resolving to two elements, an occluded target that cannot be brought into view, and a click that never completes so the agent stalls until exhaustion. The value → mechanism → fix chain is complete: unstable selectors and hidden dropdown targets are a diffable code change, not a strategy problem.

• Target B: 85.7% interaction, task completion not determined. The only site-attributable task run succeeded, but the sample fell below the minimum, so completion is honestly not scored. Interaction is close to best-in-class — a single failed action, where the agent tried to type a location into an element that is a link:

Page.fill: Error: Element is not an <input>, <textarea>, <select> or [contenteditable]
  - waiting for locator("[data-aeo-idx=\"3\"]")
    - locator resolved to <a href="/" data-aeo-idx="3" class="…logo-link…">
  - fill("location")

• Target C: entirely refused. All five agent tasks — checking a balance, paying a bill, applying for a card — were correctly classified as gated boundary: they terminate at an authentication wall and an error page. The bank's real tasks are behind login, and an instrument without credentials cannot test them. The taxonomy does not call an HTML login page a "site failure," so task completion and interaction readiness are reported not determined.

3.5 Principle 5 — the governance rule every value stands on

Principle 5 is not a score; it is the enforcement layer that makes 3.1–3.4 trustworthy:

• Not determined is a designed output across all three runs. Target B: llms.txt, correctness, task completion. Target C: crawl, schema, task completion, interaction. Where a clean measurement was impossible — blocked fetch, no citations to judge, an auth wall, a sub-floor sample — the report said so instead of interpolating.
• Below-floor samples are declared, not smoothed. The limitations sections name the exact KPIs measured under the confidence floor: A → #22, #24, #45, #62, #48; B → #22, #24, #62, #58; C → #22, #24, #45, #62.
• The model's proxy status is stated up front. Citation and SoV metrics come from a local llama3.1:8b synthesizing over live web-search results — "a proxy for AI-answer-engine behavior, not a live query to ChatGPT, Perplexity, Gemini, or Copilot."
• Deltas refuse to overclaim. Across the runs, every change is either "within noise — not a significant change" (e.g. C's citation rate −16.7% band Good → Needs improvement) or "no confidence interval available — not checked for significance" (e.g. A's SoV +14.2%).
• The tool even audits its own history. Target C's 18-run trend carries an explicit warning that the series spans multiple models (qwen2.5:7b, llama3.1:8b, granite3.2-vision:2b, gemma2:9b, llava:7b), so an apparent change may be a model difference, not a change on the site.

3.6 The composed verdicts

• Target A is "Needs attention" despite a best-in-class read layer and 100% citation correctness, because the report treats actionability as load-bearing. Readable, citable, and cited — and still one-third usable when an agent acts.
• Target B is "High risk" because it fails at the retrieval boundary itself: a technically healthy estate that no answer engine quotes.
• Target C is "Needs attention" because nearly half of readiness is unverifiable behind its defenses, and — where it could be verified — the readings are strong.

Three verdicts, three different load-bearing principles. That is the entire argument against blended scoring.

4. Epistemics in practice

One run's full accounting, per target:

5. Measurement as a standing capability

The reports are explicitly built for cadence, not one-off snapshots:

• Target A compares against the 2026-09-22 run and shows a 5-run trend (2026-09-15 → 09-24): citation rate 50 → 27.8 → 50 → 50 → 55.6; SoV 52.4 → 11.7 → 94.7 → 77.1 → 91.3; interaction 66.7 → 74.3. The SoV series — spanning 11.7% to 94.7% on low-confidence N=18 samples — is exactly why principle 5 exists: without per-delta significance disclaimers, that noise would be read as a trend.
• Target B compares against 2026-09-19: only 2 runs so far; crawl and citation flat, weighted SoV 100 → 91.7.
• Target C is the strongest cadence example: 18 completed runs since 2026-09-04, for which the report itself flags the multi-model caveat. Citation rate has swung between 0% and 61% across that history — a genuinely noisy market signal that any single-point measurement would misrepresent.

6. Reproducibility, auditability, and zero data egress

Two properties keep the instrument credible for governance use:

• The runs are re-verifiable. Each manifest embeds audit_id, start/finish timestamps, model/provider, coverage counters, requested_kpi_ids, and generation-scheme versions (task_generation: 1.0.0, citation_prompts: 1.0.0). A skeptic can re-run instead of trusting a slide-deck screenshot.
• Nothing leaves the machine. Default execution is local (Ollama + a local model), and every report footer states: "Generated locally by CitePulse — no data leaves your machine." The scoring logic is MIT-licensed and public — a metric you cannot explain to your own risk committee is disqualifying for governance purposes.

The design notes are candid about limits: a WAF returning a "just a moment" challenge page with HTTP 200 can slip through the crawl probe — "a known limitation, not a solved one." Naming the failure mode and tracking it is part of principle 5.

7. What each audited firm should do next

Target A (AI search-monitoring SaaS):

1. Fix Task Completion (P1): 1 of 3 site-attributable tasks succeeded. Re-run and confirm it leaves "Needs improvement."
2. Fix Interaction Readiness (P1): 9 of 35 actions fail on unstable selectors and occluded targets — stable selectors (test-ids, roles), unique and present before any action.
3. Close the citation-rate gap (P2): ~100% capability-level vs ~0% category-discovery — content-vs-query alignment, not a crawl problem.
4. Guard weighted SoV where a competitor edges into comparison queries.

Target B (staffing & recruitment):

1. Fix citation rate and SoV (P0): 0% across 18 answers. Publish clear, structured, quotable content (FAQs, comparison content, a valid llms.txt) and confirm the site is indexable by the engines answer engines draw from.
2. Complete Organization schema (P2): structured data present but missing required fields.
3. Add a reachable llms.txt (P2): documented paths return 404/403.

Target C (cooperative bank):

1. Unblock basic measurement: public robots.txt and schema probes were blocked — if deliberate, confirm this trade-off is acceptable; the tool can report nothing about crawler access.
2. Improve quote reliability (P1): 6 of 18, including a miss on "What is the bank?" — structured FAQs and clearly named entities.
3. Add section headers to llms.txt (P2): present but not a navigable index.
4. Expect the gated boundary: banking tasks sit behind auth — run agent task tests with credentialed sessions to measure what matters where it happens.

8. Lessons for practitioners

• Separate the failure modes; never blend them. This dataset is a complete proof: one target is cited-but-unactionable, one is technically clean but invisible, one is defensively unverifiable but strong where measurable. A blended score exists to hide exactly these distinctions.
• The value is only as good as its derivation chain. Every KPI here can be walked from claim → principle → measurement → fix. That is what makes the numbers defensible in a room where someone says "prove it."
• Test principle 4 or you are not measuring the answer layer. Querying an API tells you what a model says; attempting the task tells you what the site does. Across three audits, the action layer showed a real failure, a near-pass, and a refusal — only a browser could tell you which.
• "Not determined" is a measurement, not a caveat. Target C's four refused KPIs are more informative than any guessed number would be.
• Demand the confidence. A rate without a sample size and a confidence floor is a vibe.
• Scorecards should come with diffs — "within noise," "not checked for significance," "not comparable" — or the measurement cannot compound.
• The epistemology is the product. A tool that never says "I don't know" is telling you more about its incentives than about your visibility.

9. Sources

• CitePulse reports, run 2026-09-24, CitePulse v1.7.0, model llama3.1:8b, targets anonymized: Audit 50a7b1fb… (AI search-monitoring SaaS, 9/9 KPIs), Audit eaedd414… (staffing & recruitment, 6/9 KPIs), Audit ff7f499c… (cooperative bank, 5/9 KPIs).
• Lawrence Sanjay, "The Answer Layer" — design article for CitePulse (dev.to, 2026-09-18).
• CitePulse (open source, MIT): github.com/alsanjayllm/CitePulse-public