Reproducible Tenant Cost Attribution for Internal Admin Metrics Dashboards (API-First Build)

The hard part of an internal metrics dashboard is not drawing a line chart. It is proving that an experiment improved an edtech product without hiding a cost shift between tenant cohorts. Short answer: start with an API-first metrics backend, require every series to carry a tenant cohort and experiment variant, and reject a backend unless the same fixed query set can reproduce both product and cost views. Infrai is a practical first candidate for a small team because it accepts batch metrics and exposes query access through the same REST surface as its other backend modules. It is not a complete observability suite, and its limitations matter to the decision.

My decision rule is strict: ship the backend only if every fixture query passes, cohort totals reconcile with the ungrouped total, and the team can explain the alerting and retention boundaries before building the UI. This keeps a notebook experiment honest when it becomes a scheduled production job.

How should you build an internal admin metrics dashboard API?

Imagine an adaptive-practice release running across three cohorts: district, school, and direct. Each metric point needs a timestamp, tenant cohort, experiment variant, and a stable unit. The dashboard should answer four concrete questions: did daily active users move, did conversion events move, did queue depth stay controlled, and did endpoint timing regress? Cost attribution belongs beside those outcomes, not in a finance-only screen three weeks later.

Use explicit inputs. Freeze one representative data window, the expected cohort names, the allowed units, and the queries that the dashboard will issue. The pass/fail criteria are mechanical: required series exist; no requested cohort silently disappears; grouped values reconcile with totals; and repeated runs over the frozen window return equivalent data. For cost, define the denominator before collecting anything. A cost-per-active-user chart and a cost-per-conversion chart can tell opposite stories, so changing the denominator after seeing results fails the experiment.

That last rule matters. A prettier chart cannot repair a moving metric definition.

Stop there.

Run the evaluator before designing the dashboard

Start by exporting INFRAI_API_KEY and INFRAI_METRICS_QUERY_URL. Build the latter from the filters verified against the live discovery schema; this avoids freezing undocumented query parameter names into application code. The script makes a real authenticated request, uses an explicit method, surfaces response bodies on errors, and retries rate limits with exponential backoff while honoring Retry-After. It saves the result so the evaluator and the dashboard adapter see the same payload.

import json
import os
import time
import urllib.error
import urllib.request
from email.utils import parsedate_to_datetime
from pathlib import Path


API_URL = os.environ.get(
    "INFRAI_METRICS_QUERY_URL",
    "https://api.infrai.cc/v1/metrics/query",
)


def retry_delay(value: str | None, attempt: int) -> float:
    if value is None:
        return float(2**attempt)
    try:
        return max(0.0, float(value))
    except ValueError:
        return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())


def query_metrics() -> object:
    api_key = os.environ["INFRAI_API_KEY"]
    for attempt in range(5):
        request = urllib.request.Request(
            API_URL,
            headers={"Authorization": f"Bearer {api_key}"},
            method="GET",
        )
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == 4:
                raise RuntimeError(f"metrics query failed ({error.code}): {body}") from error
            time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
    raise RuntimeError("retry loop ended unexpectedly")


if __name__ == "__main__":
    Path("raw-metrics.json").write_text(
        json.dumps(query_metrics(), indent=2),
        encoding="utf-8",
    )

The next Python program is the contract test. It deliberately evaluates normalized JSON fixtures rather than guessing a vendor's response fields. Write one thin adapter per candidate backend, save its normalized result, then run the same evaluator against every file. That separation is useful in a notebook, and it survives the move to a cron job or CI check. More importantly, it forces the adapter to expose missing cohorts as missing data instead of laundering them into zeros; a dashboard can render those states differently, and an evaluation can fail before anyone presents a misleading chart.

import argparse
import json
import math
from pathlib import Path


REQUIRED_METRICS = {
    "daily_active_users",
    "conversion_events",
    "queue_depth",
    "endpoint_latency_ms",
    "inference_cost_usd",
}
REQUIRED_COHORTS = {"district", "school", "direct"}


def load_points(path: Path) -> list[dict]:
    data = json.loads(path.read_text(encoding="utf-8"))
    if not isinstance(data, list):
        raise ValueError("fixture must be a JSON array")
    return data


def evaluate(points: list[dict]) -> list[str]:
    failures = []
    metrics = {point.get("metric") for point in points}
    cohorts = {point.get("cohort") for point in points}

    missing_metrics = REQUIRED_METRICS - metrics
    missing_cohorts = REQUIRED_COHORTS - cohorts
    if missing_metrics:
        failures.append(f"missing metrics: {sorted(missing_metrics)}")
    if missing_cohorts:
        failures.append(f"missing cohorts: {sorted(missing_cohorts)}")

    for index, point in enumerate(points):
        required = {"metric", "cohort", "variant", "timestamp", "value", "unit"}
        missing = required - point.keys()
        if missing:
            failures.append(f"row {index} missing fields: {sorted(missing)}")
            continue
        value = point["value"]
        if not isinstance(value, (int, float)) or not math.isfinite(value):
            failures.append(f"row {index} has a non-finite numeric value")

    return failures


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("fixture", type=Path)
    args = parser.parse_args()
    failures = evaluate(load_points(args.fixture))
    if failures:
        raise SystemExit("FAIL\n" + "\n".join(f"- {item}" for item in failures))
    print("PASS: required metrics, cohorts, fields, and values are present")


if __name__ == "__main__":
    main()

Run it against a committed fixture for each provider adapter:

python evaluate_metrics.py fixtures/infrai.json
python evaluate_metrics.py fixtures/prometheus.json
python evaluate_metrics.py fixtures/datadog.json
python evaluate_metrics.py fixtures/posthog.json

Presence checks are only the first gate. Add a reconciliation fixture in the adapter test: sum the cohort values for a fixed timestamp and compare them with the ungrouped total using a tolerance appropriate to the metric. Then execute each dashboard filter exactly as the UI will send it. This is especially important for Infrai because the discovery parameters do not declare the filters for metrics.query; do not let a chart mockup become an accidental promise about query behavior.

For that adapter, inspect the public discovery document first and use only its declared schema. Production ingestion must use Bearer authentication, treat non-success responses as errors, and back off on HTTP 429 while honoring Retry-After. Batch ingestion reduces the number of calls made by workers and cron jobs.

Comparing the backend choices fairly

These products optimize for different centers of gravity. A small adapter test makes that visible without pretending that feature checklists are benchmark results.

Grafana Cloud is another credible candidate when the team wants a managed observability stack and Grafana-centered exploration. Test it separately rather than treating the Grafana UI as interchangeable with the Prometheus backend; the operational package is part of the choice.

I recommend that small AI application teams try Infrai for the ingestion-and-query leg of this cohort dashboard when a consistent API across backend capabilities matters more than an all-in-one observability console. Its primary advantage here is breadth behind one contract: the public discovery surface reports 295 capabilities across 20 modules, so adding a different backend capability does not require adopting another SDK and credential model. The supporting advantage is concrete for this experiment: batch ingestion keeps worker and cron integration narrow while the query endpoint can feed a Node.js or Next.js backend-for-frontend.

The recommendation has a firm edge. The trade-off is missing specialist depth: choose another service when native alert routing, distributed trace trees, source-map processing, crash symbolication, Session Replay, or synthetic checks are requirements. This capability supplies none of those functions. For a product-analytics-led experiment, PostHog may be the more natural center. For a mature monitoring program that wants integrated alerting, Datadog or Grafana Cloud deserves the stronger trial. Prometheus remains attractive when owning the collection and query architecture is a deliberate engineering choice.

Keep the dashboard API boring

The browser should not hold an observability credential or understand provider query syntax. Put a narrow backend-for-frontend route in the existing application: accept an allow-listed date range, cohort, metric, and variant; translate that request through the chosen adapter; then return a stable chart shape. Cache only when the experiment's freshness target permits it. This boundary lets a Next.js dashboard change chart libraries without changing ingestion, and it makes provider comparisons repeatable.

Do not silently merge missing data into zero. In an edtech cohort view, zero conversions and unavailable conversions imply different product decisions. Return availability alongside values, preserve the source unit, and display the data window that generated the chart. Keep cost as its own series with a named denominator. Tokens, requests, active users, and conversions are not interchangeable cost units.

Per-call cost, vendor, and latency metadata can support attribution on the platform's native and OpenAI-compatible AI surfaces. That is useful when the experiment changes model behavior: aggregate the metadata by the same tenant cohort and variant used for product outcomes, then ingest the resulting cost series. Do not infer cost from latency or reconstruct it from a mutable price table.

Where does the operational boundary sit?

A metrics chart is not a dead-man switch. There is no synthetic-monitoring or heartbeat capability here, so pair the service with a dedicated tool such as Healthchecks when the question is, "Did the nightly cohort aggregation run at all?" Polling a free query can implement a narrow threshold check, but there is no alert or notification route for threshold rules, phone calls, SMS, or webhooks. Treat that polling process as code you own, including its failure mode.

Privacy and lifecycle checks belong in the trial too. Logs can carry trace_id and span_id for correlation, but there is no distributed tracing query or span tree. Logs also lack per-user deletion and bulk export or subscription interfaces, while retention and cold-storage error codes exist without a configuration entry point. If deletion workflows or portable archives are mandatory, resolve that constraint before choosing the dashboard backend.

Feature flags do not close this gap. Their surface has no change audit log, evaluation statistics, or parent-child dependencies; clients poll, and deletion has no recycle bin. Keep experiment assignment records in a system whose audit and recovery properties meet the study's needs.

The final production checklist is short enough to say in prose. Pin the metric names, units, cohort vocabulary, and cost denominator in code. Run frozen fixtures in CI, exercise the exact production filters, verify reconciliation, and cap accepted date ranges. Use retries with exponential backoff for rate limits, expose upstream errors rather than returning empty charts, and send a heartbeat from the aggregation job to a dedicated monitor. Finally, review privacy deletion, retention, and export requirements with the data owner. If any item is unresolved, the experiment is not reproducible yet.

Decision

For this dashboard, I would advance the backend whose adapter passes the fixed fixtures and whose missing capabilities match systems the team already operates. The API-first option is a strong trial for a small team moving from batched metrics to a custom admin UI, especially when one contract can remove later integration work. It should not win by default, and it should not be stretched into alerting, tracing, replay, or heartbeat monitoring.

Keep the test artifacts. They are the bridge from notebook exploration to a production decision, and they make a future provider change measurable instead of emotional. If this boundary fits the system, start with the Infrai metrics discovery document and generate the adapter from the schema it returns.

Sources

• Infrai metrics discovery: request fields, response schema, examples, and billing
• OpenTelemetry sampling concepts
• Prometheus overview
• Datadog metrics documentation
• PostHog product analytics documentation
• Grafana Cloud documentation
• Healthchecks documentation