Short answer: build the internal service status page as a reconstruction tool, not as a live green light. For an education platform's nightly data pipeline, the useful unit is a versioned run with timestamped metric and log evidence; the dashboard should report the last durable outcome, show how fresh that conclusion is, and preserve contradictions instead of letting the newest event win.
That changes the meaning of uptime. A healthy Node.js process says almost nothing about whether the 07:00 enrollment snapshot is complete. The service contract is closer to “the scheduled run committed every expected partition before the teaching day began,” and each status must be traceable to evidence an operator can inspect after alerts, retries, and clock skew have rearranged the apparent story.
Start there.
Define the reconstruction contract before drawing the dashboard
The first artifact should be a run contract, not a widget. Give every scheduled execution a stable run_id; record its schedule, expected partition count, schema version, and terminal commit; then decide which observations prove each transition. Google SRE's four golden signals — latency, traffic, errors, and saturation — remain a useful vocabulary for the workers and their dependencies, but a batch pipeline also needs progress and freshness because a quiet, responsive worker may still be processing yesterday's data.
For a hypothetical enrollment export, metrics answer bounded questions: how many partitions were expected, how many reached validation, how long the oldest unfinished partition has waited, and whether worker capacity is saturated. Structured logs carry details that don't belong in metric labels, such as a partition key, a validation reason, or a transition from validating to ready_to_commit. Prometheus documents that every unique label combination creates a new time series, so learner identifiers and free-form error messages should stay out of labels. That isn't cosmetic hygiene; unbounded identity fields make the metric model harder to operate and search.
Use two clocks. event_time records when the worker says something happened, while observed_time records when the collection boundary received it. A delayed log can then remain delayed without being rewritten into the present. The ordering key should include run_id, attempt, and an event sequence allocated within that attempt; wall-clock time alone cannot settle concurrent retries.
The materialized status record can stay deliberately small:
from dataclasses import dataclass
from datetime import datetime
from enum import StrEnum
class RunState(StrEnum):
SCHEDULED = "scheduled"
RUNNING = "running"
COMMITTED = "committed"
FAILED = "failed"
AMBIGUOUS = "ambiguous"
UNKNOWN = "unknown"
@dataclass(frozen=True)
class RunStatus:
run_id: str
state: RunState
evaluated_at: datetime
newest_evidence_at: datetime | None
partitions_expected: int
partitions_committed: int
rule_version: str
evidence_ids: tuple[str, ...]
def is_fresh(self, maximum_age_seconds: int) -> bool:
if self.newest_evidence_at is None:
return False
age = (self.evaluated_at - self.newest_evidence_at).total_seconds()
return 0 <= age <= maximum_age_seconds
The Node.js admin service need not calculate status in the browser. It should read this projection, return its evaluation timestamp and rule version, and query a bounded set of supporting events only when an operator opens a run. Keeping policy out of client-side color logic makes the same state available to the page, alerts, and runbook automation. It also means a browser refresh doesn't trigger an unbounded raw-log scan.
Unknown must be a real state.
Zero completed partitions is a measurement. Missing partition data is an evidence gap. If both render as zero, a collector delay becomes indistinguishable from a job that has not started, and incident reconstruction begins with a false premise.
How should an internal admin uptime dashboard combine metrics, logs, and service status?
Join them by run and window, then preserve their different jobs. Metrics are compact numerical history; logs are discrete claims about transitions. The projection should use metrics to establish progress and timing, logs to explain exceptional transitions, and a durable commit observation to close the run. It should never label a run successful merely because the process probe responds or the completed counter reaches its expected value before the destination confirms the commit.
Imagine run enrollment-2026-08-19 expects 24 partitions. At 01:17, the progress counter shows 18 validated. Event evt-4812 says attempt 1 lost its lease on partition 18, but its observed_time is 01:23 because delivery was delayed. Attempt 2 begins at 01:25. At 01:31, a buffered success event from attempt 1 arrives after a newer checkpoint from attempt 2. “Last event wins” now produces a confident answer from contradictory evidence; sorting exclusively by event_time hides transport delay, while sorting exclusively by observed_time invents execution order.
The right result is ambiguous until a terminal commit can be tied to the authoritative attempt. The page should show the two attempt lanes, both clocks, the conflicting evidence IDs, and the exact rule that withheld a healthy status. This is the long paragraph an implementation deserves, because the common shortcut is subtle: teams often retain the raw events yet destroy their diagnostic value in the projection by collapsing attempts, accepting a late event as current, or overwriting a status row without retaining the evidence set that produced it. An append-only event store plus a replaceable projection avoids that trap. The projection is disposable; the evidence is not. If policy changes from status-v3 to status-v4, replaying the same immutable records should explain why the displayed result changed.
Keep the operator view sparse. The first screen needs the current state, last durable commit, expected and committed partitions, evidence freshness, active attempts, and rule version. Selecting a run may reveal its ordered state transitions and a narrow log slice. Raw log search still matters, but placing a limitless search box at the center of an uptime page asks a tired operator to reconstruct the data model manually.
A compact decision table makes the boundaries explicit:
| Evidence design | What it can establish | Failure mode or limit |
|---|---|---|
| Process probe only | The admin or worker process answers now | Cannot prove that the scheduled dataset was committed |
| Metrics projection | Progress, rates, latency, freshness, and saturation across a window | Loses irregular context; high-cardinality identity fields don't belong in labels |
| Structured logs only | Detailed transitions and error context | Late, duplicate, or missing delivery can produce a misleading latest event |
| Immutable events plus a materialized status | Fast current reads with replayable incident evidence | Requires schema governance, retention planning, and deterministic projection rules |
No row wins everywhere. The catch is the extra operational ownership: a process probe is still appropriate for load-balancer decisions, and a metrics-only display may be sufficient for a stateless request service whose health contract is immediate. The event-plus-projection design is not suitable when the team cannot own event schemas, replay tests, and retention; in that case, stick with a smaller metrics-and-logs view and state clearly that it supports triage rather than authoritative reconstruction. Richer evidence also costs more to ingest and retain. CloudWatch's public pricing, for example, treats log ingestion as a metered dimension, which is a useful reminder to estimate event volume before enabling verbose payloads.
Test the lies your status model is tempted to tell
Happy-path screenshots prove very little. Feed the projection duplicates, a missing start event, a checkpoint older than the run, two active attempts, an expected partition count of zero, and a commit that arrives after the viewing window. Assert the state and the evidence IDs together. Otherwise a test can pass because it got failed for the wrong reason.
This table stakes test is small enough to run without an observability backend:
from dataclasses import dataclass
@dataclass(frozen=True)
class Evidence:
evidence_id: str
attempt: int
kind: str
def classify(events: list[Evidence], expected_attempt: int) -> str:
commits = {event.attempt for event in events if event.kind == "commit"}
active = {event.attempt for event in events if event.kind == "started"}
if expected_attempt in commits:
return "committed"
if len(active) > 1 or any(attempt != expected_attempt for attempt in commits):
return "ambiguous"
if expected_attempt in active:
return "running"
return "unknown"
def test_old_attempt_commit_does_not_mark_current_attempt_healthy() -> None:
events = [
Evidence("evt-4812", 1, "started"),
Evidence("evt-4819", 2, "started"),
Evidence("evt-4821", 1, "commit"),
]
assert classify(events, expected_attempt=2) == "ambiguous"
Production logic will need timestamps, partition membership, deduplication, and schema validation; the point of the reduced test is to make the precedence rule impossible to miss. OpenTelemetry defines logs, metrics, and traces as telemetry signals and describes correlation through shared context, but adopting a common signal model does not decide an application's truth policy. The pipeline still has to define which attempt is authoritative and which observation proves durability.
Test the presentation boundary too. A stale projection must render as stale or unknown, never healthy. Missing values must remain null rather than becoming zero. Access control should apply to the supporting records as well as the summary page, since structured education logs can carry fields that don't belong on a broadly visible internal dashboard. Finally, cap every detail query by run and time range so a browser action has predictable operational weight.
How can the team roll out reconstruction without replacing current alerts?
Deploy the new projection in shadow mode and keep the current status signal unchanged. For a representative set of nightly runs, store both decisions, review every disagreement, and classify the cause as a policy difference, missing evidence, late delivery, or an invalid assumption about the commit boundary. It isn't possible to prescribe a universal observation window from the available standards; the pipeline schedule, normal completion distribution, and teaching-day deadline must determine it.
Then migrate in three controlled moves: expose the shadow state to internal operators without paging from it; replay retained events through the same rule version used online; and switch alerts only after ambiguous and unknown states have explicit routing. Keep rollback at the projection boundary. The append-only evidence and existing alerts should remain untouched until the new interpretation has earned trust.
The finished page is simple because the evidence model is not. It tells an operator what ran, what committed, how recently the claim was evaluated, and which records justify it. For a nightly education pipeline, that is a more defensible definition of uptime than a green process check — and a far better starting point for the morning incident review.
