Self-Hosted or Hosted App Log Search: Logistics Incident Evidence

Short answer: choose the least expensive operating model that can reliably reconstruct one customer incident end to end, not the option with the lowest ingestion quote. For a small logistics business, that means proving that shipment, API, worker, and AI events remain correlated and searchable for the required retention window before comparing self-hosting with a hosted search service. A missing handoff event costs more investigative time than a modest difference in storage price.

Start with the evidence path. A customer reports that shipment SHP-20418 showed the wrong delivery status after an address-classification request. The useful record is not a wall of application text; it is a bounded chain containing the request ID, shipment ID, trace ID, state transition, model decision metadata, worker attempt, and final customer-visible state. Collect those fields at emission time, minimize sensitive values, buffer them independently from the app, then index the retained copy for search. This flow works with a self-hosted stack, a managed search platform, or a hosted logs API.

The hard part is boring. Good.

How should a small business compare self-hosted and hosted app log search APIs?

Compare them with a reconstruction test, an operations budget, and a retention budget. The reconstruction test is primary: can an engineer begin with a customer ticket and retrieve the complete ordered event chain without guessing which service wrote which message? The operations budget covers upgrades, capacity, backups, access control, alerting, and restore drills. The retention budget covers ingestion, indexing, stored bytes, queries, data transfer, and any duplicated archive. A quote that exposes only one of those terms isn't comparable yet.

Use the same evidence fixture for every candidate. Seed a normal delivery, a duplicated webhook, a delayed queue task, a partial AI response, and a redacted customer field. Then walk the hypothetical SHP-20418 ticket exactly as support would: begin with the shipment ID, locate its trace, order all state transitions by event time, distinguish the first worker attempt from its retry, and connect the address classifier's prompt version to the status the customer actually saw. Now remove the classifier event and confirm that the eval fails for missing evidence rather than quietly returning a shorter timeline. Put it back, inject an email address in an unapproved field, and confirm that validation rejects the event before indexing. Finally, restore the fixture into a fresh search target and repeat the queries with an account that has investigator permissions but no access to other customer records. That sequence answers much more than a polished demo query: it tests correlation, completeness, minimization, authorization, and recoverability with one understandable case. Ask the same questions of every option: which state did the customer see, what input version produced it, which attempts ran, what was retried, and can an authorized reviewer export the evidence without exposing unrelated customers? I use pass/fail checks here because a notebook screenshot proves only that one query worked once. An eval fixture can run during deployment and after schema changes.

Test the chain.

Keep the fixture small enough to understand manually. Twenty to fifty events across three services is usually more revealing than a synthetic flood because the first failure in incident reconstruction is often correlation, not throughput. This is a test-design rule, not a capacity estimate; production sizing still needs measurements from the actual application. Your mileage may vary.

A practical acceptance rule is blunt: reject an option if any required event disappears, if correlation requires free-text guessing, or if a restore cannot reproduce the same ordered timeline. After that gate passes, compare cost and toil.

Evidence comes first.

Build a reconstruction slice before choosing storage

The following Python 3.12 example creates an append-only JSON Lines evidence file, removes fields that the incident query does not need, chains records with SHA-256, and retrieves a shipment timeline. It is deliberately storage-neutral. The output can remain a test fixture or feed whichever backend is under evaluation.

from __future__ import annotations

import hashlib
import json
from pathlib import Path
from typing import Any, Iterable

ALLOWED_FIELDS = {
    "timestamp",
    "event_id",
    "shipment_id",
    "trace_id",
    "service",
    "event_type",
    "attempt",
    "state",
    "model",
    "prompt_version",
    "input_tokens",
    "output_tokens",
}


def normalize(event: dict[str, Any]) -> dict[str, Any]:
    required = {"timestamp", "event_id", "shipment_id", "trace_id"}
    missing = required - event.keys()
    if missing:
        raise ValueError(f"missing correlation fields: {sorted(missing)}")
    return {key: event[key] for key in sorted(ALLOWED_FIELDS & event.keys())}


def append_evidence(path: Path, events: Iterable[dict[str, Any]]) -> None:
    previous_hash = "0" * 64
    with path.open("w", encoding="utf-8") as stream:
        for raw_event in events:
            event = normalize(raw_event)
            canonical = json.dumps(event, sort_keys=True, separators=(",", ":"))
            record_hash = hashlib.sha256(
                f"{previous_hash}:{canonical}".encode("utf-8")
            ).hexdigest()
            record = {
                "event": event,
                "previous_hash": previous_hash,
                "record_hash": record_hash,
            }
            stream.write(json.dumps(record, sort_keys=True) + "\n")
            previous_hash = record_hash


def find_shipment(path: Path, shipment_id: str) -> list[dict[str, Any]]:
    matches = []
    with path.open(encoding="utf-8") as stream:
        for line in stream:
            event = json.loads(line)["event"]
            if event["shipment_id"] == shipment_id:
                matches.append(event)
    return sorted(matches, key=lambda event: event["timestamp"])


shipment_id = "SHP-20418"
trace_id = "tr_7f31a2"
events = [
    {
        "timestamp": "2026-08-14T02:10:03+00:00",
        "event_id": "evt_001",
        "shipment_id": shipment_id,
        "trace_id": trace_id,
        "service": "tracking-api",
        "event_type": "address_received",
        "state": "pending_classification",
        "customer_email": "removed@example.invalid",
    },
    {
        "timestamp": "2026-08-14T02:10:04+00:00",
        "event_id": "evt_002",
        "shipment_id": shipment_id,
        "trace_id": trace_id,
        "service": "address-agent",
        "event_type": "classification_completed",
        "state": "depot_review",
        "model": "configured-model-alias",
        "prompt_version": "address-v7",
        "input_tokens": 184,
        "output_tokens": 22,
    },
    {
        "timestamp": "2026-08-14T02:10:06+00:00",
        "event_id": "evt_003",
        "shipment_id": shipment_id,
        "trace_id": trace_id,
        "service": "status-worker",
        "event_type": "status_published",
        "attempt": 1,
        "state": "awaiting_depot_review",
    },
]

fixture = Path("incident-evidence.jsonl")
append_evidence(fixture, events)
for event in find_shipment(fixture, shipment_id):
    print(event["timestamp"], event["service"], event["event_type"], event["state"])

The allowlist is the important part. GDPR Article 5 includes data minimization: personal data should be adequate, relevant, and limited to what is necessary. Logging every request body because it might help later is not an evidence strategy. Define the minimum incident questions first, keep stable identifiers and decision metadata, and omit raw addresses, emails, model prompts, and response bodies unless a documented purpose truly requires them. Retention and access rules still need legal and security review for the business's jurisdiction.

The hash chain detects accidental edits and gives the test harness an integrity signal, but it is not independent proof against an operator who can rewrite both records and hashes. Stronger assurance needs controls outside this tiny example, such as restricted write paths, separately protected archives, and audited access.

Don't oversell a checksum.

There is another boundary: JSON Lines is excellent for a fixture and modest local tests, but this linear scan is not a production search engine. Its purpose is to make the evidence contract executable before storage features complicate the discussion. Once the contract passes, translate the same assertions into queries against each candidate.

Compare control, toil, and total query cost

The names in the shopping list point to three different responsibility models. Self-hosted Loki puts the operating work with your team. Elastic Cloud represents a managed search platform. A hosted logs API puts ingestion and query access behind a service boundary. Amazon CloudWatch is a useful fourth pricing reference because its public pricing page separates log ingestion and related log operations; it reminds buyers to model billable dimensions rather than compare a single headline number. These are examples, not a ranking.

The catch is that no column wins universally. Stick with self-hosting when deployment control, local data placement, or existing operations expertise justifies the work. A managed search platform can fit when teams need flexible exploration and will actively govern schemas and retention. A hosted API can fit a narrow application evidence path, but it is not suitable when investigators require backend-specific query extensions or when export and deletion controls cannot meet policy.

Price comes later.

Model a normal week and an incident week separately, including duplicated events, retries, index expansion, long queries, and export. CloudWatch's public per-GB examples show why raw ingestion volume is only one input; exact charges and free usage conditions can change, so read the current pricing page during procurement. I'm not sure any generic calculator can predict a young app's incident-query pattern from average traffic alone. A one-week replay of sanitized production-shaped events resolves more uncertainty.

For AI-assisted logistics features, record token counts and a stable prompt version alongside the business transition, as the example does. That supports two separate evals: did the model decision satisfy the application rubric, and did the full system publish the correct shipment state? Token cost matters, but a cheaper model call that cannot be tied to the customer-visible event is nearly impossible to investigate. Keep prompts out of ordinary logs by default; version identifiers usually provide safer correlation, while a controlled evaluation store can retain approved test inputs.

Turn incident reconstruction into a deployment eval

Run the evidence test at notebook stage, in continuous integration, after deployment, and during restore drills. The fixture should assert event presence, permitted fields, ordering, correlation, retention behavior, and deletion behavior. It should also submit adversarial values: a shipment ID in a message but absent from the structured field, two events sharing an ID, timestamps arriving out of order, and an unexpected personal-data field. These checks catch schema drift before an investigator is under pressure.

One tempting shortcut is to verify only that search returns a known phrase. I started with that mental model, then the failure becomes obvious: phrase search cannot establish which retry changed state, which prompt version ran, or whether the result belongs to the same shipment. The correction is an ordered evidence assertion over stable structured fields. It is less glamorous than a dashboard and far more useful during reconstruction.

Measure the pipeline, too. Alert on rejected events, buffer age, ingestion lag, and the age of the newest searchable record, but define thresholds from the application's incident objective rather than copying defaults. If customer support promises reconstruction within a given window, the restore drill and query authorization path must fit inside it. No source supplied here establishes a universal threshold, so this is where the team needs its own service objective and measured baseline.

Keep failure handling outside the request's success path. The application should emit into a bounded buffer and expose loss or backpressure explicitly; silently dropping evidence is unacceptable, while blocking every customer request indefinitely for logging can turn an observability dependency into an application dependency. The exact policy depends on the event: an audit-relevant state transition may warrant a stronger delivery guarantee than a debug message. Document that distinction and test it.

Operate the evidence path, not just the search box

Before launch, walk one customer ticket from identifier to ordered timeline, verify that access is scoped, and confirm that sensitive fields never entered the fixture. Then replay the same test after a schema migration. Schedule restore exercises against retained data, review who can query or export it, and expire evidence when its documented purpose ends. This is operational work, but it is also the only way to know the search path survives beyond the notebook.

During an incident, preserve the query, result identifiers, relevant configuration versions, and the time range used. Avoid copying an unrestricted blob into a ticket. Afterward, add the failure shape to the eval fixture and check whether the evidence contract needs a new structured field. Don't add fields reflexively; every addition expands indexing cost and potential data exposure.

The final decision can remain simple: first require a complete, privacy-conscious reconstruction; next require repeatable deployment and restore evals; only then select the responsibility and cost model the team can sustain. Cheap search that loses the causal chain is expensive evidence.

References

• https://gdpr-info.eu/art-5-gdpr/
• https://aws.amazon.com/cloudwatch/pricing/