Shipping code is an event. Owning software is a continuing obligation.
A software project can pass every sprint demo and still fail its customer. Perhaps nobody defined what available meant. Perhaps the vendor delivered source code but not infrastructure access. Perhaps AI generated features faster than the team could verify them. Perhaps an impressive dashboard measured deployments while customers waited hours for business-critical jobs.
This article follows one fictional enterprise project from initial discovery through procurement, negotiation, implementation, launch, incidents, and handover. Its example numbers, targets, and conversations are illustrative proposals, not industry benchmarks or legal advice. Along the way, we'll connect the decisions to published guidance from DORA, NIST, OWASP, and Google SRE. No prior familiarity with enterprise-process jargon is assumed: terms are explained when they first matter, with a concrete example of what a team actually does.
How to read this guide
Each phase answers five practical questions: Why are we doing this? Who decides? What do we produce? How do we know it worked? Where can AI help without becoming the decision-maker? The project is fictional, so its numerical targets are examples of negotiated choices, not universal best practices. If you are a developer, follow the engineering decisions; if you are a product or procurement lead, pay attention to who accepts risk and signs off on results.
A validation gate is simply a checkpoint before the next costly or risky step. It is not necessarily a meeting or an approval committee: it can be a passing automated test, a customer confirming a prototype, or an operations engineer demonstrating a successful restore.
The case: a logistics platform with real consequences
Northstar Logistics operates 2,000 vehicles across several countries. Its operations staff coordinate shipments through spreadsheets, email, and a legacy application. The company hires Harbor Software, an external development partner, to build a shipment visibility and exception-management platform.
Stakeholders:
| Role | Accountable for |
|---|---|
| Executive sponsor | Funding and business outcome |
| Product owner (Northstar) | Scope, priorities, acceptance |
| Technical owner (Northstar) | Architecture, access, continuity |
| Delivery lead (Harbor) | Delivery plan, staffing, risk communication |
| Security/data owners | Security controls, privacy, retention |
| SRE/on-call owner | Production response and recovery |
| End users | Workflow feedback and UAT |
A stakeholder is anyone affected by the project or empowered to make a decision. An accountable owner is the person who must make sure an outcome happens, even if others do the work. UAT (user acceptance testing) means real users trying representative workflows to decide whether the product meets their needsānot merely whether the code runs.
Business hypothesis: reduce the median time to identify and assign a delayed shipment from 45 minutes to 15 minutes within 90 days of rollout, without increasing false alerts or manual workload.
Notice that the objective is not ābuild a dashboard.ā A dashboard is one possible intervention. The outcome is faster, trustworthy decisions.
The organizational question: management is not ownership
Before Northstar negotiates a feature, it must decide who can make which decisions. A delivery manager can keep a schedule on track without understanding the production system. An architect can understand every service without having authority to change priorities. Neither is a substitute for the other.
Project management coordinates a time-bound initiative: commitments, dependencies, budgets, and delivery risks. Product management decides which customer problems deserve investment and measures whether solving them creates value. Engineering management develops the team's capacity to deliver sustainably: hiring, coaching, staffing, and working conditions. Technical ownership means named people can explain design choices, approve or implement changes, and maintain the system over time. Service ownership adds operational accountability: someone knows when a service is failing and can lead its recovery. Executive governance determines which business risks and investments the organization will accept.
These are responsibilities, not necessarily six job titles. A small company may combine roles; a regulated enterprise may separate them. The critical requirement is to make decision rights explicit.
What modern management research actually suggests
DORA's 2024 research associates stable priorities, user focus, and transformational leadership with stronger outcomes; it also warns that platform initiatives can introduce delivery trade-offs. Its 2025 AI research argues that AI amplifies existing organizational strengths and weaknesses rather than replacing them. These are research findings and associations, not a guarantee that adopting one management framework causes success.
DORA's guidance on loosely coupled teams emphasizes teams that can test, change, and deploy with fewer cross-team dependencies. Loosely coupled does not mean isolated: it means teams agree on clear interfaces so that routine work does not require constant permission or coordination.
The Team Topologies model offers a useful vocabulary: a stream-aligned team owns a flow of customer value; a platform team supplies reusable internal capabilities; an enabling team helps others learn specialized skills; and a complicated-subsystem team concentrates scarce expertise. These are design options, not an org chart to copy blindly.
A management failure in the Northstar story
Two months into development, the vendor's delivery manager reports: āWe're green: 85% of tickets completed.ā The operations director responds: āDispatchers still cannot trust shipment arrival estimates.ā The technical owner adds: āThe event processor has no replay procedure, and only Harbor can deploy it.ā
All three statements can be true. The project is tracking output (tickets), while the business needs an outcome (trustworthy decisions), and engineering needs operability (the ability to diagnose and restore the system).
A better weekly review shows all three:
| View | Question | Example signal | Decision owner |
|---|---|---|---|
| Product | Are dispatchers better off? | % of late shipments detected before escalation | Northstar product owner |
| Delivery | Are changes flowing predictably? | Change lead time and blocked work age | Harbor delivery lead + engineering lead |
| Technical | Can we change the system safely? | Contract-test pass rate; independent deploy drill | Northstar technical owner |
| Operations | Can we recover? | Restore drill outcome and time | Named service owner |
| Commercial | Are we buying the right result? | Cost against accepted business milestones | Sponsor + procurement |
The signals are examples to instrument, not benchmark targets. Teams should establish a baseline before deciding what counts as improvement.
A negotiation about ownership, not just deadlines
Vendor delivery lead: āWe can meet the date if we keep infrastructure under our account and hand it over after launch.ā
Customer technical owner: āThat reduces your setup time, but it creates a dependency we cannot accept for a critical service.ā
Product owner: āWhat does changing that cost, and does it delay the pilot?ā
Vendor delivery lead: āProvisioning in your account adds two weeks. We can still pilot with a reduced reporting scope.ā
Executive sponsor: āAgreed, provided we see an independent deployment and restore demonstration before production acceptance.ā
This is a trade-off negotiation: the sponsor chooses between time, scope, and long-term operational control. A useful record is a one-page decision with options, consequences, decision-maker, deadline, and a test proving the choice worked. The vendor is still responsible for delivery, but the customer is not required to depend on the vendor forever.
Decision rights: use a DACI, not a maze of approvals
DACI is a lightweight decision framework: Driver coordinates the decision, Approver makes the final call, Contributors provide expertise, and Informed stakeholders receive the outcome. It differs from a RACI responsibility chart (Responsible, Accountable, Consulted, Informed), which is often used to describe ongoing work rather than one discrete decision.
| Decision | Driver | Approver | Contributors | Evidence required |
|---|---|---|---|---|
| Prioritize shipment exception alerts | Product manager | Product owner | Dispatchers, engineers | Workflow trial and baseline |
| Choose event processing design | Tech lead | Technical owner | Security, operations, vendor | ADR and failure-mode test |
| Accept lower pilot scope | Delivery lead | Sponsor + product owner | Finance, tech lead | Revised SOW and acceptance tests |
| Release critical service | Release coordinator | Authorized service owner | QA, security, on-call | Passing gates, rollback rehearsal |
| Accept contractual SLA exception | Account manager | Contractually authorized business owner | Legal, operations | Impact, remedy, expiry date |
The table is illustrative. Real approval authority must match company policy and contracts. Escalation should occur when a decision crosses an agreed boundary (budget, customer impact, compliance, architecture), not whenever an engineer changes a line of code.
Where AI belongs in modern management
AI can draft meeting summaries, identify conflicting requirements, prepare options for negotiation, flag stale risks, cluster support tickets, and generate proposed ADRs. It can also help explain technical consequences to nontechnical stakeholders. But a generated summary is not a signed decision, a predicted delivery date is not a commitment, and a model's preferred architecture is not organizational authority.
Northstar sets a simple policy: AI-generated decisions must cite their inputs, distinguish facts from suggestions, identify a human approver, and remain reviewable. Sensitive contract terms or customer data are not sent to tools without approved data-handling controls.
To test whether AI helps rather than adds bureaucracy, the team measures: decision latency (time from an identified decision to an approved choice), decision reversals (choices reopened due to missing evidence), meeting-to-action completion (actions completed by the agreed date), and review correction rate (AI drafts requiring material correction). Measure these at team or workflow level, not as a leaderboard of individuals. Compare similar work before and after adoption, and inspect adverse effects such as missed context or extra review time.
Validate the management model itself
At the end of the pilot, ask five practical questions: Can a dispatcher demonstrate business improvement? Can the team deploy a small change without a cross-company approval chain? Can the customer restore the service without vendor credentials? Is there a named human who can accept an operational risk? Can the organization explain which decisions AI supported and who approved them?
If any answer is no, the issue may be organizational rather than a missing feature. A project can have excellent ceremonies and still lack meaningful ownership.
Phase 0 ā Qualify the problem before selling a solution
The vendor runs discovery interviews with dispatchers, operations managers, security, and the legacy-system team. It maps a shipment's journey and records where data is missing or delayed.
Sample conversation:
Customer: We need live tracking for every vehicle.
Engineer: What decision becomes possible with live tracking that isn't possible today?
Customer: Dispatchers need to react before a delivery misses its window.
Engineer: Would a two-minute update plus a reliable delay alert meet that need? Continuous streaming changes mobile connectivity, storage, and support costs.
Customer: Let's test it with dispatchers rather than assume.
An artifact is a durable piece of project evidence: a document, recorded decision, prototype, test report, or dashboard. A baseline is the measured starting point against which later improvement is compared. An assumptions register lists claims the team has not yet verified, such as āthe carrier API can handle 100 requests per second.ā
Artifacts: problem statement, current-state workflow, stakeholder map, data inventory, baseline metrics, assumptions register, risks, and a one-page business case.
Validation gate: sponsor and product owner confirm the problem, a measurable outcome, and the baseline measurement method. Run a small user observation or prototype before committing to architecture.
AI contribution: transcribe approved interviews, cluster recurring pain points, draft process maps, identify contradictory statements. Human check: verify every consequential finding against recordings, participants, or system evidence. Never send confidential interviews to an unapproved model.
Phase 1 ā Negotiate scope, economics, and control
The parties choose a phased engagement instead of treating a fixed-price contract as a substitute for discovery.
Illustrative commercial proposal:
- Discovery: two weeks, fixed fee, producing a validated backlog, risk register, and delivery estimate.
- Pilot: eight weeks, limited to 100 vehicles and two dispatcher teams; price and scope capped.
- Expansion: funded after pilot acceptance, with explicit change control.
- Support: separate operational agreement defining hours, severity, escalation, and service credits if applicable.
Sample negotiation:
Customer procurement: We need a fixed deadline and fixed cost for all countries.
Vendor lead: We can fix the pilot budget and deadline. We cannot responsibly fix the full integration scope before inspecting the legacy interfaces. We propose a two-week discovery with an option to stop.
Customer CTO: We also need the ability to replace your team.
Vendor lead: Agreed. Repositories, infrastructure accounts, pipelines, documentation, and observability will live under your organization's control. We'll define an exit rehearsal as an acceptance condition.
Customer procurement: How do we handle new compliance requirements?
Vendor lead: We'll document their impact on scope, risk, cost, and schedule, and obtain written approval before changing the baseline.
A statement of work (SOW) spells out what the supplier will deliver, by when, and under what acceptance conditions. Scope is the agreed boundary of the work. Change control is how the parties decide whether a new request changes the price, timeline, or risk. IP (intellectual property) concerns rights to code, designs, and other deliverables; third-party licenses can limit how dependencies may be used or redistributed. An exit strategy makes it possible to switch providers without losing access or knowledge.
For example, āadd customs integration for another countryā is not automatically a small change just because the screen looks similar. The customer describes the business need; the supplier estimates integration, legal, testing, and support impacts; the authorized owners approve, defer, or trade it against existing scope. Negotiation should expose trade-offs, not disguise uncertainty as certainty.
Contract checklist: statement of work, deliverables, acceptance windows, change-request mechanism, IP ownership and third-party licensing, confidentiality, data processing, subcontractors, security incident notice, service/support responsibilities, termination assistance, source and infrastructure access, warranties, and dispute escalation. Legal counsel should adapt these terms to the jurisdiction.
Validation gate: both parties sign the scope baseline, assumptions, decision rights, acceptance method, and exit responsibilities. No one should confuse ācode handed overā with āsystem operable independently.ā
Phase 2 ā Turn a business goal into testable specifications
Functional example
Story: As a dispatcher, I want delayed shipments ranked by urgency so I can assign interventions.
Acceptance criteria:
- Given a shipment predicted to miss its delivery window by more than 20 minutes, when the forecast is updated, the shipment appears in the exception queue.
- A dispatcher can acknowledge an alert and assign an owner.
- Replayed telemetry does not create duplicate active alerts for the same shipment and rule version.
- If location data is stale, the UI clearly shows its age rather than presenting it as current.
- Every acknowledgment and assignment records actor, time, and shipment identifier.
Nonfunctional example
A functional requirement describes what the system does: āa dispatcher can assign an alert.ā A nonfunctional requirement (NFR) describes how well or under what constraints it must work: speed, reliability, accessibility, security, or recoverability. An acceptance criterion is an observable condition that lets the customer say yes or no. In the example above, the stale-data warning is not decorative: it prevents a dispatcher from mistaking yesterday's location for a live position.
A percentile summarizes a distribution. If API latency at p95 is 400 milliseconds, about 95% of measured requests completed in 400 ms or less, under the specified test conditions; the slowest 5% may take longer. Averages alone can hide those slow experiences. RTO (recovery time objective) is the targeted time to restore a service after disruption; RPO (recovery point objective) is the maximum targeted amount of recent data that could be lost, expressed as time. Neither is a guarantee unless the recovery design and tests support it.
| Concern | Proposed measurable requirement | Verification |
|---|---|---|
| API latency | p95 under 400 ms for selected reads at agreed load | Load test and production traces |
| Availability | 99.9% monthly for defined customer-facing API | Synthetic checks plus request SLIs |
| Data freshness | 95% of accepted telemetry reflected within 120 seconds | Event-to-visibility timestamps |
| Authorization | Users can access only permitted tenants and roles | Negative authorization tests |
| Recovery | RTO 2 hours; RPO 15 minutes for critical state | Restore and failover exercise |
| Auditability | Changes to assignments traceable to an actor | Audit log test |
| Accessibility | Target WCAG 2.2 AA for relevant web workflows | Automated and manual review |
These are starting negotiation values, not promises that every system can or should meet. Define the population, exclusions, load profile, and measurement window before approving them.
Validation gate: product, engineering, security, and operations sign off on representative examples, failure behavior, data ownership, and measurable NFRs. Trace each critical requirement to a test or observable signal.
AI contribution: propose edge cases, convert approved requirements into draft Gherkin scenarios, review ambiguities. Human check: independently compare generated tests to the requirement; an AI can write a test that faithfully validates the wrong behavior.
Phase 3 ā Architecture and risk decisions
The team chooses an incremental architecture: a telemetry ingestion adapter, event stream, shipment-state processor, alert rules service, operational API, and web UI. It deliberately avoids splitting every function into a microservice.
flowchart LR
T[Vehicle telemetry] --> I[Ingestion adapter]
I --> Q[Durable event stream]
Q --> P[Shipment state processor]
P --> D[(Operational data store)]
P --> R[Alert rules]
R --> D
D --> A[API]
A --> U[Dispatcher UI]
P --> O[Metrics and traces]
A --> O
An architecture decision record (ADR) is a short explanation of a significant technical choice, its alternatives, and its consequences. It helps the next engineer understand why the system looks this way. A durable event stream retains messages so processors can recover or replay them. Idempotency means processing the same business event twice has the same intended effect as processing it once. A dead-letter queue isolates messages that repeatedly fail so they can be inspected instead of blocking everything. Eventual consistency means different parts of the system may briefly disagree while updates propagate.
Imagine a vehicle losing connectivity, then uploading the same position twice. Without idempotency, the system could create two delay alerts. With a stable event identifier and deduplication rule, the second upload does not cause a second business action.
ADR-004 ā Asynchronous telemetry processing
- Context: connectivity is intermittent; messages may be duplicated or reordered.
- Decision: durable events, explicit idempotency keys, bounded retries, dead-letter handling, and freshness indicators.
- Alternatives: synchronous direct writes; batch imports.
- Consequences: additional operational complexity, eventual consistency, replay and schema-evolution obligations.
- Revisit when: volume, delay, or operational burden invalidates the assumptions.
Threat model: tenant isolation, forged telemetry, compromised API tokens, overprivileged service accounts, leaked location history, supply-chain compromise, and AI-agent access to production secrets.
Validation gate: architecture review checks failure modes, load estimates, security boundaries, ADR trade-offs, and a working vertical slice. A diagram alone is not evidence.
Phase 4 ā Agree on how the team communicates
Communication is an engineering control when it preserves decisions and surfaces uncertainty. A cadence is the agreed frequency of communication. Async updates are written updates that do not require everyone in a meeting. A risk register is a living list of potential problems, their probability or impact, owners, and mitigation plans. A steering meeting is where sponsors resolve decisions the delivery team cannot make alone.
A useful status report does not merely say ā80% done.ā It states what was verified, what could prevent success, who must decide, and by when. Amber means attention is required; it does not mean failure.
| Cadence | Participants | Output |
|---|---|---|
| Daily async update | Delivery team | Done / next / blocked / risk |
| Weekly product review | Product + engineering | Demo, acceptance, priority decisions |
| Weekly risk review | Delivery, technical and security owners | Updated risk register and mitigations |
| Architecture decision as needed | Relevant engineers + owner | ADR with alternatives and consequences |
| Monthly steering meeting | Sponsor + vendor leadership | Budget, outcomes, escalations, scope decisions |
| Incident bridge | On-call, incident commander, stakeholders | Timeline, mitigations, next update time |
Example weekly update:
Status: Amber ā telemetry integration
Completed: replay-safe ingestion prototype, 14 integration tests.
Evidence: duplicate events no longer create duplicate shipment-state transitions in the test environment.
Risk: partner API rate limit is undocumented; pilot throughput may be affected.
Decision needed by Thursday: authorize a partner load test or accept a lower initial fleet limit.
Owner: Northstar integration lead. Next update: Friday 15:00.
Validation gate: decisions have owners and deadlines; risks have mitigation or explicit acceptance. A green status without evidence is not a control.
Phase 5 ā Build with AI, without outsourcing accountability
AI can help throughout the lifecycle, but different tasks require different permissions and verification. Here AI agent means software that can use tools and take multiple steps (for example, reading an issue, editing a branch, and running tests), not just a chat assistant answering a question. A human-in-the-loop control means an authorized person must review or approve a consequential action. A code owner is the designated reviewer responsible for a portion of the repository.
The central risk is verification debt: code, tests, and documents can be generated faster than the team can check their correctness. A polished AI summary is not evidence that an interviewee said something; a passing AI-written test is not evidence that the intended business rule was correct. Give agents the least permissions they need and preserve an audit trail of actions.
| Stage | Useful AI work | Required human verification |
|---|---|---|
| Discovery | Interview synthesis, gap detection | Source-backed findings and user confirmation |
| Specifications | Edge cases, acceptance-test drafts | Product owner approves intended behavior |
| Architecture | Alternatives, failure scenarios | Engineers test assumptions and own ADRs |
| Coding | Scaffolding, refactoring, migrations | Code review, tests, security checks |
| Testing | Test generation, fuzz-case suggestions | Independent oracle and meaningful assertions |
| Operations | Incident summaries, query suggestions | On-call confirms diagnosis and commands |
| Reporting | Release-note and metric summaries | Validate against telemetry and change records |
Example AI-agent policy:
agent_policy:
approved_tools: [repository_reader, test_runner, issue_tracker]
default_permissions: read_only
allowed_changes: [feature_branch, pull_request]
prohibited: [production_deploy, secret_read, permission_grants]
required_checks:
- human_code_owner_review
- automated_tests
- dependency_and_secret_scans
- security_review_for_sensitive_changes
audit:
record_agent_identity: true
record_tool_actions: true
This is a conceptual policy, not a vendor-specific enforceable configuration. Enforce permissions in the actual identity, repository, CI, and infrastructure systems. Treat retrieved documents, issue descriptions, and tool output as untrusted inputs to agents.
A useful AI evaluation compares similar work over time: cycle time, review burden, escaped defects, rework, and customer outcomes. Do not infer productivity from generated lines of code or number of prompts. Use matched tasks or staged pilots where possible; control for task complexity and team differences.
DORA's 2025 research reported widespread AI adoption and positive associations with throughput and product performance, while finding a negative association with delivery stability. That is a warning against equating faster code production with a healthier delivery system. DORA 2025.
OWASP SAMM's AI-assisted development guidance similarly emphasizes approved tools, scoped permissions, source and licensing policies, and deterministic verification beyond AI-based review: OWASP SAMM guidance.
Phase 6 ā Define objective release gates
A quality gate is a condition that must pass before a change proceeds. CI/CD (continuous integration / continuous delivery or deployment) automates building, checking, and preparing or releasing changes. A pull request (PR) is a proposed code change reviewed before merging. A contract test checks that two components agree on their interface. A canary release exposes a new version to a small fraction of traffic first; a rollback returns to the previous version, while roll-forward fixes the issue in a new version. Artifact provenance records how a deployable package was built and from which source.
The team creates a Definition of Done for a feature and a stricter production release gate for a deployable change.
Feature done: reviewed acceptance criteria, passing relevant tests, observability included, documentation updated, no unapproved critical security findings, and product-owner acceptance for user-facing behavior.
Production gate:
- Build is reproducible and artifacts have provenance.
- Unit, integration, contract, and risk-relevant end-to-end tests pass.
- Dependency, secrets, and static-security checks satisfy policy.
- Performance checks meet the agreed load envelope.
- Database migrations are forward/backward compatible or have a safe plan.
- Feature flags, canary rollout, and rollback or roll-forward procedures are rehearsed.
- Dashboards and alerts cover the new critical path.
- Named owner accepts residual risks and authorizes release.
Failure example: the generated code passes unit tests but accidentally allows one tenant to query another tenant's shipments. The negative authorization suite fails. The pipeline blocks release even though the feature demo looks correct.
Validation gate: store the evidence in the PR, build, test report, and release record. Exceptions require a named approver, expiry date, mitigation, and follow-up issue.
NIST's Secure Software Development Framework provides a useful organizing model: prepare the organization, protect software, produce well-secured software, and respond to vulnerabilities.
Phase 7 ā SLIs, SLOs, SLAs, and operational negotiation
An SLI (service level indicator) is the number measured in production, such as the percentage of successful requests. An SLO (service level objective) is the internal target for that indicator, such as 99.9% in 30 days. An SLA (service level agreement) is a customer-facing contractual promise that defines coverage, measurement, support, exclusions, and possibly remedies if commitments are missed. An SLO does not automatically become an SLA.
An error budget is the amount of unreliability allowed by an SLO. At 99.9% request success, the budget is 0.1% failed eligible requests over the chosen window. Teams can use that budget to decide whether to prioritize new features or reliability improvements. Sev-1 is a severity classification for a major incident; each organization must define its actual impact threshold. Acknowledgment time is how quickly someone responds to an incident report, not how quickly the system is fixed.
For the pilot, Northstar and Harbor propose:
- Availability SLI: successful eligible API requests / all eligible API requests, with explicit rules for timeouts and excluded traffic.
- Availability SLO: at least 99.9% over a rolling 30-day window.
- Freshness SLI: share of accepted events visible to dispatchers within 120 seconds.
- Freshness SLO: at least 95% over seven days.
- Support SLA: severity-based response and escalation commitments, distinct from time-to-resolution promises.
For one million eligible requests in the SLO window, a 99.9% target allows 1,000 unsuccessful requests. If the system records 1,300, it misses the objective. A 99.9% time-based target corresponds to about 43.2 minutes of unavailability in 30 days, but request-based and time-based budgets are not interchangeable.
Example support negotiation:
Customer: We want all incidents fixed within one hour.
Operations lead: We can commit to a 15-minute acknowledgment for Sev-1 incidents with 24/7 coverage, but cannot guarantee every root cause is resolved in an hour. Let's separately define restoration targets, escalation, communication intervals, and service remedies.
Customer: What if we lose tracking during a carrier outage?
Operations lead: The system should mark the data stale, queue retries where possible, and report dependency impact. The contract needs to distinguish our controllable failures from upstream outages without hiding the user's experience.
Illustrative severity matrix:
| Severity | Example | Acknowledgment target | Update cadence |
|---|---|---|---|
| Sev-1 | Widespread loss of critical dispatch functionality | 15 min, 24/7 | Every 30 min |
| Sev-2 | Major degradation with workaround | 1 hour during support coverage | Every 2 hours |
| Sev-3 | Limited issue | 1 business day | Agreed in ticket |
Do not copy these commitments into a contract without verifying staffing, monitoring, holidays, time zones, and actual ability to respond.
Error-budget policy: when the service exhausts its agreed budget, pause discretionary risky releases and prioritize reliability, with exceptions for urgent security and recovery work. Google's SRE error-budget policy example illustrates how to formalize that decision without using reliability as a punishment.
Phase 8 ā Measure the entire system, not just the developers
Metrics are useful only when they support decisions. A KPI (key performance indicator) is a selected metric tied to a goal, such as reducing time to assign delayed shipments. Instrumentation is the code or configuration that records events, logs, traces, and metrics. A trace follows a request through components; a log records an event; a metric aggregates a measurement over time. DORA is a research program that studies software-delivery performance and provides operational metrics; its measures are not individual developer scorecards.
Create a measurement plan before the pilot. Every metric needs a definition, event source, owner, review cadence, and response when it moves.
| Layer | Metric | Formula or method | Data source | Decision it informs |
|---|---|---|---|---|
| Business | Median alert-to-assignment time | Median(assignment time ā actionable alert time) | Audit events | Does the product improve dispatch? |
| Business | False-alert rate | False alerts / reviewed alerts | Labeled review sample | Are we creating noise? |
| Product | Weekly active dispatchers | Distinct users completing core action | Privacy-aware events | Is the workflow adopted? |
| Delivery | Change lead time | Commit-to-production elapsed time | Git + deployment records | Where is delivery delayed? |
| Delivery | Deployment frequency | Production deployments per period | CI/CD | Can changes move safely? |
| Delivery | Change fail rate | Deployments requiring immediate intervention / deployments | Incidents + deployments | Are releases destabilizing service? |
| Delivery | Failed deployment recovery time | Time to recover from a failed deployment | Incident timeline | How quickly can we recover? |
| Delivery | Deployment rework rate | Unplanned incident-related deployments / deployments | Deployment classifications | How much emergency rework occurs? |
| Reliability | API success ratio | Successful eligible requests / eligible requests | Gateway telemetry | Is the SLO met? |
| Reliability | p95 latency | 95th percentile of eligible request durations | Traces/metrics | Is the experience responsive? |
| Data | Duplicate effect rate | Duplicate state changes / replayed or duplicate messages | Processor audit | Is idempotency working? |
| Security | Critical finding age | Days since confirmed finding opened | Vulnerability tracker | Are critical risks being reduced? |
| Cost | Cost per tracked vehicle-day | Allocated service spend / active vehicle-days | Cloud billing + usage | Does scaling make economic sense? |
| AI | AI-assisted rework rate | AI-assisted changes requiring rework / AI-assisted changes | PR classification + issues | Is AI creating downstream burden? |
DORA now defines five software-delivery performance metrics, including deployment rework rate. Use its current definitions rather than an outdated four-metric slide: DORA metrics guide.
Example instrumentation
shipment_alert_created{shipment_id, rule_version, timestamp}
shipment_alert_assigned{shipment_id, actor_id, timestamp}
telemetry_accepted{event_id, source, timestamp}
telemetry_visible{event_id, timestamp}
deployment_completed{service, version, timestamp}
incident_opened{severity, service, timestamp}
incident_restored{incident_id, timestamp}
This is an event schema illustration, not a recommendation to put shipment IDs, event IDs, or actor IDs into Prometheus metric labels. High-cardinality identifiers belong in appropriately protected logs, traces, or event stores; aggregate metrics should use bounded labels. Protect personal and location data with access controls and retention limits.
How to validate a metric
A measurement window is the period being analyzed. A denominator is the population included in a rate; changing it can make a metric look better without improving reality. p90 means 90% of observations are at or below that value. High-cardinality labels are labels with very many distinct values (for example, one per shipment); putting them on time-series metrics can overwhelm monitoring systems.
Worked example: Suppose 80 of 100 reviewed alerts were actionable and 20 were false alarms. The false-alert rate is 20/100 = 20%, provided the sample is representative. If the team quietly excludes difficult cases, that number becomes misleading. Suppose assignment times are 5, 10, 15, 20, and 120 minutes. The median is 15 minutes, but the 120-minute outlier still matters for operations; track a tail percentile too.
For alert-to-assignment time, verify the start event represents an actionable alert, not a background recalculation; use consistent clocks; exclude test data; inspect missing assignments; compare median and p90; segment by region and shift; and sample raw records manually. Without these checks, a falling median might simply mean the team stopped recording difficult cases.
For AI-assisted changes, establish a baseline and compare review time, escaped defects, change fail rate, and outcomes. Label changes consistently, note mixed human/AI work, and avoid ranking individual engineers. An observational improvement is not proof that AI caused it.
Phase 9 ā Run a pilot and decide with evidence
A pilot is a limited real-world rollout designed to test assumptions before broad deployment. A go/no-go decision is an explicit choice to proceed, delay, or stop based on agreed evidence. A control group, where feasible, is a comparable group not yet receiving the change; it helps distinguish the product's effect from unrelated improvements. A pilot without predetermined success conditions can easily become a demonstration where every result is interpreted as success.
Pilot plan: 100 vehicles, two teams, four weeks of live operation after readiness approval. Compare against a baseline and, where feasible, a similar team not yet using the new workflow.
Illustrative pilot results:
| Measure | Baseline | Pilot | Interpretation |
|---|---|---|---|
| Median alert-to-assignment | 45 min | 18 min | Better, but short of 15-min target |
| p90 alert-to-assignment | 110 min | 75 min | Tail remains problematic |
| False-alert rate | 12% | 9% | Improvement; review sample quality |
| API availability | Not applicable | 99.95% | Above proposed 99.9% SLO |
| Freshness within 120 sec | Not applicable | 93% | Below proposed 95% SLO |
| Dispatcher satisfaction | 3.0/5 | 4.0/5 | Positive but survey bias possible |
Steering committee decision: do not declare complete success based on one favorable number. Extend the pilot for two weeks, prioritize freshness and p90 delays, and then reassess. The contract's acceptance criteria and change-control mechanism determine whether this is remediation within scope or new work.
Validation gate: documented go/no-go decision, evidence pack, residual risks, and named owner for every unresolved issue.
Phase 10 ā Operate, respond, learn
At 09:12, telemetry ingestion falls behind after a partner sends a burst of messages.
Example incident communications:
09:20 ā Initial notice: We are investigating delayed shipment updates affecting some dispatchers. Current location timestamps remain visible. Next update by 09:50.
09:42 ā Progress: Ingestion backlog is elevated. We have reduced nonessential processing and are draining the queue. No evidence of lost accepted events so far; verification continues.
10:18 ā Restoration: Processing delay has returned within the operational threshold. We will monitor recovery and publish a post-incident review within three business days.
A runbook is a tested operational procedure: how to recognize a problem, where to look, what actions are safe, and when to escalate. A postmortem (or post-incident review) examines how an incident happened and how the system can be improved. Observability is the ability to understand internal system behavior from its signals, rather than guessing from a green server-status light.
The postmortem documents impact, detection delay, timeline, contributing factors, what worked, and corrective actions. Avoid blaming individuals; focus on system improvements.
Example follow-up: add backlog-based autoscaling within tested limits, replay a production-like burst in staging, and introduce an alert on the freshness SLI rather than relying solely on CPU utilization.
Validation gate: run the new test, verify alerts fire, assign remediation owners, and confirm that the same failure mode is less likely or less damaging. Closing the incident ticket is not equivalent to closing the risk.
Phase 11 ā Handover and the ownership test
Handover transfers practical responsibility, not just files. Infrastructure as code (IaC) means environments are defined and reproducible through versioned configuration. A restore exercise proves that backups can actually recover usable data. Operational independence means the customer can deploy, diagnose, recover, and make changes without hidden reliance on the supplier.
A mature handover is not a ZIP file and a farewell meeting. Northstar should be able to deploy, restore, diagnose, and evolve the system without depending on Harbor's undocumented knowledge.
Exit rehearsal: Northstar engineers, using their own identities, provision a nonproduction environment, deploy a release, rotate a secret, diagnose a simulated outage, restore data from backup, and make a small approved change. Harbor observes but does not operate the keyboard.
Handover evidence: repository and artifact ownership; infrastructure-as-code; architecture and ADRs; API/schema contracts; access and credential inventories; runbooks; dashboards; on-call routing; dependency/license inventory; backup restoration records; known risks; costs; and a transition support plan.
Validation gate: the customer performs the exit rehearsal successfully, with agreed exceptions tracked to resolution. Ownership becomes a demonstrated capability, not a contract adjective.
A reusable gate matrix
| Gate | Owner | Required evidence | Fail action |
|---|---|---|---|
| Problem validated | Product sponsor | Baseline, interviews, outcome | Redo discovery |
| Contract agreed | Commercial/legal owners | SOW, acceptance, IP, exit terms | Renegotiate |
| Requirements approved | Product + technical owners | Testable stories and NFRs | Clarify and rescope |
| Architecture accepted | Technical + security owners | ADRs, threat model, prototype | Reduce risk or redesign |
| Change ready | Code owner | PR, tests, scans, review | Block merge |
| Release ready | Release owner | Test evidence, rollback, telemetry | Block deployment |
| Pilot accepted | Product owner | Outcome and SLO evidence | Remediate or change scope |
| Operations ready | Service owner | Runbooks, on-call, restore test | Delay handover |
| Exit validated | Customer technical owner | Independent operational rehearsal | Extend transition |
What to automateāand what not to
Automate repeatable evidence collection: builds, tests, security scans, deployment records, service indicators, change logs, and cost allocation. Use AI to reduce clerical work and broaden analysis. Keep accountable human decisions at business acceptance, security exceptions, architectural trade-offs, contractual changes, and high-impact production operations.
The deeper point is that AI makes the gap between producing code and owning outcomes more visible. If generated code accelerates delivery but review, integration, operations, and customer learning remain bottlenecks, the organization has only moved the queue.
In 2026, enterprise software maturity should be judged less by how quickly a team produces artifacts and more by whether it can prove value, explain decisions, recover from failure, control risk, and transfer operational independence.
A short glossary to keep nearby
| Term | Plain-language meaning |
|---|---|
| SLA / SLO / SLI | Customer commitment / internal reliability target / actual measurement |
| NFR | A requirement about quality or constraints, not just functionality |
| ADR | A written record explaining a technical decision and trade-offs |
| RTO / RPO | Target recovery time / maximum targeted data-loss window |
| UAT | Users checking whether the solution meets real needs |
| SOW | Contractual description of work, deliverables, and acceptance |
| CI/CD | Automated integration, verification, and release workflows |
| DORA | Research-backed software-delivery performance measures |
| Error budget | Reliability failure allowance implied by an SLO |
| Idempotency | Repeating an operation does not repeat its business effect |
| Canary | Releasing to a small audience before full rollout |
| Runbook | A repeatable guide for operating or recovering a service |
| FinOps | Collaborative management of cloud spending and value |
| DACI / RACI | Decision roles (Driver, Approver, Contributors, Informed) / work responsibility roles (Responsible, Accountable, Consulted, Informed) |
| Stream-aligned team | Team organized around a customer-value flow, rather than a single technical layer |
| Decision latency | Time between identifying a decision and obtaining an authorized decision |
Further reading and research basis
These are primary or authoritative practitioner sources. They support the frameworks and general findings above; the fictional scenario, dialogue, targets, and pilot data are original teaching examples.
- DORA ā Software Delivery Performance Metrics. Current five-metric definitions, measurement pitfalls, and application-level comparisons. https://dora.dev/guides/dora-metrics/
- Google Cloud / DORA ā 2025 State of AI-Assisted Software Development. Research on AI adoption, organizational capabilities, throughput, product performance, and stability. https://cloud.google.com/resources/content/2025-dora-ai-assisted-software-development-report
- Google Cloud ā Announcing the 2025 DORA Report. Summary of survey findings and their interpretation. https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
- NIST ā Secure Software Development Framework (SP 800-218). Secure development practices and governance. https://csrc.nist.gov/projects/SSDF
- NIST NCCoE ā DevSecOps Notional Reference Model. Lifecycle gates, supply-chain provenance, and AI-assisted DevSecOps considerations. https://pages.nist.gov/nccoe-devsecops/notational-reference-model.html
- OWASP SAMM ā Guidance for AI-Assisted Development. AI governance, security testing, tools, autonomy, and verification. https://owaspsamm.org/docs/guides/ai-assisted-development/
- Google SRE Workbook ā Error Budget Policy. Operational policy example for balancing releases and reliability. https://sre.google/workbook/error-budget-policy/
- DORA ā Loosely Coupled Teams. Team autonomy, deployability, and coordination dependencies. https://dora.dev/capabilities/loosely-coupled-teams/
- DORA ā 2024 Accelerate State of DevOps. Leadership, stable priorities, user-centricity, and platform trade-offs. https://dora.dev/research/2024/dora-report/
- Atlassian ā Team Topologies. Overview of stream-aligned, platform, enabling, and complicated-subsystem teams. https://www.atlassian.com/devops/frameworks/team-topologies
- UK Government Digital Service ā Service Level Objectives. Practical service objectives and continuous improvement. https://gds-way.digital.cabinet-office.gov.uk/standards/slo.html
What is the strongest proof of ownership you've seen in an enterprise handover: source-code access, a successful restore, or the customer's ability to ship the next feature without the vendor?













