Why First-Pass Rate Does Not Measure Engineering Delivery

This article explains why a 91.7% first-pass rate does not measure engineering delivery, which definitions and records are missing, and how lifecycle events can make gate, repair and delivery states queryable.

The reported 91.7% first-pass rate describes performance at one gate. The gate definition, denominator and source are unavailable.

The same material reports zero second passes, zero repair cycles and delivery of 52.2% of started work. These figures may use different boundaries or definitions. Without the underlying records, they cannot be reconciled or treated as substantive evidence.

A first-pass result measures one gate. It does not measure delivery.

A change can pass a gate on its first recorded attempt and still await later checks, be superseded or fail to meet the definition of delivered.

Evidence limitations

The original report, dashboard or dataset has not been supplied. The figures cannot be cited or independently verified.

The available material also omits:

• Raw numerators and denominators
• Cohort dates and observation windows
• Exclusion and ageing rules
• Definitions of started work, attempt, pass, repair cycle and delivered
• The position of review findings relative to the measured gate
• Breakdowns by risk class, repository, change type or work age
• Repository counts, readiness results and aggregation rules
• A policy linking risk classes to planning, review and approval controls
• A recorded end-to-end change trace
• Cost-attribution rules and comparable cost periods

This matters when classifying unfinished work. A change still open at the end of an observation window is not necessarily a failed delivery. That judgement requires an explicit ageing rule.

The reported figures below identify the claims under examination. They are not verified results.

The material states that automated reviewers raised all recorded findings and that every finding was categorised as a correctness issue. It also lists model-routing assignments. Those assignments describe configuration, not measured performance. They provide no evidence about assurance outcomes.

Define the gate and delivery state

A first-pass measure needs a fixed boundary. Its definition must state:

• Which artefact is assessed
• What starts an attempt
• Which checks apply
• What records a pass or failure
• How corrections before the result are treated
• How reopened or resubmitted work is treated

Moving the gate from implementation to review, or from review to production readiness, changes the result without necessarily changing the engineering work.

Delivery needs the same precision. It should be represented by an observable event, not reconstructed later from incomplete workflow states.

Denominators change the result

Consider four synthetic changes. These records are illustrative and do not come from the reported data.

These records produce different measures:

first-pass rate = 3 ÷ 4 = 75%
delivered share = 2 ÷ 4 = 50%
recorded second-attempt share = 1 ÷ 4 = 25%
repair cycles per started change = 1 ÷ 4 = 0.25

The first-pass rate is valid for the specified gate. It says nothing about whether changes B and D were delivered.

This distinction is missing from the reported 91.7% and 52.2% figures. Both may be correct within their respective boundaries, but the underlying records are needed to establish that.

Make repair activity visible

The recorded review findings require explanation:

• Automated reviewers raised all recorded findings
• Every recorded finding was categorised as a correctness issue
• 63.3% of recorded findings were blocking

The material does not show when these findings occurred or what happened afterwards. Corrections may have taken place before the recorded gate, appeared as continuations rather than repair cycles, or moved outside the tracked workflow.

Zero recorded repair cycles does not establish that no correction occurred. Until repair states are explicit, the first-pass rate should be reported only as gate performance.

Unrecorded correction also affects capacity and forecasting. Two changes can share the same recorded outcome while requiring different amounts of engineering work.

Record lifecycle events

A common event record would make attempts, repairs and delivery states queryable. This is a proposed measurement design, not a description of the current data.

A minimum event set could include:

• change_started
• gate_attempted
• gate_passed
• gate_failed
• repair_started
• repair_completed
• change_superseded
• change_abandoned
• change_delivered

Corrections before gate_attempted would not count as post-attempt repair. Corrections after gate_failed would appear through repair events and a later attempt number.

The resulting measures would be:

first_pass_rate =
  changes passing the specified gate on attempt 1
  ÷ changes with recorded activity at that gate

delivered_share =
  started changes with a delivered event
  ÷ all started changes in the cohort

recorded_second_attempt_share =
  changes with attempt_number = 2 at the specified gate
  ÷ changes with recorded activity at that gate

repair_cycles_per_started_change =
  completed repair cycles
  ÷ all started changes in the cohort

A recorded pass counts as evidence that the change reached the gate. This avoids requiring a separate gate_attempted event when a gate_passed event already exists.

A full reference query belongs in an implementation appendix or companion repository. The essential gate logic is:

MAX(CASE
    WHEN gate_name = :gate_name
     AND event_type IN ('gate_attempted', 'gate_passed', 'gate_failed')
    THEN 1 ELSE 0
END) AS reached_gate,

MAX(CASE
    WHEN gate_name = :gate_name
     AND event_type = 'gate_passed'
     AND attempt_number = 1
    THEN 1 ELSE 0
END) AS passed_first_attempt,

MAX(CASE
    WHEN gate_name = :gate_name
     AND attempt_number = 2
     AND event_type IN ('gate_attempted', 'gate_passed', 'gate_failed')
    THEN 1 ELSE 0
END) AS recorded_second_attempt

This detects a recorded second attempt. It does not show that the attempt was required unless the query also identifies a preceding failure.

Separate results by risk

The material reports that high-risk work accounts for 21.7% of the mix and critical financial work accounts for 13%. These figures remain unverified without the underlying source.

Such work warrants stronger planning, challenge and review than low-risk implementation. A throughput comparison that combines risk bands may favour easier work rather than safe delivery of valuable work.

Approved plans reportedly cover 34.8% of changes. That result cannot be judged without a policy stating which risk classes require approval. The relevant comparison is coverage against the applicable policy, not coverage in isolation.

Report readiness by control

The material reports an overall repository-readiness pass rate of 92.6%. It also reports complete pass rates for:

• Runtime pinning
• Lockfiles
• Observed pipeline checks
• One-command checks
• Testing guidance
• Absence of committed secrets

Agent tools and default-branch protection each reportedly pass at 33.3%.

The aggregate cannot be interpreted without repository counts, repository-level results and the weighting method. Reporting should show each control by repository and state whether the total weights repositories, checks or observations equally.

Tie cost to delivery

Cost per delivered change keeps the denominator tied to an accepted outcome.

Attributable cost should include:

• Agent use
• Human planning and review
• Delivery-pipeline activity
• Verification
• Repair effort

cost per delivered change =
  attributable cohort cost
  ÷ delivered changes in the cohort

Started work that does not reach delivery should remain visible unless an explicit attribution rule excludes it.

No directional cost claim can be made from the supplied material. Comparison requires consistent cost attribution, delivery definitions, observation windows and risk bands.

Publication criteria

A later report should not present first-pass performance as delivery efficiency until:

• The underlying source can be cited and verified
• Numerators, denominators, cohort dates and exclusions are published
• Gate, delivery, observation and ageing rules are explicit
• Lifecycle events expose attempts, repairs and final disposition
• Risk policy defines required planning and review
• Readiness aggregation and repository-level results are available
• Cost periods and attribution rules are comparable
• At least one anonymised change has a complete recorded trace

No owners or delivery dates are supplied, so none can be assigned.

What this means

The current figures describe reported gate performance, delivery share and control indicators. They do not form a reconciled delivery measure. The full change lifecycle must be recorded and verified before delivery, cost and risk can be compared on the same basis.

Originally published at AFT Group Engineering Insights.