There is a particular kind of meeting every engineer eventually attends.
A metric is projected onto a screen. It has moved in the wrong direction. Someone asks why. Another person suggests we should make sure it never moves in that direction again.
And, somewhere in the organization, a perfectly sensible system is about to be made slightly worse.
Manufacturing has units per hour. Customer support has tickets closed. Software teams have deployment frequency, utilization, velocity and, for reasons historians will struggle to explain, story points.
The attraction is obvious. Movement looks productive.
A factory line that keeps moving feels efficient. A support queue that keeps shrinking looks healthy. A deployment pipeline that keeps shipping looks fast.
Toyota built one of the world's most influential production systems partly by accepting something rather uncomfortable:
sometimes the correct thing for a production line to do is stop.
Decades later, Amazon borrowed that idea.
Except Amazon didn't have a car assembly line to stop.
It had a website.
And that small jump — from stopping defective cars to stopping defective products from being sold online — is far more interesting to me than either story on its own.
Because this isn't really an article about Toyota.
It isn't really about Amazon either.
It's about recognizing when another industry has already solved your engineering problem, only with completely different nouns.
Toyota's strange idea: give people permission to interrupt production
One of the foundations of the Toyota Production System is jidoka, often translated as “automation with a human touch.”
The idea predates Toyota's automobile factories.
Toyota traces it back to Sakichi Toyoda and his automatic looms. One of the important innovations was deceptively simple: when something abnormal happened, such as a thread breaking, the machine could stop rather than continuing to produce defective fabric.
That distinction matters.
A machine can have magnificent uptime while producing absolute rubbish.
If you've worked with distributed systems, you have probably met its descendants.
Toyota carried this principle into automobile manufacturing. Its historical documentation records the adoption of an Andon visual-control system in 1950 and stop-button-linked Andons on the Crown assembly line at the Honsha Plant in 1955.
The Andon made abnormalities visible. Workers could signal for assistance when something wasn't right, and if the issue could not be resolved within the work cycle, production could be stopped rather than allowing the defect to continue downstream.
Toyota's own description of the Toyota Production System still emphasizes this principle: equipment stops when abnormalities are detected, and operators can stop production when necessary so defective products are not simply passed to the next process.
That's quite a bit more radical than it sounds today.
Imagine explaining the idea purely through a utilization dashboard.
“We installed a system that allows employees to interrupt our extremely expensive production machinery.”
Wonderful.
Did Finance approve this?
The internet version of Andon misses the clever bit
The story is often compressed into:
Every Toyota worker can pull a cord and stop the entire factory.
It's memorable. It also makes Toyota sound slightly more theatrical than it actually is.
Toyota's implementation evolved over time, and one particularly interesting design is the fixed-position line-stop system.
When a worker detects an abnormality, the first action isn't necessarily an immediate emergency stop. The Andon signals the problem and alerts the team leader while the vehicle continues toward a designated position.
That creates a small intervention window.
If the team leader reaches the worker and the problem can be resolved within the cycle, production continues. If it can't, the affected line stops at the fixed position.
So the mechanism isn't simply:
problem → STOP EVERYTHING
It's closer to:
abnormality detected
↓
make it visible
↓
bring help to the problem
↓
can we resolve it safely within the cycle?
/ \
yes no
↓ ↓
continue stop
↓
resolve cause
↓
resume
I like this much more than the viral version because it's recognizably good systems engineering.
There is detection, escalation, a bounded recovery opportunity and a controlled stop.
More importantly, the abnormality isn't allowed to quietly travel downstream just because stopping would make today's production chart look unpleasant.
A factory can be extremely efficient at producing defects
Consider two imaginary factories.
Factory A runs for an entire shift without interruption. Management sees beautiful utilization numbers.
Unfortunately, several defects occur early in the process and aren't caught until much later. Components have already moved through additional stations. Labor has been added. Other parts have been attached. Some products require rework.
Factory B stops several times during the same shift. Its utilization graph is uglier, but abnormalities are caught close to where they originate.
Which one is more productive?
You can't answer that until you decide where the system boundary is.
If you're measuring minutes the assembly line moved, Factory A looks fantastic.
If you're measuring good products delivered with minimum total waste, the answer may reverse completely.
Software teams make this mistake all the time because local metrics are wonderfully easy to put on dashboards.
A developer closes more tickets. Good.
Deployment frequency increases. Good.
CPU utilization improves. Good.
Except perhaps the tickets are being reopened, the deployments are generating incidents and the service now has no headroom during traffic spikes.
None of those metrics are useless. The problem starts when the metric becomes the objective instead of evidence about the objective.
And defects have an especially nasty property: they become more expensive as they travel.
A bug noticed while writing the code might cost minutes.
The same bug found by a unit test costs a little more.
Found during integration testing, more again.
Found after deployment, now we may have logs, rollback decisions, customer impact, corrupted state, support tickets and several engineers trying to remember who changed what on Tuesday.
By the time a customer discovers it, the bug has acquired colleagues.
Toyota's production philosophy attacks that propagation.
Which brings us to Amazon.
Amazon looked at a car factory and saw a website
In December 2006, Marc Onetto joined Amazon as Senior Vice President of Worldwide Operations. Amazon's SEC filings from the period confirm the role, and Onetto brought extensive manufacturing and operations experience from companies including GE and Solectron.
Soon after arriving, he discussed applying Toyota's Andon concept at Amazon.
He later described the story in an interview with McKinsey.
Jeff Bezos supported the idea.
Amazon implemented it in roughly six months.
But there was an obvious problem.
There was no assembly-line worker standing beside the Amazon product page.
There was, however, someone unusually well positioned to detect defects:
the customer.
And there was another group receiving those defect reports all day:
customer service.
Suppose customers repeatedly contact Amazon because a particular product arrives with the wrong accessory.
A conventional support system is very good at treating these as individual transactions.
Refund the customer. Replace the item. Apologize. Close the ticket.
Then sell another one.
The next customer reports exactly the same defect.
Refund. Replace.
Another customer buys it.
Same problem.
At some point you no longer have a customer-service problem. You have built a defect subscription service.
Amazon changed the feedback loop.
According to Onetto, when customer-service agents identified a repetitive defect, they were empowered to “stop the line” by removing the product from the website until the underlying problem could be investigated and corrected.
Amazon also built a background process around identifying, tracking and resolving those defects.
Onetto reported that the approach eliminated tens of thousands of defects per year.
That claim is worth dwelling on because this story sometimes gets repeated online as management folklore. In this case, we have the Amazon executive responsible for the implementation describing what happened, contemporary corporate records establishing his role, and lean-management organizations documenting the practice.
Amazon really did build a digital Andon.
What exactly did Amazon copy?
This is the part I find most useful as an engineer.
Amazon obviously didn't copy Toyota literally.
There were no ropes hanging beside customer-service desks. Nobody in Seattle needed to understand how to assemble a Corolla.
Instead, somebody stripped away the domain-specific implementation and noticed the underlying structure.
Toyota had something like this:
worker observes abnormality
↓
abnormality becomes visible
↓
team responds
↓
unresolved defect stops flow
↓
cause is corrected
↓
flow resumes
Amazon could map its own system onto the same structure:
customer experiences abnormality
↓
customer service sees the signal
↓
repeated defect is identified
↓
product availability is stopped
↓
cause is investigated/corrected
↓
selling resumes
The interesting object isn't the Andon cord.
It's the feedback architecture.
Detection happens near the place where evidence first appears. The signal is made visible. Someone has authority to act on it. Flow can be interrupted before more defective output is created. Then the system has a mechanism for recovery.
Once you describe it that way, the automobile disappears from the idea completely.
Toyota had solved a defect-propagation problem.
Amazon also had a defect-propagation problem.
The physical implementation was almost irrelevant.
This is one of my favourite forms of innovation because it requires less invention and more abstraction.
You look at something that works elsewhere and ask:
Which parts of this solution are accidental details of that industry, and which parts are fundamental?
That's a very engineering way of stealing ideas.
We've already built Andon cords into software
Once I started looking at Andon this way, it became difficult not to see it everywhere in software architecture.
A CI pipeline failing because a test breaks is an Andon-like mechanism.
So is a circuit breaker.
A canary deployment that detects an increase in error rate and automatically halts rollout is probably an even cleaner example.
Suppose we're deploying version v2 across 200 instances.
Instead of replacing all 200:
v1 → v2 everywhere
we deliberately expose a small portion of traffic first:
5% traffic → v2
95% traffic → v1
observe:
error rate
p95/p99 latency
saturation
business failures
If the error budget starts burning unusually fast, rollout stops.
From a narrow deployment-throughput perspective, this is slower. We've added monitoring periods, thresholds and rollback logic to something Kubernetes could otherwise execute quite enthusiastically.
From a system perspective, we're paying a small interruption cost to limit the blast radius of a defect.
The same reasoning appears in message processing.
If an event fails schema validation, you can reject or quarantine it. Or you can be “resilient,” accept it anyway and discover six services later that one malformed event has been enthusiastically replicated across Kafka topics, databases, caches and analytics pipelines.
The second architecture has excellent throughput right up until everyone opens Slack.
Circuit breakers make the same trade.
When a dependency is failing, continuing to send requests may feel like maintaining availability. In practice, you're often amplifying the failure: consuming threads, sockets, connection pools and retry budgets while increasing load on a service already struggling to recover.
Opening the circuit deliberately refuses work for a period.
Locally, that looks like failure.
Globally, it may be the thing preventing one unhealthy service from dragging five healthy services down with it.
These mechanisms don't exist because engineers dislike throughput.
They exist because uncontrolled propagation is usually more expensive than controlled interruption.
The uncomfortable part isn't stopping. It's authority.
This is where Andon becomes less of a technical pattern and more of an organizational one.
Installing an alert is easy.
Giving the person who sees the alert authority to interrupt something important is considerably harder.
Toyota's worker isn't merely reporting an abnormality into a system where somebody might eventually create an action item.
The signal demands a response.
Amazon's customer-service agents weren't limited to apologizing more elegantly. The system gave frontline employees a path that could result in a product being removed from sale.
That has commercial consequences.
Which is why copying the visible mechanics of systems like Andon usually isn't enough.
An organization can build a beautiful escalation workflow:
Problem detected
↓
Ticket created
↓
Assigned to team
↓
Prioritized next sprint
↓
Moved to backlog
↓
Renamed "Known Limitation"
JIRA archaeology begins surprisingly quickly.
Marc Onetto later discussed another revealing detail: unresolved Andon issues could eventually reach senior leadership. In an interview published by Planet Lean, he described personally reviewing issues that had remained unresolved for two weeks.
That's the other half of the design.
Escalation has to lead somewhere.
Otherwise, you've built an expensive notification system for problems everybody already knows about.
The pattern I would steal
If I were borrowing the Toyota/Amazon idea for a software platform, I wouldn't create something called an “Andon Service” and start adding Toyota terminology to architecture diagrams.
That's how useful ideas end up becoming internal frameworks with 43 configuration options.
I'd steal the constraints.
For a critical flow, ask:
1. Where is the earliest reliable signal that something is abnormal?
Don't wait for the final customer-facing failure if the system can identify the condition three stages earlier.
2. Can that signal automatically prevent propagation?
Validation, circuit breaking, deployment gates, quarantines and feature flags are all useful here.
3. Is the interruption scoped appropriately?
Toyota's fixed-position approach is instructive. Don't shut down the planet because one request failed. Stop the smallest safe unit of flow.
In distributed systems that could mean a tenant, partition, feature, deployment cohort, consumer group or dependency route.
4. What is the recovery path?
A stop without recovery is just an outage.
Who investigates? What proves the condition is safe again? Is recovery automatic or manual? What state needs reconciliation?
5. Does the organization actually permit the stop?
This one isn't technical.
You can have flawless automated rollback logic, but if every failed deployment threshold gets manually overridden because “the release was promised for Thursday,” your Andon cord is decorative.
That last failure mode is more common than I'd like.
There is also a danger in romanticizing this
“Stop the line” sounds so sensible that it can become another slogan.
Stopping has a cost.
False positives have a cost.
An overly sensitive circuit breaker can create an outage where none existed. Aggressive fraud controls can block legitimate customers. A noisy deployment gate can train engineers to override it. A marketplace that removes products based on weak signals can hurt sellers and customers.
Toyota's system itself is evidence that the answer isn't simply “stop more often.”
The fixed-position line-stop design introduced a bounded opportunity to resolve the issue before stopping became necessary.
That's the sophisticated part.
The design problem is deciding:
When is the confidence and expected downstream cost
high enough that interruption is cheaper than continuation?
That's a threshold problem.
And thresholds need evidence.
In software, I would want hysteresis, minimum sample sizes, error-budget context and carefully chosen recovery criteria rather than:
if errors > 0:
shut_down_company()
Tempting though that implementation occasionally feels.
The Toyota lesson isn't that stopping is good.
It's that continuing is not automatically good.
Those are very different ideas.
The abstraction is the innovation
What stays with me about the Amazon story isn't really that Amazon learned something from Toyota.
Companies benchmark one another constantly.
The clever move was recognizing what not to copy.
Toyota had factories, assembly workers, Andon boards, production cycles and physical vehicles.
Amazon had warehouses, customer-service agents, web pages, suppliers and orders.
At the surface, they're different systems.
Underneath, both had the same uncomfortable question:
What should happen when we know the output is defective but the system is perfectly capable of continuing anyway?
Toyota's answer was to make abnormality visible and allow production to stop before the defect travelled further.
Amazon translated that principle into e-commerce and gave customer service a way to interrupt the source of repeated defects rather than endlessly cleaning up the consequences.
Software architecture has been rediscovering versions of the same idea ever since.
Circuit breakers.
Canary rollouts.
Automated rollback.
Dead-letter queues.
Schema gates.
Kill switches.
Feature flags.
Different industries. Different vocabulary. Same architecture.
And that's probably the part worth carrying into our own engineering work.
When I'm looking at a difficult systems problem, the useful question may not always be:
“Which new technology solves this?”
Sometimes it's:
“Who has already had this problem in a completely different industry?”
The answer might be buried in manufacturing, aviation, logistics, medicine or finance rather than another engineering blog.
So perhaps before adding another framework to package.json, we should occasionally ask a stranger question:
Have you tried looking at a factory?
