"Never stops running" is a lie in the literal sense , i.e. all hardware fails, all software runs out of memory, every deploy changes running code with different code. Failure is expected, it's being dealt with and it doesn't slow down, or "never stops running" as it is put. The design is not a failure prevention one. Failure is a normal, survivable, boring event not an outage!
Redundancy is not the same as availability
The naive version of high availability is "run two of everything." That's necessary but not sufficient, and teams that stop there get burned by split-brain: two nodes, each convinced it's the only one alive, both accepting writes, producing two divergent versions of the truth that someone has to reconcile by hand at 4 a.m.
What is needed is a consensus layer that sets 'who is in charge right now' on a fact the whole system agrees on, as opposed to a local fact set by each of the nodes. There are raft and Paxos-family protocols made specially for this purpose, and etcd, Consul and most managed database failover systems are based on one of them. The real guarantee you need is: At any time at least one node thinks that it's the leader. Failure of a single node is better than failure of multiple nodes; if a single node fails loudly, it's recoverable, but if multiple nodes agree quietly, it is data corruption before anyone notices.
// A leader-check that actually matters: verify leadership
// immediately before a write, not just at startup.
func (n *Node) Write(ctx context.Context, key, val string) error {
if !n.raft.IsLeader() {
return ErrNotLeader
}
// Leadership can be lost between this check and the write
// completing — always route the actual write through the
// consensus log, never write directly to local state here.
return n.raft.Apply(ctx, Command{Key: key, Value: val})
}
Graceful degradation beats heroic uptime
If a system is up, but delivering errors for all its requests, that's not a system, it's just a technically running one. True resilience is the ability to agree, before the event, what the system should do if it is unable to access a dependency and making that action a conscious one, not the action that falls out of an unhandled exception.
This entails providing an answer to "what happens if this doesn't come back in time" other than "the request hangs forever". The standard tool is your circuit breaker: If a dependency fails repeatedly over a period of time, do not call it anymore and fail fast instead, to not tie up all of the threads in your process waiting for it to come back.
type CircuitBreaker struct {
mu sync.Mutex
failures int
threshold int
state string // "closed", "open", "half-open"
lastFailure time.Time
cooldown time.Duration
}
func (cb *CircuitBreaker) Call(fn func() error) error {
cb.mu.Lock()
if cb.state == "open" {
if time.Since(cb.lastFailure) < cb.cooldown {
cb.mu.Unlock()
return ErrCircuitOpen
}
cb.state = "half-open"
}
cb.mu.Unlock()
err := fn()
cb.mu.Lock()
defer cb.mu.Unlock()
if err != nil {
cb.failures++
cb.lastFailure = time.Now()
if cb.failures >= cb.threshold {
cb.state = "open"
}
return err
}
cb.failures = 0
cb.state = "closed"
return nil
}
The real point isn't the "circuit breaker pattern" itself, it's the design habit it's a part of: Figure out what the user will see if this particular dependency is down, and make that a design decision, not an incident retro.
Zero-downtime deploys are a data problem, not a process problem
The solution to process side is rolling deploys — creating new instances to replace old ones, rather than taking down the entire fleet. The more difficult challenge is deploying old and new code on the same data during rollout time, since for any deployment that takes more than a few seconds on a fleet, there is a period where old and new code live together on the same data.
That's why schema changes should be backward compatible both ways during migration: new code has to be able to handle the old schema, and old code (which will be serving some traffic during the migration) has to be able to deal with the new schema. The basic pattern is to split one migration into a number of deploys: insert the new column as nullable, deploy code that writes to both the old and new column, backfill, deploy code that reads only from the new column, and finally drop the old column in a later deploy when there is no reference to it. A single glance at the "rename column and redeploy" step is all it takes to transform a normal deploy into an incident, as during a rollout, some percentage of your fleet will be using code that depends on a column that is no longer present.
Self-healing means detecting and correcting without a human in the loop
Health checks that only respond to "is the process alive" see crashes but don't see the common failure: process is alive, it's accepting connections, it just hung on a dependency, it's leaking memory, it will reach an OOM kill, but hasn't yet. A liveness check should return something other than 200.
func (s *Server) HealthCheck(w http.ResponseWriter, r *http.Request) {
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()
if err := s.db.PingContext(ctx); err != nil {
http.Error(w, "db unreachable", http.StatusServiceUnavailable)
return
}
if s.queueDepth.Load() > s.queueDepthAlarmThreshold {
http.Error(w, "queue backed up", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
}
Combine that with an orchestrator that actually takes action on this indicator (restart or remove node from rotation) and it becomes "a node is quietly degraded" to "a node is automatically replaced" and no one is paged for a problem the system could solve itself.
The actual architecture is organizational as much as technical
If the team's incentive is to pretend the above has not happened, or to cover up any degradation that is seen, none of these works. The “never stop running” systems are, at their core, systems constructed in a way where failure is a matter of fact and not a matter of code that will be executed after failure. So the uptime is a consequence of that assumption being executed to the hilt on every layer and not an add-on after the system is finished.
