Circuit Breaker Pattern in Go: Stop Cascading Failures

A circuit breaker stops your Go service from hammering a failing dependency,
preventing cascading failures that consume goroutines, sockets, and memory until the entire system collapses.

The hard part is not the state machine. It is deciding where the breaker belongs, what counts as failure, how it interacts with timeouts and retries, and what your service should do when the circuit is open.

In Go, the circuit breaker pattern is especially useful around outbound calls: HTTP APIs, payment gateways, search services, email providers, LLM gateways, internal microservices, and other dependencies that can become slow, overloaded, or partially unavailable. Used well, a circuit breaker reduces cascading failures. Used badly, it becomes another obscure failure mode.

What Problem Does a Circuit Breaker Solve?

Distributed systems rarely fail cleanly.

A dependency might not be fully down. It might be:

• returning 500 errors
• returning 429 rate limit responses
• accepting TCP connections but never replying
• responding in 30 seconds instead of 300 milliseconds
• failing only for some requests
• overloaded because every client is retrying at once

The worst case is often not a hard failure. It is a slow dependency.

Slow calls consume goroutines, sockets, database connections, memory, and worker capacity. If your service keeps waiting on a dependency that is already unhealthy, your service can become unhealthy too.

A circuit breaker prevents that by failing fast after the dependency crosses a failure threshold.

Instead of doing this forever:

request -> call dependency -> wait -> timeout -> retry -> wait -> fail

the service eventually does this:

request -> circuit open -> return fallback or error immediately

That fast failure is not always pleasant, but it is predictable. Predictable failure is easier to operate than a slow collapse.

The Three Circuit Breaker States

Most circuit breakers use three states.

Closed

The circuit is closed during normal operation.

Requests are allowed through. The breaker records successes and failures. If the number or ratio of failures crosses a threshold, the breaker opens.

Closed does not mean "safe forever." It means "traffic is currently allowed."

Open

The circuit is open when the dependency is considered unhealthy.

Requests are rejected immediately. The service should return a fallback, cached response, degraded response, or a clear upstream error.

Open does not fix the dependency. It gives the dependency time to recover and protects the caller from wasting resources.

Half-Open

After a cool-down period, the breaker enters a half-open state.

Only a limited number of trial requests are allowed through. If they succeed, the breaker closes. If they fail, the breaker opens again.

Half-open is important because it avoids two bad extremes:

• never trying the dependency again
• sending full traffic back too quickly

The state transitions look like this:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Failure threshold reached
    Open --> HalfOpen: Timeout elapsed
    HalfOpen --> Closed: Trial succeeds
    HalfOpen --> Open: Trial fails

Circuit Breaker vs Timeout vs Retry

A common mistake is treating circuit breakers, retries, and timeouts as interchangeable. They are related, but they solve different problems.

Timeout

A timeout limits how long one operation can run.

In Go, this usually means passing a context.Context with a deadline or timeout into the outbound call.

A timeout answers this question:

How long am I willing to wait for this one call?

Retry

A retry repeats an operation when the failure might be temporary.

Retries are useful for short network glitches, temporary 503 responses, connection resets, and other transient failures.

A retry answers this question:

Should I try this call again?

Circuit Breaker

A circuit breaker stops calls when the dependency is probably unhealthy.

It answers this question:

Should I call this dependency at all right now?

Rate Limiter

A rate limiter controls how much traffic is allowed over time.

It answers this question:

How much traffic should this caller send?

Bulkhead

A bulkhead isolates resources so one dependency cannot consume everything.

It answers this question:

How much of my service can this dependency damage?

These patterns are strongest when used together. A circuit breaker without timeouts is weak. Retries without jitter can create retry storms. A fallback without metrics can hide an outage.

When to Use a Circuit Breaker in Go

Use a circuit breaker when your service calls a dependency that can fail independently from your service.

Good candidates include:

• external HTTP APIs
• payment processors
• email and SMS providers
• search services
• recommendation services
• LLM inference gateways
• internal microservice endpoints
• third-party SaaS APIs
• slow or overloaded read-side services

Circuit breakers are especially useful when the caller can degrade gracefully.

For example:

• return cached product data
• skip a recommendation block
• mark a payment provider as temporarily unavailable
• queue work for later
• return a partial response
• fail fast with a clear temporary error

The important question is not "can this call fail?" Everything can fail. The better question is:

If this dependency is failing, should we continue sending full traffic to it?

If the answer is no, a circuit breaker may help.

When Not to Use a Circuit Breaker

Do not add a circuit breaker to every function just because the pattern sounds responsible.

A circuit breaker is usually not useful for:

• local in-process function calls
• simple CRUD inside a monolith
• validation logic
• deterministic business rules
• CPU-only local operations
• code paths where no useful fallback exists
• write operations that are not idempotent
• dependencies already protected by a stronger workflow layer

A circuit breaker also does not replace basic hygiene:

• set timeouts
• propagate context
• use connection pools correctly
• handle errors explicitly
• make retries safe
• observe failure rates

A bad circuit breaker can make a system harder to reason about. It can hide the real problem, reject traffic too aggressively, or create confusing behavior during recovery.

The slightly opinionated rule is simple:

Add circuit breakers at dependency boundaries, not everywhere.

Choosing a Go Circuit Breaker Library

You can implement a basic circuit breaker yourself, but most production Go services should use a library.

The most common simple choice is sony/gobreaker.

It gives you:

• closed, open, and half-open states
• configurable failure thresholds
• configurable open-state timeout
• state change callbacks
• request counters
• generic support in v2
• a small API surface

For larger resilience pipelines, you may also look at libraries that compose multiple policies, such as retry, timeout, fallback, rate limiting, bulkhead isolation, and circuit breaking. That can be useful when you want a single resilience layer around an operation.

For many Go services, though, gobreaker is enough.

Go Circuit Breaker Packages Compared

Go does not include a built-in circuit breaker in the standard library. In practice, you usually choose between a small circuit breaker library, a larger resilience framework, or an older Hystrix-style package.

For most new Go services, the decision is simple:

• use sony/gobreaker if you want a small, focused circuit breaker
• use failsafe-go if you want circuit breakers composed with retries, timeouts, fallbacks, bulkheads, rate limits, and other resilience policies
• avoid starting new projects on hystrix-go unless you already have legacy code using it

My default recommendation is boring on purpose: start with sony/gobreaker/v2 when you only need a circuit breaker. Reach for failsafe-go when you want to express a complete resilience policy in one place.

That split keeps the architecture clean. A small service client does not need a full resilience framework just to stop calling a failing dependency. But a gateway, aggregator, API client SDK, or high-traffic integration layer may benefit from composed policies.

Installing gobreaker

Use the v2 package for new code:

go get github.com/sony/gobreaker/v2

Then import it:

import "github.com/sony/gobreaker/v2"

A Basic Circuit Breaker in Go

Here is a small example around an HTTP call.

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "time"

    "github.com/sony/gobreaker/v2"
)

var ErrTemporaryUnavailable = errors.New("dependency temporarily unavailable")

type UserClient struct {
    baseURL string
    http    *http.Client
    cb      *gobreaker.CircuitBreaker[[]byte]
}

func NewUserClient(baseURL string) *UserClient {
    settings := gobreaker.Settings{
        Name:        "user-service",
        MaxRequests: 3,
        Interval:    30 * time.Second,
        Timeout:     10 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.ConsecutiveFailures >= 5
        },
        OnStateChange: func(name string, from gobreaker.State, to gobreaker.State) {
            fmt.Printf("circuit breaker %s changed from %s to %s\n", name, from, to)
        },
    }

    return &UserClient{
        baseURL: baseURL,
        http: &http.Client{
            Timeout: 3 * time.Second,
        },
        cb: gobreaker.NewCircuitBreaker[[]byte](settings),
    }
}

func (c *UserClient) GetUser(ctx context.Context, userID string) ([]byte, error) {
    result, err := c.cb.Execute(func() ([]byte, error) {
        req, err := http.NewRequestWithContext(
            ctx,
            http.MethodGet,
            c.baseURL+"/users/"+userID,
            nil,
        )
        if err != nil {
            return nil, err
        }

        resp, err := c.http.Do(req)
        if err != nil {
            return nil, err
        }
        defer resp.Body.Close()

        if resp.StatusCode >= 500 {
            return nil, fmt.Errorf("user service returned %d", resp.StatusCode)
        }

        if resp.StatusCode == http.StatusNotFound {
            return nil, fmt.Errorf("user not found")
        }

        if resp.StatusCode >= 400 {
            return nil, fmt.Errorf("user service client error: %d", resp.StatusCode)
        }

        return io.ReadAll(resp.Body)
    })

    if errors.Is(err, gobreaker.ErrOpenState) {
        return nil, ErrTemporaryUnavailable
    }

    if errors.Is(err, gobreaker.ErrTooManyRequests) {
        return nil, ErrTemporaryUnavailable
    }

    return result, err
}

This is not a complete production client, but it shows the shape:

• the breaker wraps the outbound call
• the HTTP request receives a context
• the HTTP client has a timeout
• server-side failures count as breaker failures
• open circuit errors are mapped into an application error

Configuring gobreaker Settings

The key settings are worth understanding.

Name

Name identifies the breaker.

Use a stable, specific name:

payment-api
search-service
llm-gateway
user-service

Avoid vague names like:

http-client
external-call
default

You will want this name in logs and metrics.

MaxRequests

MaxRequests controls how many requests are allowed while the breaker is half-open.

A small number is usually safer. The purpose of half-open is to test recovery, not to send full traffic immediately.

Interval

Interval controls when internal counts are cleared while the breaker is closed.

If it is zero, counts are not cleared automatically. A non-zero interval gives the breaker a rolling-ish memory window, although it is not the same as a full sliding window implementation.

Timeout

Timeout controls how long the breaker stays open before moving to half-open.

If the timeout is too short, your service will keep probing a dependency that has not recovered. If it is too long, recovery will be delayed.

Start with something conservative, such as 10 to 30 seconds, then tune from production metrics.

ReadyToTrip

ReadyToTrip decides when the breaker should open.

A simple rule is consecutive failures:

ReadyToTrip: func(counts gobreaker.Counts) bool {
    return counts.ConsecutiveFailures >= 5
}

That is easy to reason about, but it may not be right for high-volume services.

Another option is failure ratio after a minimum number of requests:

ReadyToTrip: func(counts gobreaker.Counts) bool {
    total := counts.Requests
    failures := counts.TotalFailures

    if total < 20 {
        return false
    }

    return float64(failures)/float64(total) >= 0.5
}

This avoids opening the circuit after a tiny sample size.

OnStateChange

OnStateChange is where you should emit logs or metrics.

At minimum, record:

• breaker name
• old state
• new state
• timestamp

For production systems, expose breaker state as a metric. Logs are useful for debugging, but metrics are better for alerting and dashboards.

IsSuccessful

IsSuccessful lets you decide which errors count as failures.

This is important.

Not every error should open the breaker. For example, a 404 Not Found from a user service may be a valid business result. A 400 Bad Request might be the caller's fault, not the dependency's fault.

A 503 Service Unavailable, timeout, connection reset, or 429 Too Many Requests may be a real dependency health signal.

Be careful here. Counting the wrong errors is one of the easiest ways to build a noisy circuit breaker.

What Should Count as Failure?

This is where engineering judgement matters.

Usually count these as failures:

• network timeouts
• connection refused
• connection reset
• HTTP 500
• HTTP 502
• HTTP 503
• HTTP 504
• repeated 429 responses
• malformed responses from the dependency
• context deadline exceeded during the outbound call

Usually do not count these as dependency failures:

• validation errors
• local serialization errors
• expected 404 responses
• caller-side authorization failures
• business rule rejections
• user input errors

The breaker should represent dependency health, not general application failure.

Circuit Breakers and context.Context

In Go, circuit breakers should not replace context.Context.

A circuit breaker decides whether to attempt a call. A context controls how long that call may run and whether it should stop when the caller is gone.

A good outbound call should usually have both:

ctx, cancel := context.WithTimeout(parentCtx, 2*time.Second)
defer cancel()

data, err := client.GetUser(ctx, userID)

The context should flow through the call chain:

incoming request context
-> service method
-> client method
-> HTTP request
-> dependency

Avoid creating detached background contexts inside request-scoped code. If the user request is canceled, the downstream work should usually stop too.

The calm rule is:

The breaker protects the system. The context protects the request.

You normally need both.

Circuit Breakers and Retries

Retries and circuit breakers can work well together, but the order matters.

The safest default is:

timeout per attempt
retry with backoff and jitter
circuit breaker around the dependency call

But there is no universal answer. Think about what you want to count.

If each retry attempt passes through the breaker, one user request can contribute multiple failures. That may open the breaker faster, which can be good or bad.

If the breaker wraps the whole retry operation, the breaker sees one final success or failure per user request. That is calmer, but it may hide the number of failed attempts.

For many application services, this shape is reasonable:

user request
-> circuit breaker
   -> retry policy
      -> one HTTP attempt with timeout

That means the breaker tracks whether the dependency operation ultimately worked for the caller.

For lower-level clients, this shape can also make sense:

user request
-> retry policy
   -> circuit breaker
      -> one HTTP attempt with timeout

That means the breaker protects each attempt.

The more important rule is this:

Do not retry blindly.

Use:

• a small maximum retry count
• exponential backoff
• jitter
• per-attempt timeouts
• an overall request deadline
• idempotency for writes
• metrics for retry attempts

Without those, retries can turn a small outage into a larger one. For a deeper treatment of retry safety, see Idempotency in Distributed Systems That Actually Works.

Circuit Breakers and Idempotency

Circuit breakers often appear next to retries, and retries raise the question of idempotency.

For read operations, retrying is usually safe.

For write operations, retrying can be dangerous.

Consider this payment call:

POST /charge

If the request times out, did the payment fail? Maybe. Did it succeed but the response was lost? Also maybe.

If you retry without an idempotency key, you might charge twice.

For write operations, use one or more of these:

• idempotency keys
• request IDs
• operation IDs
• unique constraints
• transactional outbox
• workflow orchestration
• explicit reconciliation

A circuit breaker can stop you from continuing to call a failing payment provider, but it cannot make unsafe retries safe.

Circuit Breakers and Fallbacks

When the circuit is open, your service needs a plan.

Possible fallback strategies include:

• return cached data
• return stale data with a warning
• omit a non-critical section
• queue work for later
• switch to another provider
• return a temporary error
• show degraded functionality
• fail the request quickly

A fallback should be honest.

For example, this is usually good:

{
  "status": "temporary_unavailable",
  "message": "Recommendations are temporarily unavailable"
}

This is risky:

{
  "recommendations": []
}

An empty list may look like a valid result. It can hide an outage, confuse users, and make debugging harder.

Silent fallbacks are tempting. They are also dangerous.

Circuit Breakers and Observability

A circuit breaker without observability is mostly a surprise generator.

Track at least these metrics:

• current breaker state
• state changes
• calls allowed
• calls rejected
• successes
• failures
• timeouts
• fallback responses
• retry attempts
• downstream latency
• downstream status codes

Useful labels include:

• breaker name
• dependency name
• operation name
• status class
• error category

Avoid high-cardinality labels such as user ID, full URL, request ID, or raw error messages.

You should be able to answer these questions from dashboards:

• Which circuit breakers are open right now?
• How often do they open?
• Which dependency caused the opening?
• Are users seeing fallback responses?
• Did latency improve after the breaker opened?
• Did retry volume spike before the breaker opened?
• Did the dependency recover?

If you cannot observe the breaker, you cannot tune it. For structured logging that pairs well with metrics, see Structured Logging in Go with slog.

A More Production-Friendly HTTP Client Shape

For real services, avoid scattering circuit breaker logic across handlers.

Create a small client package around the dependency.

Example structure:

internal/
  userservice/
    client.go
    errors.go
    metrics.go

The handler should not know the details of gobreaker. It should depend on a domain-level client method:

type UserService interface {
    GetUser(ctx context.Context, userID string) (*User, error)
}

Then the implementation can contain:

• HTTP request creation
• context propagation
• breaker execution
• status code handling
• response decoding
• metrics
• error mapping

This keeps the resilience policy close to the dependency boundary. For more on error classification at boundaries, see Go Error Handling Architecture: Boundaries and Patterns.

Where Circuit Breakers Fit in Application Architecture

The circuit breaker pattern belongs at integration boundaries.

In a Go application, that usually means:

graph LR
    A[Handler] --> B[Application Service]
    B --> C[Dependency Client]
    C --> D[Circuit Breaker]
    D --> E[HTTP / RPC / DB / Queue]

Keep the breaker out of business logic when possible.

The business layer should understand domain errors like:

payment provider unavailable
recommendations unavailable
profile service timeout

It should not need to understand gobreaker states.

This separation keeps the architecture clean:

• transport concerns stay in clients
• resilience policy stays near dependencies
• domain logic stays readable
• handlers stay thin
• tests are easier to write

Common Mistakes

Mistake 1: No Timeout

A circuit breaker does not magically stop slow calls unless the calls return.

If the outbound operation can hang forever, the breaker may not see a failure quickly enough.

Always use timeouts.

Mistake 2: One Global Breaker for Everything

Do not use one breaker for all dependencies.

A failing email provider should not open the circuit for your payment provider. A slow search endpoint should not block user profile calls.

Use separate breakers for separate dependency operations when their failure modes differ.

Mistake 3: Counting Caller Errors as Dependency Failures

If your service sends bad input and receives 400 Bad Request, that is usually not a downstream outage.

Do not train the breaker on your own bugs.

Mistake 4: Retrying Non-Idempotent Writes

Retries are not free. They can duplicate writes, payments, messages, or side effects.

Make writes idempotent before retrying them.

Mistake 5: Hiding Outages Behind Fallbacks

Fallbacks should degrade gracefully, not falsify reality.

If a dependency is down, your metrics and logs should make that obvious.

Mistake 6: Tuning Without Production Data

Thresholds copied from examples are only starting points.

Tune based on:

• request volume
• normal error rate
• dependency latency
• user impact
• recovery time
• fallback quality

Mistake 7: Using Circuit Breakers Instead of Capacity Management

A circuit breaker is not a substitute for:

• load shedding
• rate limiting
• queue limits
• autoscaling
• database tuning
• connection pool limits
• upstream quotas

It is one part of a resilience strategy.

Practical Defaults

For a typical Go service calling an internal HTTP dependency, a reasonable starting point might be:

HTTP client timeout: 2 to 5 seconds
per-request context timeout: based on caller SLA
breaker failure rule: 5 consecutive failures or 50 percent failure after 20 requests
open timeout: 10 to 30 seconds
half-open requests: 1 to 5
retry count: 1 to 3 attempts
retry backoff: exponential with jitter

These are not universal values. They are safe-ish starting points.

For user-facing APIs, keep total latency budgets tight. For background jobs, you may tolerate longer waits. For payment providers, be much more careful with retries and idempotency.

Circuit Breaker Checklist

Before adding a circuit breaker, answer these questions:

• What dependency is being protected?
• What operation is being protected?
• What errors count as dependency failure?
• What errors should be ignored by the breaker?
• What timeout applies to each call?
• Are retries allowed?
• Are writes idempotent?
• What happens when the circuit is open?
• Is there a fallback?
• Is the fallback visible in metrics?
• Who gets alerted if the circuit keeps opening?
• How will the breaker be tuned after deployment?

If you cannot answer these, adding a breaker may create more confusion than resilience.

Testing Circuit Breakers in Go

Test behavior, not the internal state machine of the library.

Useful tests include:

• dependency succeeds and response is returned
• dependency fails repeatedly and circuit opens
• open circuit returns a temporary error
• client-side validation errors do not trip the breaker
• context timeout is respected
• fallback response is returned when expected
• metrics are emitted on state changes

Use fake HTTP servers for integration-style tests:

server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    http.Error(w, "unavailable", http.StatusServiceUnavailable)
}))
defer server.Close()

For unit tests, hide the dependency behind an interface and inject a fake implementation.

Keep tests deterministic. Avoid sleeping for long real durations. Configure short breaker timeouts in tests. For more on testing concurrent Go code with fake time and isolated bubbles, see Testing Concurrent Go Code with testing/synctest.

Should You Build Your Own Circuit Breaker?

Building a small circuit breaker is a good learning exercise. It helps you understand the state machine.

For production code, prefer a maintained library unless your needs are very specific.

A production breaker needs to handle:

• concurrency
• state transitions
• counters
• half-open probes
• callbacks
• custom failure classification
• race-free behavior
• predictable error handling

That is not impossible, but it is easy to get subtly wrong.

The boring library is usually the better choice.

Conclusion

The circuit breaker pattern is not magic reliability dust.

In Go, it works best when it is part of a small, explicit resilience stack:

context timeout
+ retry with backoff and jitter
+ circuit breaker
+ fallback
+ metrics

The pattern is most useful at dependency boundaries, especially around remote services that can become slow or partially unavailable.

Use it to stop cascading failures. Use it to fail fast when a dependency is clearly unhealthy. Use it to give overloaded systems room to recover.

But do not use it as an excuse to ignore timeouts, idempotency, observability, or clean architecture.

A good circuit breaker makes failure clearer and cheaper. A bad one just makes failure more mysterious.

References

• Go Microservices for AI/ML Orchestration — broader orchestration context where circuit breakers fit
• Saga Pattern in Distributed Transactions — distributed transaction patterns that pair with circuit breakers
• Idempotency in Distributed Systems — retry safety and idempotent operations
• Transactional Outbox Pattern in Go — reliable event delivery alongside resilience patterns
• Go Error Handling Architecture — error classification at dependency boundaries
• Testing Concurrent Go Code with synctest — testing async behavior with circuit breakers
• Structured Logging in Go with slog — observability alongside circuit breakers
• github.com/sony/gobreaker/v2 — official gobreaker v2 package
• Go Context Cancellation and Timeouts — context patterns that pair with circuit breakers