What Is Middleware? The Onion Model, and Why the Order Is the Configuration

📺 Prefer to watch? 90-second YouTube Short · 💬 Telegram

Originally published on software-engineer-blog.com.

One small service. One process, one port, six business endpoints plus a health check.

Three things had to happen on almost every one of them: check the caller's token, write a log line with a request id, start and stop a timer. Nine lines. They were written once, and then pasted by hand into endpoint after endpoint.

# the SAME nine lines, pasted by hand into endpoint after endpoint
async def ep_account(scope, receive, send):
    user = user_for(header(scope, b"authorization"))        # pasted
    if user is None:                                        # pasted
        return await send_json(send, 401, {"error": "no"})  # pasted
    started = clock()                                       # pasted
    log.info("GET /account", rid=rid(scope))                # pasted
    result = handlers.account(user)                         # the actual work
    metrics.observe("GET /account", clock() - started)      # pasted
    await send_json(send, 200, result)

Then the sixth endpoint was added. It was copied from the one next to it, and in the copy the two auth lines did not come along.

async def ep_create_review(scope, receive, send):
    started = clock()                                       # pasted
    log.info("POST /reviews", rid=rid(scope))               # pasted
    result = handlers.create_review(user_of(scope), body)   # the actual work
    metrics.observe("POST /reviews", clock() - started)     # pasted
    await send_json(send, 200, result)   # the auth lines are not here

Sweep the API with no Authorization header at all and you get this:

GET     /books            401  {"error": "unauthorized"}
GET     /books/1          401  {"error": "unauthorized"}
POST    /orders           401  {"error": "unauthorized"}
GET     /orders/1         401  {"error": "unauthorized"}
GET     /account          401  {"error": "unauthorized"}
POST    /reviews          200  {"review": {"user": "anonymous", "book": 1, "stars": 5}}
GET     /healthz          200  {"ok": true}

POST /reviews answered everyone. Nothing threw. No test failed. No error log line was ever written — the endpoint was working exactly as its code said. Counting the cross-cutting lines inside the endpoint functions gives 50 across the module: nine on each of five endpoints, and five on the sixth. The missing four are the entire incident.

That is the problem middleware solves. Not "less typing" — we will get to the honest bill later, and it is not smaller. One place.

What a middleware actually is

A middleware is a function that is handed two things: the request, and the next thing to call.

That second argument is the entire idea.

def timing(request, call_next):
    started = clock()                      # 1. before  — runs on the way IN
    response = call_next(request)          # 2. hand it on, and WAIT here
    took = clock() - started               # 3. after   — runs on the way OUT
    response.headers["x-took-ms"] = took
    return response

Look at line 2. It is not "finish my job and hand over." It blocks until everything deeper in the chain has run and come back. Which is precisely why line 3 is able to know how long the whole rest of the request took.

If you delete the call_next argument, you no longer have a middleware. You have a hook.

The chain is an onion, not a queue

Almost everyone's first mental picture of a middleware chain is a row of turnstiles: the request passes through gate one, then gate two, then gate three, then reaches your handler. It is a tidy picture and it is wrong.

The steps do not stand in a row. Each one wraps the next. So every step runs twice: once on the way in, and once on the way back out.

Here is a real trace, printed by a chain of five:

--> ERRORS
--> REQUEST ID
--> LOG + TIMER
--> AUTH
--> RATE LIMIT
    [ YOUR HANDLER RUNS ]
<-- RATE LIMIT
<-- AUTH
<-- LOG + TIMER
<-- REQUEST ID
<-- ERRORS

entered 5 steps, exited 5 steps
exits are the entries REVERSED : True

Read the two halves. Going in, the order is the list order. Coming out, it is the list reversed. That reversal is the whole shape, and it is not a stylistic detail — it is what makes several of the most common middlewares possible at all:

• A timer on the outside can measure the whole request, because its "after" half runs last.
• An error handler on the outside can catch an exception thrown by anything beneath it.
• A response-header step can set a header only after the response exists.

Measured on a chain of four: the outermost layer's mean was 68.3 µs, the innermost (the handler alone) 33.9 µs. The handler was 49.7 % of what the outermost layer measured, and the outer figure was greater than or equal to the inner one on every one of 2,000 requests. That is the onion, in a number.

A step is allowed to refuse

The second consequence of "you are handed the next thing to call" is that you may decline to call it.

def auth(request, call_next):
    if not valid_token(request.headers.get("authorization")):
        return json({"detail": "unauthorized"}, status=401)   # no call_next
    request.state.user = user_for(request)
    return call_next(request)        # only a valid caller gets past this line

Fifty requests with no token, through a chain of auth → rate limit → database lookup → handler:

AUTH reached           50
RATE LIMIT reached      0   <- never ran
database lookup         0   <- never ran
your handler reached    0   <- never ran
responses              50 x 401

Your handler did not decide to refuse those. Your handler was never asked. A browser CORS preflight that never reaches your code is the same move, and so is a WAF rule, and so is an API gateway's quota check.

This is also the first hint of a real operational problem. If a request is rejected two layers above your handler, your handler's own metrics never see it. In one run, 160 requests arrived, the handler counted 50, and the gap of 110 reconciled exactly to 80 rejections plus 30 throttles. That is 68.8 % of all traffic invisible to the dashboard most teams actually look at.

The order is the configuration

Here is the part that makes middleware a design decision rather than a convenience.

# the whole ordering decision, in one list, outermost first.
CHAIN = [errors, request_id, logging, auth, rate_limit]   # A: auth, then limit
CHAIN = [errors, request_id, logging, rate_limit, auth]   # B: limit, then auth

Same two components. Same code inside them. One line moved. Now send the identical 90-request burst at both — 30 from one authenticated user, 30 from thirty different authenticated users, 30 with no token, limit 10 per key:

40 successful responses versus 7. That is not a mild trade-off, it is two different products.

And neither order is wrong. Order A is a per-user quota; order B is a per-source flood gate. The limiter can only key by user if something above it has already identified one — placed first it has no user yet, so it falls back to keying by client address, and behind a shared address that means thirty innocent users share one bucket. That is not a bug. It is the consequence of the position.

The same principle bites a request-id step. Placed first, all four downstream log lines carry the id. Placed last, only 1 of 4 does:

request-id FIRST                      request-id LAST
[log:edge]    rid=ffc68931            [log:edge]    rid=-
[log:access]  rid=ffc68931            [log:access]  rid=-
[log:audit]   rid=ffc68931            [log:audit]   rid=-
[log:app]     rid=ffc68931            [log:app]     rid=fe9f57b1

Note that it is 3 of 4 lost, not all four — the step still tags whatever runs beneath it. And the error handler is the sharpest case of all. Outermost, with a failing layer below it: HTTP 500, body {"error": "internal server error"}, caught. Innermost, with the failing layer above it: no HTTP response at all — the exception escapes the application entirely.

The worst property of an ordering mistake is that it is silent. On that 90-request burst under the wrong order: exceptions escaped 0, exceptions caught 0, ERROR log lines 0, 5xx responses 0, log lines written 90 — all INFO, every status code documented and expected. The only visible symptom, and only if you already knew your intent, was that 27 requests from users who had sent one request each came back 429.

The honest bill

Middleware is usually sold as a cleanup. Measured, it is a trade, and it is worth being precise about what you are trading.

It does not mean fewer lines. The endpoint module shrank from 92 SLOC to 52. But the steps themselves are another 137 SLOC, plus an 8-line list to order them. At six endpoints, the repository is bigger. What you bought is not brevity — it is that the auth rule exists in one place, so the sixth endpoint cannot forget it. On the chain variant, the same unauthenticated sweep returns 401 on all six business endpoints, including the newest one, which nobody had to remember.

It costs something on every request. A chain of eight layers against a bare handler: bare 4.3 µs, full chain 30.7 µs — an overhead of 26.4 µs/request, about 7.1×, roughly 3.3 µs per layer. Quoted alone, that number sounds alarming. So here is the other one: measured end to end over a real socket, a request took 640.1 µs, and the chain was 4.1 % of it. Quote either figure by itself and you are misleading someone. The chain is expensive relative to a function call and cheap relative to a request.

Every request pays, including the ones that did not need to. One layer doing a small database read added +13.8 µs/request to /healthz — 2.02× the same chain without it — and performed 5,300 of 5,300 database reads for an endpoint that needed exactly none.

It is invisible from the handler, and the blast radius is the whole API. Changing if scheme != "Bearer": to if scheme == "Bearer": is a one-character diff in a shared layer. Result: 6 of 6 business endpoints reject a valid credential, 1 file changed, 0 endpoint files changed, and handlers.py byte-identical before and after. Nothing in the handler you are staring at explains the failure.

The same shape, in front of a model

If you work on LLM serving, you have this chain whether you called it middleware or not — it is what an inference gateway is. And the vocabulary maps cleanly:

• Auth becomes API-key resolution to a tenant and a model allowlist.
• Rate limiting becomes a token budget rather than a request budget, and the ordering lesson lands twice as hard: a limiter placed before authentication cannot key by tenant, so one noisy customer and thirty quiet ones share a bucket — the exact 27-out-of-30 failure above, except now it is a paying customer's SLA.
• Request id becomes the trace id that has to survive a response lasting several seconds.
• Guardrails — prompt-injection screening on the way in, PII or policy filtering on the way out — are the onion's two halves, and the reason they belong in a wrapper rather than in the handler is the same reason auth did: the next endpoint you add must not be able to skip them.
• Cost accounting is a pure "way out" step: you cannot bill for tokens you have not generated yet.

But one thing genuinely does not transfer, and it catches people. The onion's "on the way back out" half assumes the response is a single object handed back up the stack. When you stream tokens over SSE, the response object returns almost immediately and the body arrives afterwards. A timing middleware written the ordinary way will therefore record time to first token, not total generation time — a number that can be twenty times smaller and looks perfectly healthy on a dashboard. The same applies to an output filter: by the time your "after" half runs, the first tokens are already on the client's screen. Streaming-aware guardrails have to wrap the iterator, not the response.

The shape holds. The assumption that a request is one atomic unit of work does not.

Copy-paste versus chain, side by side

The verdict

Use middleware for the things that are genuinely true of almost every request — authentication, request ids, error handling, logging, rate limiting, CORS. Those earn the wrapper, because their failure mode is omission, and a chain makes omission impossible.

Do not use it for things that are true of some requests. A layer that reads the database for every call so that three endpoints can avoid a lookup is a tax collected 5,300 times to be spent 3 times; that belongs in a dependency the three endpoints ask for.

And then treat the order as what it is. It is not registration boilerplate at the bottom of a file — it is a configuration file for your API's behaviour, written in the least obvious syntax imaginable, and it fails without raising anything. Put the list somewhere a reviewer will look at it, write down why each step sits where it does, and test the order the way you would test a feature: send a burst, count the status codes, and check you got the product you meant to build.

Your handler is not the first code that runs. Know what is in front of it.

References and further reading

On the problem — cross-cutting concerns and the pasted nine lines

• Gregor Kiczales et al., Aspect-Oriented Programming (ECOOP, 1997) — the paper that named cross-cutting concerns and the scattering/tangling problem; the missing auth block in POST /reviews is a textbook instance of scattering.
• Deepak Alur, John Crupi & Dan Malks, Core J2EE Patterns, 2nd ed. (Prentice Hall, 2003) — the Intercepting Filter pattern: a configurable, ordered chain of filters around a request handler. This is the pattern middleware implements, described before the word "middleware" was common in web frameworks.

On the shape — why each step wraps the next

• PEP 3333 — Python Web Server Gateway Interface v1.0.1 — the primary source for the wrapping definition: middleware is itself an application that plays server to the application below it. The onion is written into the spec, not into a framework.
• The ASGI Specification — the async successor, and the place to read how the model changes when a response is a stream of events rather than one return value.
• Writing middleware for use in Express apps (Express documentation) — the shortest statement of the next() contract, including what happens when you decline to call it.
• Roy T. Fielding, Architectural Styles and the Design of Network-based Software Architectures, ch. 5 (dissertation, 2000) — the layered system constraint: why intermediaries a client cannot see are a deliberate property of the web, not an accident.

On ordering, refusal, and what it costs you operationally

• Michael T. Nygard, Release It!, 2nd ed. (Pragmatic Bookshelf, 2018) — the case for guards that run before your code at all (bulkheads, circuit breakers, handshaking), and why a component that sheds load must sit where it can actually see the load.
• Charity Majors, Liz Fong-Jones & George Miranda, Observability Engineering (O'Reilly, 2022) — instrumenting at the edge versus inside the handler; the direct answer to the 68.8 % of traffic the handler's own counters could not see.
• Jeffrey Dean & Luiz André Barroso, The Tail at Scale (Communications of the ACM, 56(2), 2013) — why a small fixed per-request cost is usually the wrong thing to worry about, and variance is the right one.

On the model-serving section

• Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP, 2023) — the vLLM paper; useful here for why a request to a model server is not one atomic unit of work, which is exactly the assumption a "way back out" middleware makes.

If a reference you'd expect is missing, say so in the comments and I'll add it.

Watch the reel: Middleware — the code that runs before your handler · or the full episode.