Staging traffic is never quite like production. Real users send odd query strings, old client versions and payloads nobody wrote a test for. Shadow testing (also called traffic mirroring or a dark launch) means sending a copy of real production requests to the new version of a service, while users keep getting answers from the current version. If v2 crashes or returns something different, nobody notices except you.
TrafficMirror is a small Python proxy built on aiohttp that does exactly that. It forwards each request to a primary backend and returns that response to the client. In the background it fires an identical copy at a shadow backend without waiting for the result. Everything below was run locally.
Setup
pip install aiohttp
python3 trafficmirror.py \
--primary http://127.0.0.1:5801 \
--shadow http://127.0.0.1:5802 \
--port 8781 --host 127.0.0.1
[INFO] TrafficMirror: Initializing TrafficMirror on port 8781...
[INFO] TrafficMirror: Primary upstream: http://127.0.0.1:5801
[INFO] TrafficMirror: Shadow mirror: http://127.0.0.1:5802
For the demo, v1 (primary) and v2 (shadow) are two copies of a pricing service. They log each request they get, and v2 rounds the total slightly differently. In a real setup, you'd put TrafficMirror where your load balancer or ingress currently sends traffic to v1.
Requests go to both versions, users only see v1
curl -s "localhost:8781/quote?sku=A12&qty=3"
# {"version": "v1", "total": 107.0}
curl -s -X POST localhost:8781/quote -H 'Content-Type: application/json' -d '{"sku":"B7","qty":1}'
# {"version": "v1", "total": 107.0}
The logs of the two services:
--- v1
[v1] GET /quote?sku=A12&qty=3 body=-
[v1] POST /quote body={"sku":"B7","qty":1}
--- v2
[v2] GET /quote?sku=A12&qty=3 body=-
[v2] POST /quote body={"sku":"B7","qty":1}
Same method, path, query string and body, on both. Headers are copied too, except hop-by-hop ones (Connection, Transfer-Encoding and so on), and Host is rewritten for each target.
The shadow can't slow users down or break them
The shadow call runs as a separate asyncio task, and the client response doesn't wait for it. To check, I made v2 sleep 3 seconds per request:
curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" "localhost:8781/quote?sku=A12"
# 200 in 0.003065s
v2 still received the request three seconds later. Then I stopped v2 entirely. The client still got its answer:
curl -s -w " [%{http_code}]\n" "localhost:8781/quote?sku=A12&qty=3"
# {"version": "v1", "total": 107.0} [200]
and TrafficMirror only logged a warning:
[WARNING] TrafficMirror: Mirror request failed: GET http://127.0.0.1:5802/quote?sku=A12&qty=3 - Error: Cannot connect to host 127.0.0.1:5802 ...
Shadow requests time out after 10 seconds, and primary requests after 30. If the primary times out the client gets a 504, and if it can't be reached a 502.
The core of it
# 1. Dispatch shadow mirror task asynchronously without blocking
asyncio.create_task(
self._send_shadow_request(method=request.method, path_qs=path_qs,
headers=headers_dict, body=body)
)
# 2. Forward to PRIMARY target and return its response
async with self.session.request(method=request.method, url=primary_target_url,
headers=primary_headers, data=body,
timeout=self.timeout) as response:
resp_body = await response.read()
return web.Response(status=response.status, body=resp_body, headers=...)
The request body is read once and reused for both targets, so POST/PUT payloads reach the shadow intact. The primary's status, headers and body are relayed byte for byte, compressed responses included.
Comparing v1 and v2
TrafficMirror sends traffic. It doesn't compare responses. The simplest way to find differences is to log on the shadow side. Have v2 log its responses (or compute v1's answer too and log mismatches), then go through those logs. In the demo, v2's log shows it was handling the same requests, and a response log would show 107.01 vs 107.0. That's the kind of rounding difference you want to find before switching over.
Limitations
-
The shadow receives writes too. A mirrored
POST /orderswill create an order in v2's database. Give the shadow its own database, and its own fake or sandbox payment and email integrations, or mirror only to read-only endpoints. - No response diffing or sampling. Every request is mirrored, and comparison is up to you (see above). Shadow results are only logged on failure or at debug level.
- Responses are buffered in memory before being returned, so very large downloads or streaming endpoints aren't a good fit.
- Timeouts (30 s primary, 10 s shadow) are set in the code, not by flags.
- HTTP only, with no TLS termination. Run it behind your existing TLS proxy or ingress.
- The default bind address is
0.0.0.0. Pass--host 127.0.0.1when it only needs local access.
FAQ
Does mirroring double the latency? No. The client waits only for the primary. In the test above, a 3-second shadow added nothing measurable.
Can I mirror to several shadows? One shadow per instance. You can chain instances or run several side by side.
Dependencies? Python 3 and aiohttp.
TrafficMirror is a one-time purchase that includes the source and README: get TrafficMirror ($29). More details are on the TrafficMirror product page.
