Your retry logic, timeouts and circuit breakers only get tested when a dependency actually misbehaves, and that usually happens in production. A fault-injection proxy lets you cause that on purpose. Put it between your service and the dependency, then tell it to add 1.5 s of latency, return a 503 on 30% of requests, or hang up mid-request.
ChaosProxy is a small reverse proxy written in Python on aiohttp that does this. Rules are managed at runtime over a JSON API, so you can turn faults on and off from a test script without restarting anything. Everything below was run locally.
Setup
pip install aiohttp multidict
python3 chaosproxy.py --upstream http://127.0.0.1:5700 --port 8780 --host 127.0.0.1
Point your client at http://localhost:8780 instead of the real service. With no rules, requests pass straight through, including method, query string, body and headers (hop-by-hop headers are filtered out). For this post the upstream is a tiny API that returns an order:
curl -s localhost:8780/orders/1001
# {"id": "1001", "status": "paid"}
Fault 1: latency
curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' -d '{
"rule_id": "slow-orders",
"path_pattern": "^/orders/",
"fault_type": "latency",
"params": {"delay_seconds": 1.5, "jitter_seconds": 0.3}
}'
{"status": "created", "rule": {"rule_id": "slow-orders", "path_pattern": "^/orders/", "method": "*", "probability": 1.0, "fault_type": "latency", "params": {"delay_seconds": 1.5, "jitter_seconds": 0.3}, "enabled": true}}
curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" localhost:8780/orders/1001
# 200 in 1.634716s
The request still reaches the upstream and returns 200, just late. That's how you find out whether your client's 1 s timeout really fires, and what the user sees when it does.
Remove the rule when you're done:
curl -s -X DELETE localhost:8780/__chaos/rules/slow-orders
# {"status": "deleted", "rule_id": "slow-orders"}
Fault 2: intermittent errors
probability makes a rule fire on a fraction of matching requests, and method limits it to one HTTP verb:
curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' -d '{
"rule_id": "flaky", "path_pattern": "^/orders/", "method": "GET",
"probability": 0.3, "fault_type": "http_error",
"params": {"status_code": 503, "body": {"error": "upstream unavailable"}}
}'
for i in $(seq 1 20); do curl -s -o /dev/null -w "%{http_code} " localhost:8780/orders/$i; done
503 200 200 200 200 200 200 503 200 200 200 200 200 200 503 200 503 200 503 200
5 of 20, close to the 30% asked for. This is the realistic case for testing retries with backoff. Failed requests return your JSON body with the status you chose, and they never reach the upstream.
Fault 3: dropped connections and truncated bodies
curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' \
-d '{"rule_id":"drop","path_pattern":"^/orders/9$","fault_type":"connection_drop"}'
curl -s localhost:8780/orders/9; echo "curl exit $?"
# curl exit 52 (empty reply from server)
payload_corrupt fetches the real response and then truncates it ("strategy": "truncate") or flips random bytes (any other strategy value):
curl -s -X POST localhost:8780/__chaos/rules -H 'Content-Type: application/json' \
-d '{"rule_id":"trunc","path_pattern":"^/orders/7$","fault_type":"payload_corrupt","params":{"strategy":"truncate"}}'
curl -s localhost:8780/orders/7
# {"id": "7", "s
That's exactly what a JSON parser hates. Good for checking that your client reports a clear error instead of crashing.
Seeing what happened
curl -s localhost:8780/__chaos/rules # active rules
curl -s localhost:8780/__chaos/stats
{"requests_total": 23, "requests_proxied": 18, "faults_injected": 6,
"fault_breakdown": {"latency": 1, "http_error": 5, "payload_corrupt": 0, "connection_drop": 0}}
Every injected fault is also logged:
[WARNING] [ChaosProxy] INJECTING FAULT [HTTP_ERROR] via rule [flaky] on GET /orders/1
Using it in an automated test
Because rules are just HTTP calls, a test can switch a fault on, assert on behaviour, and clean up:
import requests
CHAOS = "http://localhost:8780/__chaos/rules"
def test_checkout_survives_slow_payments():
requests.post(CHAOS, json={"rule_id": "slow-pay", "path_pattern": "^/payments",
"fault_type": "latency", "params": {"delay_seconds": 3}})
try:
r = requests.post("http://localhost:3000/checkout", json={"cart": 1}, timeout=10)
assert r.status_code == 202 # we expect async fallback, not a 500
finally:
requests.delete(f"{CHAOS}/slow-pay")
How matching works
Each rule has a regex path_pattern matched against the path plus query string, a method (* by default), a probability between 0 and 1, and an enabled flag. Rules are checked in the order they were created, and the first match wins. Paths under /__chaos are reserved for the control API and never forwarded.
Limitations
-
The control API has no authentication. Anyone who can reach the port can add rules. The default bind address is
0.0.0.0, so pass--host 127.0.0.1(as above) or keep it on an isolated test network. Don't expose it publicly. - Rules live in memory. Restarting the proxy clears them, so re-create them from your test setup.
- One upstream per proxy instance. Run several instances on different ports for several dependencies.
-
payload_corruptreads the whole upstream response into memory before corrupting it, so it isn't meant for very large downloads. Normal forwarding streams. - HTTP only. There's no TCP-level chaos (packet loss, bandwidth limits) and no TLS termination. Put it on the plain-HTTP side of your stack.
FAQ
Does it change responses when no rule matches? No. Status, headers and body are relayed as received, including compressed bodies.
What's the default upstream? https://httpbin.org. Always pass --upstream for real use.
Python version? Python 3 with aiohttp and multidict.
ChaosProxy is a one-time purchase that includes the source and README: get ChaosProxy ($29). More details are on the ChaosProxy product page.
