When an automation job fails, the visible symptom rarely identifies the failing layer. A timeout may come from the proxy, the target, the client, or an overloaded worker. A 403 or 429 may follow an IP, account, cookie, request pattern, or rolling rate window.
Here is a small test harness design that changes one variable at a time.
1. Define four cells
A: baseline route -> neutral endpoint
B: candidate proxy -> neutral endpoint
C: baseline route -> target endpoint
D: candidate proxy -> target endpoint
Use the same method, headers, body, browser version, and timeouts in every cell. Interleave at least 20 requests per cell so a time-based target policy does not align with one entire batch.
The neutral endpoint must be stable and independent of the target. If direct access is not allowed, make the baseline a separately operated known-good proxy.
2. Record layer-specific timings
const observation = {
runId,
cell,
proxyConnectMs,
dnsMode,
addressFamily,
tlsMs,
firstByteMs,
totalMs,
status,
redirectCount,
responseBytes,
observedExit,
retryNumber,
businessOutcome
};
Redact session identifiers and never log passwords, cookies, tokens, or personal data.
3. Classify the pattern
function classify({ A, B, C, D }) {
if (A.ok && !B.ok) return "candidate_route";
if (B.ok && C.ok && !D.ok) return "route_target_interaction";
if (!C.ok && !D.ok) return "target_or_application";
return "inconclusive";
}
This simplified classifier is a starting point, not a verdict. Split results by status family, exit, ASN, region, session age, and request rate before taking action.
Useful interpretations:
- B fails while A succeeds: inspect proxy authentication, DNS, transport, TLS, and exit health.
- B and C succeed while D fails: the target may treat the proxy route differently. Test more exits before rejecting the whole pool.
- C and D fail similarly: check target health, account state, selector drift, or request behavior.
-
Failure tracks rate: reduce rate, add jitter, and honor
Retry-After. - Failure tracks a session across exits: rotation is unlikely to help; inspect identity state.
4. Confirm with a controlled staircase
Hold short stages at 1, 2, 4, and 8 requests per second, or lower values appropriate for the target. Control concurrency separately from throughput. At every stage calculate:
- successful business outcomes per minute;
- p50/p95 time to first byte;
- 403, 407, 429, 5xx, timeout, and reset rates;
- unique healthy exits;
- retry amplification (
attempts / successful outcomes).
Stop when authorization, terms, robots directives, published limits, or safety thresholds require it. The goal is to identify a permitted operating envelope, not bypass controls.
5. Make a layer-specific decision
Quarantine an unhealthy route. Reduce traffic for target throttling. Preserve a legitimate session when continuity matters. Fix expired authentication or parser changes when the network succeeded but the business check failed.
I work with 98IP. If you need an independent proxy route for authorized test comparisons, see https://en.98ip.com/?k=dev
Disclosure: I work with 98IP. Use proxies only for lawful, authorized access and comply with target terms, privacy requirements, robots directives, and rate limits.
