Stop Calling Every Short Scraping Response a Proxy Failure

HTTP 200 is not a body-integrity guarantee. A connection can end after headers, a client can stop reading, decompression can fail, or an intermediary can apply a limit. Before replacing a proxy pool, record where the response became incomplete.

Keep three lengths separate

With Content-Encoding: gzip, Content-Length often describes compressed transfer bytes while your library exposes a decoded body. Comparing those numbers directly creates false alarms.

Track:

const evidence = {
  declaredTransferBytes: null,
  observedWireBytes: null,
  decodedBodyBytes: 0,
  contentEncoding: null,
  contentRange: null,
  sentinelPresent: null,
  bodyHash: null,
  firstByteMs: null,
  totalMs: null
};

Stream the body and classify it

async function collectWithIntegrity(response, expected) {
  const chunks = [];
  let decodedBodyBytes = 0;

  for await (const chunk of response.body) {
    decodedBodyBytes += chunk.byteLength;
    chunks.push(chunk);
  }

  const body = Buffer.concat(chunks);
  const declared = Number(response.headers.get("content-length")) || null;
  const encoding = response.headers.get("content-encoding");
  const range = response.headers.get("content-range");
  const sentinelPresent = expected.sentinel
    ? body.includes(Buffer.from(expected.sentinel))
    : null;

  return {
    body,
    evidence: {
      status: response.status,
      declaredTransferBytes: declared,
      decodedBodyBytes,
      contentEncoding: encoding,
      contentRange: range,
      sentinelPresent
    }
  };
}

The classifier should allow complete, truncated, intentional_partial, and inconclusive. A valid 206 plus a matching Content-Range may be correct; HTTP 200 with a missing application sentinel may still be incomplete.

Use a four-route control

Run the same authorized request through:

1. a direct control;
2. a known-good proxy;
3. the suspected route;
4. an independent exit in the same region.

Keep URL, method, headers, locale, session, parser, and time window stable. Compare status, framing, encoding, byte counts, body hash, sentinel, and timing.

• One route fails: investigate that exit or upstream path.
• Every proxy stops at the same byte boundary: inspect shared client or gateway limits.
• Direct and proxy routes match: investigate the target, request profile, or client.
• Cutoffs happen at the same elapsed time: investigate read timeouts.

On infrastructure you own, test controlled payloads such as 256 KB, 1 MB, 2 MB, 4 MB, and 8 MB. Use compressible and incompressible data separately. This reveals whether the boundary is based on wire bytes, decoded bytes, or time.

Preserve the first failed body and last successful offset. Use bounded retries; do not rotate exits unless route dependence is the experiment.

I work with 98IP. For authorized route comparisons: https://en.98ip.com/?k=dev

Disclosure: collect only data you are authorized to access. Follow site terms, privacy and data-protection rules, contractual limits, and reasonable rates. Do not use proxies to bypass authentication, paywalls, access controls, or enforcement.