I stopped letting my AI agent open 50 browser tabs at once and the CAPTCHA chaos finally calmed down

I thought I needed better stealth.

What I actually needed was less chaos.

My scraping agent was opening way too many Playwright sessions at once, getting hit with CAPTCHAs, retrying too aggressively, and then dragging the rest of the workflow down with it. CPU spiked. Throughput got worse. Downstream LLM steps started processing junk and duplicates.

The fix was not clever:

• cap active browser workers
• queue the overflow
• pace requests per domain
• keep retries bounded
• kill stuck sessions fast

A setup as simple as 5 active + 5 queued beat my 50-tab mess by a mile.

If you run browser automation inside agents, n8n workflows, Make scenarios, Zapier automations, OpenClaw, or custom Node/Python pipelines, this matters more than people admit.

The real problem was not the CAPTCHA page

I kept blaming Cloudflare, DataDome, and challenge pages.

They were part of the problem, sure.

But the bigger issue was self-inflicted load.

Here’s the pattern:

1. One worker gets blocked
2. It retries immediately
3. Ten more workers hit the same domain
4. They retry too
5. Old browser sessions hang around longer than they should
6. CPU and memory get eaten by sessions that are already doomed
7. The rest of the agent pipeline backs up behind them

That’s the death spiral.

It looks like a target-site problem in logs.

A lot of the time, it’s your own concurrency model.

Why 50 browser workers usually makes things worse

People treat browser concurrency like raw throughput.

It isn’t.

With Playwright or Puppeteer, every extra browser or context adds pressure:

• more CPU contention
• more memory pressure
• more event-loop lag
• more bursty traffic to the same domain
• more retries happening at the same time

That means two bad things happen at once:

• you look more bot-like to the target
• your own browser stack gets less stable

That second one surprised me more.

I expected anti-bot systems to get annoyed.

I didn’t expect my own infrastructure to become the bottleneck so quickly.

Lower concurrency can be faster

This is the part most scraping tutorials skip.

If each browser session gets enough CPU and memory, pages finish more reliably. Fewer timeouts. Fewer half-dead sessions. Fewer retries. Better throughput.

So yes, 5 healthy workers can absolutely beat 50 noisy ones.

That sounds backwards until you’ve watched a machine spend half its time babysitting stuck Chromium processes.

Scraping is not drag racing.

It’s traffic engineering.

The queue is the feature, not a nice-to-have

The best mental model I found is simple:

• allow work up to a concurrency limit
• queue the overflow
• reject the rest

That’s a lot better than pretending every request deserves immediate execution.

A bounded queue turns overload into delay instead of collapse.

Without a queue, burst traffic becomes:

• immediate failures
• aggressive retries
• more blocks
• more duplicate work downstream

With a queue, your system stays legible.

What I changed

1. Hard cap active browser sessions

If your first instinct is 20, try 5.

Seriously.

Here’s a Browserless example:

docker run -p 3000:3000 \
  -e "TOKEN=my-secure-token" \
  -e "CONCURRENT=5" \
  -e "QUEUED=5" \
  -e "TIMEOUT=300000" \
  registry.browserless.io/browserless/browserless/enterprise:latest

That gives you:

• 5 active sessions
• 5 queued sessions
• a real timeout budget for long pages

That one change already forces better behavior.

2. Add queueing on purpose

Queueing is how you stop spikes from turning into retry storms.

If your browser service is full, let requests wait.

If the queue is full, reject with something explicit like 429 and handle that upstream.

That is much better than pretending the system has infinite capacity.

3. Pace requests by domain

Global rate limits help.

Per-domain pacing helps more.

If ten workers all hammer the same host in a tight burst, you will get noticed.

Here’s a sane Crawlee starter config:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
  maxConcurrency: 5,
  minConcurrency: 1,
  maxRequestsPerMinute: 60,
  sameDomainDelaySecs: 2,
  maxRequestRetries: 2,
  retryOnBlocked: true,
  useSessionPool: true,
  proxyConfiguration,
  requestHandler: async ({ page, request, log }) => {
    log.info(`Processing ${request.url}`);
    // scrape here
  },
});

That won’t magically beat strong fingerprinting.

But it will stop you from acting like a denial-of-service attack wearing a trench coat.

4. Keep retries bounded

Unbounded retries are budget leaks with extra steps.

Two retries is often enough.

After that, you’re usually just paying to confirm the site still doesn’t like you.

A simple pattern:

async function fetchWithBoundedRetry(task, retries = 2) {
  let lastError;

  for (let attempt = 0; attempt <= retries; attempt++) {
    try {
      return await task();
    } catch (err) {
      lastError = err;
      if (attempt < retries) {
        await new Promise(r => setTimeout(r, 1000 * (attempt + 1)));
      }
    }
  }

  throw lastError;
}

Not fancy. Very effective.

5. Kill zombie sessions fast

This one is boring and hugely important.

If a page crashes, close it.
If a context is done, close it.
If a browser is stuck, kill it.

Every hanging session steals CPU and memory from useful work.

Basic Playwright hygiene:

const browser = await playwright.chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();

try {
  await page.goto(url, { timeout: 30000 });
  // scrape
} finally {
  await page.close().catch(() => {});
  await context.close().catch(() => {});
  await browser.close().catch(() => {});
}

A lot of “anti-bot instability” is really cleanup debt.

The settings that actually matter

If you’re tuning browser automation, these controls matter more than another generic “use stealth” tip.

These settings map directly to real failure modes.

• maxConcurrency keeps worker count bounded
• sameDomainDelaySecs reduces burstiness against one host
• maxRequestsPerMinute shapes global pressure
• maxRequestRetries prevents thrash
• retryOnBlocked lets you handle anti-bot responses differently
• useSessionPool helps with identity rotation without total randomness

How this blows up AI agent workflows

This is the part I care about most.

A flaky browser step doesn’t stay isolated.

In n8n, Make, Zapier, OpenClaw, or custom agents, browser failures fan out into the rest of the pipeline.

A typical flow looks like this:

1. Playwright collects pages
2. extraction step cleans content
3. GPT-5.4, Claude Opus 4.6, or Grok 4.20 classifies or enriches it
4. results get written to a database or sent to another system

Now imagine the browser layer is running 50 sessions and getting blocked.

You don’t just get more CAPTCHA pages.

You get:

• duplicate extraction jobs
• duplicate LLM calls
• partial pages being summarized
• queue depth growing across the workflow
• operators blaming the model layer for a browser-layer problem

I’ve seen teams chase “LLM instability” when the root cause was uncontrolled browser concurrency three steps earlier.

That’s why stable throughput beats fake peak throughput.

Why this matters even more when you pay for model usage

If your pipeline fans browser retries into more LLM calls, the cost problem gets ugly fast.

A bad browser concurrency setup can create:

• duplicate summaries
• duplicate classifications
• retries on junk HTML
• repeated enrichment on blocked pages

That’s how a scraping mistake turns into an AI billing mistake.

This is one reason I like the flat-rate model from Standard Compute for agent workflows. If you’re routing work across models like GPT-5.4, Claude Opus 4.6, and Grok 4.20, the last thing you want is per-token pricing punishing you for retry storms and noisy automations.

Standard Compute gives you an OpenAI-compatible API with predictable monthly pricing, which is a much better fit for long-running agents than constantly watching token spend every time a workflow gets messy.

You still want to fix the browser layer, obviously.

But predictable AI costs make the whole system less stressful while you do.

A practical starting point

If your scraper is flaky right now, I’d start here:

active browser sessions: 5
queued sessions: 5
same-domain delay: 2s
max requests per minute: 60
max retries: 2
session cleanup: aggressive

Then watch:

• success rate
• median page completion time
• CPU and memory usage
• queue depth
• duplicate downstream jobs
• LLM calls per successful page

That last metric matters a lot in agent pipelines.

If LLM calls per successful page is climbing, your browser layer is probably creating garbage work.

When one browser session at a time is the right answer

Sometimes the correct concurrency is 1.

If you’re dealing with:

• authenticated flows
• checkout funnels
• fragile dashboards
• aggressive reputation scoring
• high-friction anti-bot systems

then one page at a time may be the sane choice.

The lesson is not “always parallelize.”

The lesson is: never leave concurrency implicit.

Pick an upper bound.
Measure it.
Adjust it.

What bounded parallelism does not solve

This is not a magic bypass.

If a site has strong fingerprinting, challenge systems, or solid reputation scoring, lower concurrency will not make those disappear.

You may still need:

• better proxies
• stronger session management
• challenge-page handling
• fallback paths for human verification

But bounded parallelism gives you something extremely valuable:

a system that fails gracefully instead of catastrophically.

That matters.

The boring fix was the real fix

I went looking for smarter evasion.

What helped first was much less exciting:

• fewer active browsers
• explicit queue limits
• per-domain pacing
• bounded retries
• reliable cleanup
• session pools and proxy rotation where needed

Once I stopped flooding both the target and my own infrastructure, CAPTCHAs didn’t disappear.

They just stopped multiplying.

And that changed the economics of the whole workflow.

The browser layer got calmer.
The queue got saner.
The downstream LLM steps stopped processing as much junk.
The agent started acting like a service instead of a panic attack.

If you remember one thing, make it this:

The enemy is not just the CAPTCHA page. It’s the feedback loop where blocked sessions trigger more noisy sessions, which trigger more downstream automation work.

Break that loop with bounded workers and a queue, and the rest of your agent stack gets a lot easier to run.