Shape Free-Endpoint Load With a Batcher Before the Queue Overruns Your Free Server

A small team ran a PR-summary service on a stateless free server. Each pull request produced a task, and each task requested one autogenerated diff summary from a free model endpoint. The first morning, the queue reached 600 tasks, the server stayed stateless, and every task that failed was re-queued by a retry loop. The endpoint returned 429, then 503, then the queue overran the worker pool and the whole batch stalled.

The team's first instinct was to add more workers. That made things worse, because the endpoint was the bottleneck, not the server. The real fix was to shape the load before it reached the endpoint, and that shaping had to live in a place that survives server restarts. This article walks through a small batcher design that fits the constraint set of free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's open-source workflow can run this batcher against its free model endpoints and a free server option, which is stateless, so the queue must live outside the process.

The Constraint Set That Forced the Batcher

Three facts define the system boundary.

1. The free server is stateless. Any in-memory queue dies with the process. The only durable structure is an external store: Redis, Postgres, or a plain directory of files.
2. The free model endpoint has some rate limit. The exact number is irrelevant; what matters is that bursts violate it and bursts come from naive parallelism.
3. Tasks are independent and idempotent. Each task is a (repo, pr_number, base_sha) tuple that maps to exactly one summary.

Given those, the pipeline becomes: server polls external queue, collects tasks for a short window, submits them as a batch to the endpoint, then writes results back to the store. The batcher does not speed up the endpoint. It prevents the client from making the endpoint slower.

Why Naive Parallelism Fails

Suppose the worker pool simply takes one task per thread. Under a per-minute rate limit, the first 20 tasks succeed, the 21st gets a 429, the retry re-enqueues it, and the retry races with the next 20 tasks. The effective concurrency collapses to zero because every retry is also a request. The server sees a firehose, not a batch.

A batch is the opposite: collect tasks, then send them in a bounded stream with a backoff between chunks. The endpoint sees a predictable trapezoid of load, and the free server only needs enough memory to hold one batch at a time.

The Batcher Design

Here is the data flow.

PR webhook -> external queue -> polling worker
                                   |
                                   v
                            batch collector (size / time)
                                   |
                                   v
                            concurrency limiter
                                   |
                                   v
                            free model endpoint
                                   |
                                   v
                            result store

The batcher has three knobs: max_batch_size, window_seconds, and min_interval_between_batches. The collector fills until either threshold, then hands the batch to a single sender. The sender runs at most one batch at a time.

Minimal Python Implementation

The following is a toy batcher. It uses a filesystem directory as the external queue so it survives any server restart. No third-party dependencies.

# batch_pr_summaries.py -- collect tasks, batch them, send them
import json, pathlib, time, subprocess, sys

QUEUE_DIR = pathlib.Path("queue_dir")
RESULTS_DIR = pathlib.Path("results_dir")
MAX_BATCH_SIZE = 5
WINDOW_SECONDS = 10
MIN_INTERVAL_SECONDS = 2


def drain_queue():
    tasks = []
    for path in sorted(QUEUE_DIR.glob("task_*.json")):
        if len(tasks) >= MAX_BATCH_SIZE:
            break
        tasks.append(json.loads(path.read_text()))
        path.unlink()
    return tasks


def send_batch(tasks):
    """Send a list of tasks to the free model endpoint.

    In this example we just sleep and write a canned result.
    Replace this with the actual HTTP call to your model provider.
    """
    time.sleep(2)
    for task in tasks:
        result = {"repo": task["repo"], "pr": task["pr"], "summary": "ok"}
        out = RESULTS_DIR / f"result_{task['pr']}.json"
        out.write_text(json.dumps(result))


def main():
    while True:
        batch = drain_queue()
        if not batch:
            time.sleep(1)
            continue

        send_batch(batch)
        time.sleep(MIN_INTERVAL_SECONDS)


if __name__ == "__main__":
    sys.exit(main())

This is not production code. It lacks error handling, backoff on 429, and result versioning. It is a minimal fixture to make the scheduling decision concrete, not a library.

Production Decisions as a Table

The batcher works only if you choose the knobs deliberately. Here is the tradeoff table I use when deploying this pattern.

Small batches give fast feedback but invite 429s. Large batches protect the endpoint but make one outage expensive. The correct size depends on the endpoint's documented or observed limit. If you do not know the limit, start medium and measure.

Failure Injection That Validates the Design

The batcher is only trustworthy if it behaves correctly when the endpoint fails. Run these injections in a staging directory.

1. 429 storm: swap send_batch to return a fake 429 for the first three calls. The limiter must not resend the same batch until its backoff timer expires.
2. Server restart mid-batch: kill the process after drain_queue and before send_batch. Task files that were already unlinked are lost. To survive that, the drain step must copy, not move, and the sender must mark tasks as claimed in the result store. The toy above does not do this.
3. Duplicate batch: let two workers drain overlapping task sets. The result store must be keyed by PR number so the second write is a no-op.

The first and third are cheap to fix in the toy. The second requires a state machine per task: pending -> claimed -> done. That is why the batcher needs an external store with atomic compare-and-swap, not just a directory.

Who Should Not Use This Pattern

If your workload has fewer than twenty tasks per day, a batcher adds complexity without payoff. If your endpoint has no rate limit, the limiter is unnecessary. If your tasks are not idempotent, batching multiplies the damage of a retry. In those cases, a simple synchronous loop is the honest engineering choice.

The pattern earns its keep when three conditions all hold: the endpoint is shared or rate-limited, the server is stateless, and the task volume is high enough that bursts reliably trigger throttling. That is precisely the situation for which MonkeyCode's free model endpoints and free server option are a reasonable fit, provided you bring your own durable queue.

The Counterexample Question

Here is the test that exposes most naive batchers: two tasks for the same PR arrive in different batches, and the first batch succeeds after the second has already entered the limiter. The result store receives one summary, then overwrites it. Which event order breaks your invariant? If you reject the second batch, you lose a valid update. If you replay it, you may produce a stale overwrite. Decide before the queue grows, because the free endpoint will not decide for you.