Why use BullMQ if the simulation worker is allowed to process only one job at a time?
The usual queue story is parallelism. Add workers, raise concurrency, increase throughput. That story did not fit this gateway. A MiroFish run starts graph construction, profile preparation, an agent simulation, action-log import, and report generation. The upstream container image occupies about 14 GB after extraction, and the live process shares a local host with PostgreSQL, Redis, the gateway, the frontend, and development tooling.
During the first full English simulation, the process ran for 28 minutes and died with exit code -9. There was no application exception to catch. The operating system killed it under memory pressure. I stopped 21 nonessential containers and recovered roughly 7 GB before rerunning.
After that failure, concurrency: 1 stopped looking conservative. It became the capacity contract.
The queue separates admission from execution
An API request should not hold an HTTP connection open while a swarm runs. The simulation route creates a database record, places a self-contained job on Redis, and returns the simulation ID with a pending status. A worker later receives the tenant ID, scenario ID, simulation ID, agent count, round count, and model provider needed to run the job.
That job shape matters. It contains identifiers and configuration, not open database connections or request objects. BullMQ can persist it across a gateway restart. The worker reloads the scenario and validates tenant ownership when execution begins.
With one active slot, the queue performs three jobs that concurrency alone cannot describe. It absorbs bursts, preserves ordering under limited memory, and gives every accepted simulation an observable state before heavy work starts. A burst can become pending records without becoming many MiroFish processes fighting over the same host.
The trade-off is queueing delay. A long simulation blocks every job behind it. I accepted that because an honest pending state is better than concurrent runs that the machine cannot finish. If demand grows, the next unit of scale is another isolated worker host, not a larger concurrency number on the same host.
Backpressure also has to be visible to the caller. The API stores a simulation before queue admission and returns its ID immediately. A progress route maps pending, queued, graph-building, simulating, reporting, and terminal states into plain phase labels. It reports elapsed time, agent count, round count, and whether the run is active.
The current response does not expose queue position or an estimated start time. With one worker, that omission becomes noticeable as soon as more than a few simulations wait. I would add job age and queue position before adding parallel hosts so operators can distinguish slow execution from slow admission. A queue that protects the machine but leaves users guessing has moved the failure rather than solved it.
The number is in code, not operator folklore
The worker factory carries the limit and the failure instrumentation together:
export function createSimulationWorker(): Worker {
const worker = new Worker(
QUEUE_NAMES.RUN_SIMULATION,
async (job) => {
const {
simulationId,
scenarioId,
tenantId,
agentCount,
roundCount,
llmProvider,
} = job.data;
await runSimulation({
simulationId,
scenarioId,
tenantId,
agentCount,
roundCount,
llmProvider,
});
},
{
connection: parseRedisUrl(env.REDIS_URL),
concurrency: 1,
},
);
return worker;
}
createSimulationWorker() is created once during service startup. The API and quick-launch routes both add run-simulation jobs with three attempts and exponential backoff starting at 60 seconds. The worker emits a warning for an intermediate failure and an error with permanentFailure: true when all configured attempts are spent.
I kept the concurrency value beside the worker rather than making it an unchecked environment variable. On this deployment, raising it changes the memory safety model. That deserves a code review and a load test, not a late-night configuration edit.
There is a cost to that choice. Different hosts cannot tune independently without a release, and a future production cluster may have worker classes with different capacities. I would introduce a bounded configuration value when there are at least two proven host profiles. Until then, a configurable footgun does not count as operational flexibility.
Exit code -9 changed what I measured
Before the killed run, I was watching application logs and MiroFish status. Neither source showed a TypeScript error because neither process decided to fail. The host reclaimed memory.
That surprised me because the run had already survived graph generation and entered the expensive simulation phase. A health endpoint could still answer while the machine moved toward exhaustion. Service health and workload capacity were different signals.
The verified rerun used one agent and one round. It moved from orchestration start to completion in about seven minutes and 39 seconds, then produced 12 graph nodes, 12 edges, 37 stored episodes, two profiles, an 8,366-character report, and four predictions. Those figures prove the path works at the measured configuration. They do not prove that the same host can safely run the default 4,096-agent, five-round template shown in the frontend.
I want that limitation stated plainly. Product defaults and operator-verified capacity are not yet the same thing. The queue stops simultaneous runs from multiplying pressure, but it cannot make one oversized run fit.
At higher loads, I would capture per-phase resident memory, container memory peaks, and runtime before changing either agent count or concurrency. The scaling decision needs a measured envelope. A second worker on separate hardware may double throughput. Two jobs inside one memory boundary may only double the chance of another kill.
Retries are bounded, not magical
Three attempts sound like recovery. They are useful for temporary Redis, database, or upstream connection failures. They do not make every orchestration step idempotent.
The gateway writes its simulation record before enqueueing, so a retry reuses that local ID. runSimulation() confirms that the record belongs to the same tenant and scenario. It also saves upstream project and simulation IDs as they become available. Those records give an operator evidence about where a run reached.
The current orchestrator still starts its remote sequence from the graph phase on a fresh BullMQ attempt. If a response was lost after MiroFish created remote state, a retry can create another upstream project. BullMQ provides at-least-once delivery. It does not provide exactly-once behavior across PostgreSQL, Redis, and MiroFish.
I considered making every step resumable before the first deployment. That would require remote idempotency keys or a local execution ledger with step-level reconciliation. The codebase had no evidence that MiroFish accepts a client-supplied key for each state-changing action. Building resume logic on remote IDs alone could be worse than a clean retry because the gateway might continue a project whose actual state it cannot verify.
Bounded duplicate risk remains during ambiguous network failures. The service records remote IDs and errors, caps attempts, and raises a permanent-failure signal. A human can inspect the upstream state before replaying a dead job. I would add a step ledger once run frequency makes manual reconciliation a recurring operation.
Cancellation exposes the next orchestration gap
The API has a cancellation route. It verifies tenant ownership, rejects simulations already in a terminal state, and marks an active record as cancelled with a completion timestamp. That protects the state transition visible to the caller.
It does not yet interrupt a running MiroFish operation. The BullMQ processor calls runSimulation() and waits. The orchestrator does not poll the local simulation row for a cancellation flag between remote phases. A user can cancel the record while remote work continues, and the worker may later try to write another status.
I would not hide that gap behind the word cancellation. Today the route cancels the gateway's declared intent. It is not a remote kill switch.
Cooperative cancellation belongs at phase boundaries. The orchestrator could check the tenant-scoped row before graph polling, simulation start, report generation, and final commit. BullMQ job removal could stop work that has not begun. A true mid-simulation stop depends on MiroFish offering a supported termination operation.
Response time is the cost. Checking local state adds database reads to a long workflow, while checking too rarely makes cancellation feel false. Phase-boundary checks are the sensible first step because they prevent new expensive work without pretending an in-flight remote call can be recalled.
A crash-recovery test has a precise claim
The repository includes a BullMQ crash-recovery test that checks the contracts surrounding Redis connection parsing, self-contained job data, retry options, worker event handlers, and shutdown ordering. It does not kill a live process in the middle of a real MiroFish run.
That boundary matters. The test proves the code has the pieces BullMQ needs to recover a persisted job. It cannot prove how the full stack behaves after the operating system terminates the worker between a remote side effect and a local status update. Calling it a process-kill test would overstate the evidence.
The service shutdown path is still useful. On SIGTERM or SIGINT, it stops HTTP intake and scheduled polling, closes cleanup work, asks the BullMQ worker to close, then disconnects the queue, Redis, database, and telemetry. worker.close() lets active work drain during a normal shutdown. A forced kill remains a different event.
I would test that event with a real child process and real Redis before claiming automatic crash recovery. The test would enqueue a deterministic long job, kill the worker after a known checkpoint, start a replacement, and verify both job state and external side effects. MiroFish makes the last part expensive, which is exactly why the claim should wait for the test.
One slot can still be an architectural choice
The queue gives the gateway durable admission, retries, backoff, failure events, and a clean HTTP boundary. Concurrency one gives the host a chance to finish the work it accepts. They solve different problems.
This design will reach a limit. A growing queue can make results arrive too late even when every job succeeds. The answer then is capacity isolation: dedicated worker hosts, resource limits, and scheduling based on measured run size. The API and job contract can remain while execution moves outward.
For the current deployment, the most important throughput metric is not jobs started per minute. It is simulations completed without the operating system killing the process.
