createSimulation() returned an ID. startSimulation() still failed.
That was the sixth version of the gateway's live MiroFish path. Five earlier runs had already found a wrong Redis instance, a duplicate simulation record, a synchronous ontology response mistaken for an asynchronous task, a project ID nested under data, and graph status nested under the same wrapper. The new ID looked like progress. MiroFish answered that the simulation had not been prepared.
The seventh run added preparation and exposed one final missing operation: report generation had to be triggered explicitly after the swarm stopped. The eighth run completed the whole route.
None of those failures required a clever algorithm. They required the client to admit that the upstream API was a state machine, even though its surface looked like a collection of ordinary HTTP calls.
A typed client can still encode the wrong lifecycle
I started with the reasonable model: upload context, ask MiroFish to build a graph, start a simulation, then fetch its report. Each action had a client method. Each response could be assigned a TypeScript type. The call sequence was still wrong.
The first clue was ontology generation. I expected a task ID and wrote polling logic around it. The live service returned the ontology synchronously and placed the project ID inside a { data, success } envelope. Graph construction did return a task ID and required polling. Two adjacent operations used different completion models.
Then the graph-status response used another nested payload. The initial poller read status at the top level, so it kept waiting even after the service had completed the job. That is a dangerous integration failure because nothing crashes. The worker appears busy until its timeout expires.
Simulation startup had a deeper sequence. The project ID from graph construction did not double as a simulation ID. The gateway had to create a simulation, retain the returned simulation ID, prepare profiles and configuration, wait for preparation, then start the run. A report was another asynchronous job with its own task ID and poll cycle.
TypeScript helped once those facts were known. It could not discover them from an API that returned HTTP success for an operation whose downstream state was not ready. Live execution supplied the missing contract.
The response normalizers now stay close to the operations they describe. Profile payloads, action logs, graph data, task status, and report state do not pass through one catch-all decoder. Each normalizer accepts the wrapper shapes observed for that operation and rejects data that cannot produce its domain type.
That creates more small functions than one generic unwrapData() helper. The duplication is intentional. A universal unwrapping function would tell the caller that a payload exists, while concealing whether it contains project_id, simulation_id, task_id, or runner_status. The client needs those distinctions to decide what can happen next.
The live failures also changed the test boundary. Unit tests cover the client methods and poll behavior, but recorded response shapes are only hypotheses until a real MiroFish checkout answers. The launch run became a contract test for the whole lifecycle. I would keep a small fixture set from verified responses so future upstream updates fail at the decoder rather than halfway through a simulation.
I turned the call list into phases
The orchestrator records graph_building, simulating, reporting, completed, and failed states in PostgreSQL. Those states are more than UI labels. They mark ownership transitions between the gateway and MiroFish.
During graph building, the gateway generates a seed document from the tenant's scenario, asks for an ontology, starts graph construction, polls the graph task when one exists, fetches graph data, and mirrors it locally. During simulation, it creates and prepares the upstream run before asking the swarm to start. During reporting, it triggers report generation, waits for that task, fetches the report, parses predictions, and commits the terminal result.
This is the actual center of the simulation phase:
const createResult = await mirofishClient.createSimulation(mirofishProjectId);
const mirofishSimId = createResult.data.simulation_id;
await updateSimulationStatus(
simulationId,
tenantId,
SIMULATION_STATUS.SIMULATING,
{ mirofishSimId },
);
await mirofishClient.prepareSimulation(mirofishSimId);
await mirofishClient.pollPrepareStatus(mirofishSimId, GRAPH_TIMEOUT_MS);
await mirofishClient.startSimulation(mirofishSimId, config);
await mirofishClient.pollSimulationStatus(
mirofishSimId,
SIMULATION_TIMEOUT_MS,
);
Every method in that sample exists in MirofishClient. The order is fixed because each call creates state required by the next one. Saving mirofishSimId before preparation gives the gateway a recovery handle if the process fails after the upstream simulation has been created.
I separated graph and simulation timeouts as well. Graph-related polls allow 600,000 milliseconds. A simulation can run for 1,800,000 milliseconds. Using one generic timeout would either abandon legitimate simulations too early or leave a stuck graph task alive for far too long.
The trade-off is explicit coupling. The orchestration code knows more about MiroFish than a generic HTTP adapter would. I accepted that because hiding the lifecycle behind a vague execute() method would make recovery impossible. The gateway needs to know which remote ID exists, which phase owns it, and whether repeating an operation is safe.
Retry only the failures that have not produced remote state
The HTTP layer retries connection failures such as refusal, timeout, DNS failure, and reset. It makes at most three attempts with exponential delay starting at one second. It does not retry every non-success HTTP status.
That boundary is deliberate. Repeating a connection attempt is different from repeating a state-changing request. If the network drops after MiroFish creates a project but before the gateway receives the response, an automatic retry could create a second project. The client cannot infer idempotency from an error code. Broad retries would trade a visible failure for hidden duplicate state.
Polling has its own rules. It accepts complete and completed, understands that simulation state may live under runner_status, and throws when the upstream task reports failure. Delay and timeout are explicit inputs. A graph task and a swarm run share polling mechanics without sharing assumptions about duration or response shape.
I was wrong about how much could be normalized at the HTTP boundary. The { data, success } wrapper looked like the obvious common shape, but completion data still differed by operation. Normalizing every response into one universal result type would erase distinctions the orchestrator needs. The final client keeps operation-specific return types and puts only transport behavior in the shared request helper.
Not every missing artifact should fail the run
Once preparation completes, the gateway tries to fetch agent profiles. After simulation, it tries to fetch action logs. Both are valuable because they power stance summaries and local episode memory. Neither is the report itself.
Those enrichment calls sit inside narrow try blocks. A profile-fetch failure logs a warning and lets the run continue. The same rule applies to action logs. Report generation is different. If the report cannot be triggered, polled, or fetched, the simulation cannot be marked complete because the product has no deliverable.
This distinction prevented a small remote-data mismatch from destroying an otherwise useful run. It also prevents the opposite lie: a simulation with no report cannot pass merely because agent profiles were saved.
The fallback path for action logs shows why this is necessary. MiroFish may expose the log through its API, but the gateway also knows how to read the upstream file output when the API shape is unavailable. Malformed lines are skipped and counted rather than crashing the whole import. The importer hashes an action's simulation identity, agent, round, type, and content into a source key, so retrying the import does not duplicate episodes.
That fallback is useful, but it is also coupling to an upstream implementation detail. I would remove it when the API provides a stable action-log contract across releases. Until then, the gateway records the gap instead of pretending the integration is cleaner than it is.
Failure status must not replace the original failure
The most subtle recovery bug sat in the catch block. When orchestration fails, the gateway updates the simulation row to failed, stores the error, and emits a failure event. But the database update can fail too.
An earlier shape risked replacing the original MiroFish error with the secondary persistence error. That would leave the operator debugging the cleanup step rather than the operation that broke the run. The current code wraps failSimulation() in its own try, logs both errors when persistence fails, and rethrows the original exception.
That ordering is an operational contract. Diagnostic cleanup should add evidence. It should not rewrite history.
Completion events follow a related rule. The gateway marks the report and completion time in PostgreSQL before it publishes simulation.completed. Notification delivery is feature-flagged and best effort. An unavailable notification service cannot retroactively turn a finished swarm run into a failed one. The database remains the authority for simulation outcome.
An event consumer may miss a completion notification during an outage. The service keeps that failure visible in logs, and the simulation can still be recovered from the API. Making the event transactionally guaranteed would require an outbox and a delivery worker. I would add that when downstream automation depends on every event, not merely when notifications are useful.
Eight attempts produced one explicit contract
The final live path generated an ontology, built and mirrored a graph, prepared profiles, ran the swarm, imported action logs, generated a report, and parsed four predictions. Its importance is not the happy-path result. The eight attempts left a client that names each remote phase and a database record that tells an operator where work stopped.
I would still change one part at higher volume. Remote IDs and phase timestamps belong in a dedicated execution-attempt table once the same simulation can be replayed or resumed more than once. The current simulation row is enough for one active attempt and bounded retries. It would become ambiguous if several upstream runs competed for the same logical simulation.
For now, the gateway refuses that ambiguity. One simulation owns one scenario, one upstream project, one upstream simulation ID, and one terminal report. The client does not disguise MiroFish as a single call.
An external API is easiest to operate when the code records the states its documentation leaves between the lines.
