At a certain point, data migration stops being just about moving records from one place to another. On paper, simplicity sounds clean, but once you are dealing with large datasets, it can quickly spin out of control. You begin to struggle with fetching safely, processing reliably, recovering from failure, and resuming without corrupting data.
This was the challenge in a wallet log migration I worked on while moving ledger operations into a dedicated core-banking service. The destination system needed those logs represented as transaction records, with the correct account mappings, client associations, balances, timestamps, metadata, and deterministic references. The migration had to be retry-safe, recover cleanly from failure, and remain practical under real operational constraints. The existing implementation handled everything in one continuous flow: fetch cursor-paginated records, map them inline, and write directly into the destination database. That might work at smaller scale, but not in a distributed pipeline moving over 3 million records. I had to redesign it into a migration pipeline that could handle real load, recover cleanly, and run with much more control.
For the purpose of this article, we will refer to the source service as the core-service and the destination service as the core-banking-service.
The Original Approach And Why It Would Fail
The original implementation was a single job in the core-banking-service that owned the entire lifecycle. It fetched cursor-paginated API pages from the core-service, transformed the records for each page, and upserted them directly into the respective table. It fetched 100 records per page and specified a limit of 500 page fetches. The migration stopped at the page safety limit instead of handing off to a continuation job and recovery depended mostly on Laravel retries, not persisted checkpoints.
Even with the retries and page limits, it still coupled network IO, CPU-heavy mapping, and DB writes in one control loop. That coupling created an operational bottleneck. As volume increased, a single failure domain controlled throughput and recovery.
For 3 million records, 500 pages only covers about 50,000 records. Even if, by some incredible stroke of luck, the timeout didn’t kill the job first, the design had no real continuation model for the remaining data. The design was workable for smaller runs, but too rigid for multi-million-row imports.
The Real Risks: Memory Exhaustion, Weak Recovery, Concurrency
The biggest risk was that the design would fail gradually under load.
A single job was responsible for thousands of HTTP requests, repeated transformations, database lookups, and upserts. That increased the risk of timeouts, memory pressure, and weak recovery if the job failed halfway through. Without strong checkpoints, it would be difficult to tell what had already been fetched, what had been written successfully, and what still needed to be replayed.
There was also a concurrency risk once processing became parallel. Multiple workers upserting into the same transactions table could contend on indexes or hit deadlocks. That meant the pipeline had to keep chunk sizes small, preserve deterministic write order, and handle deadlock retries without becoming brittle.
The Two-Phase Redesign
The redesign split the migration pipeline into two separate phases: fetch first, process later.
That distinction mattered because the two parts of the work had different constraints. Fetching from the core-service had to follow the API’s cursor pagination, so it was naturally sequential. But once the records were inside our database, processing no longer depended on the source API. At that point, the work could be divided into chunks and handled by multiple queue workers.
So instead of one job fetching, transforming, and writing everything directly into the DB, the new flow became:
- Fetch wallet logs from the
core-serviceand stage the raw payloads. - Read staged records, transform them, and upsert them into the DB in parallel.
This ensured that if processing failed, I did not need to go back to the core-service for the same data. If a worker died, the staged records were still there. If the migration had to continue later, the cursor and segment state gave the pipeline a place to resume from.
A key part of the redesign was the idea of a segment. A segment is a bounded subset of the import: fetch up to a fixed number of pages, stage those records, process them in parallel running jobs, clean them up, then continue from the next cursor if more data exists. This gave the pipeline a repeatable cycle: fetch a segment, process a segment, move to the next segment.
Phase 1: Fetch And Stage Raw Records
The first phase is responsible for getting data safely out of the core-service. When the migration starts, an artisan command creates a new session ID and dispatches a FetchJob. That job calls the wallet log export endpoint page by page, using the cursor returned by the API. Each page contains a limited number of records, and those records are written as raw JSON into a staging table. Each segment limits how much data a single fetch cycle stages before the pipeline moves forward.
Each staged row is tied to a session and segment so the import can process records in bounded batches instead of one unbroken run. The fetch phase also records progress in a checkpoint table, including the active segment, cursor, page, and status. This gave the migration a checkpoint to resume from after a timeout, failure, or retry.
Phase 2: Transform And Upsert In Parallel
Once a segment has been staged, the second phase begins. A BatchJob loads the staged rows for that session and segment, splits them into small ID-based chunks, and dispatches chunk jobs using Laravel’s batch system. Each chunk job reads only its assigned staged records, builds the needed account and client lookup maps, transforms the wallet log payloads into transaction rows, and upserts them into the transactions table.
The upsert is designed to be retry-safe. Each transaction gets a deterministic reference based on stable source fields, so if a chunk is retried, it updates the same transaction instead of creating a duplicate. Because multiple workers can write to the same table at once, the upsert path also includes deadlock retry handling.
Once all chunk jobs in a segment complete successfully, the staging rows for that segment can be deleted. If the source API returns another cursor, the next fetch segment (phase 1) starts. Otherwise, the import session is marked as completed.
Why Staging Tables Mattered
The staging table was the buffer between core-service and the core-banking-service.
In the original approach, each API response was fetched, transformed, and written in one pass. That meant fetching and processing were tightly coupled. If the job failed halfway through, it was difficult to separate what had been fetched from what had been successfully written.
Instead of treating the API response as temporary in-memory data, the import first persisted each raw wallet log payload into a staging table. That gave the pipeline a durable checkpoint between the source system and the final table. If processing failed, the records for that segment were still available locally and could be retried without starting from scratch.
Staging tables also helped with deduplication. Each raw record received a fingerprint, and the table enforced uniqueness within a session and segment. Retries could ignore records that had already been staged for the same session and segment.
The staging table gave the pipeline a safer rhythm: fetch a bounded segment, store it, process it, then clean it up. The table was not meant to permanently archive the import. It was a short-lived buffer that made the migration recoverable.
How Batching Changed The Game
While staging tables made the import recoverable, batching made it scalable.
The core-service endpoint used cursor pagination, so fetching had to remain sequential. The core-banking-service could not safely request multiple pages at the same time because the API did not expose independent ranges for a full export.
But once the records were staged locally, that limitation no longer existed. Each segment now had a fixed set of rows that could be split into chunks and processed independently.
That is where Laravel batch jobs changed everything.
Instead of one job transforming and upserting a huge dataset, the BatchJob created many smaller chunk jobs. Each handled a small set of staged row IDs, making the workload easier for queue workers to process, retry, and monitor.
It also improved throughput. More queue workers did not make the source API faster, but it did speed up the transformation and database writes. The migration was no longer limited by a single PHP process doing all the work.
There was a tradeoff: parallel writes introduced concurrency risks. Multiple workers upserting into the same table could contend on indexes or hit deadlocks. To handle that, chunk sizes were kept small, rows were ordered before writes, and deadlock retries were added around the upsert operation.
The result was a better balance: fetching remained controlled and sequential, while processing became parallel and bounded. What had been a fragile long-running job became a queue-driven workload that could be scaled, retried, and monitored.
What Happened
The redesigned pipeline made the migration practical to run across environments.
In development, the goal was correctness. I had to confirm wallet log payloads mapped correctly to the new transaction schema, account and client lookups resolved properly, and reruns did not create duplicates.
In staging, the focus shifted to behaviour under load. Fetch jobs had to stay within timeout limits, processing jobs had to run in manageable chunks, and failures had to be retryable without losing progress.
By the time the migration ran on the production-like environment, the pipeline had become a sequence of bounded fetch segments and parallel processing batches. Each segment could be fetched, staged, processed, cleaned up, and resumed from the next cursor.
The result was a migration pipeline that could handle millions of wallet log records with much better control over retries, failures, and throughput.
- Total records imported: ~4 million records across all targeted environments
- Environment import runs: Development (~33k), staging (~1 million), production-like (~3 million)
- Failed/retried jobs: Some chunk jobs were retried during execution, but the pipeline completed without requiring a restart from scratch
- Final outcome: Completed successfully across all targeted environments
Tradeoffs and Improvements
The two-phase design worked well. Separating fetch from processing made the pipeline easier to reason about and easier to recover, while queue batches distributed the heavy write phase across workers.
The idempotency model also proved effective in preventing duplicate records. Deterministic transaction references and raw-record fingerprinting made retries much safer, which is critical when moving financial records.
One of the main weaknesses was visibility. The progress table worked as a checkpoint, but not as a full audit trail. A better version would include import history for each segment: staged, processed, skipped, retried, and cleaned up.
I would also add performance metrics earlier. Fetch time per page, processing time per chunk, deadlock frequency, and records processed per minute would have made tuning easier across environments.
Finally, cursor pagination kept fetching correct but limited parallelism. If this migration pattern became recurring, I would consider extending the source API to support safer parallel fetching.
Conclusion
The biggest lesson from this migration was that scale changes the nature of the problem. Once volume increases, the challenge is no longer just moving data. It is control: how to pause, retry, resume, and verify progress without corrupting financial data. What starts as a straightforward fetch-and-store job can quickly become fragile once the dataset grows, the source is another service, and retries have to be safe.
The two-phase pipeline worked because it respected those realities. Fetching stayed controlled, processing became parallel, and failure no longer meant starting over from scratch.
The important shift was not just technical but a change in mindset. I stopped trying to make one big job survive and started designing a pipeline that expected failure, recovered from it, and kept moving. At that point, the work was no longer just about migrating wallet log records. It was about building a migration pipeline that could handle real load and still be trusted.
