I was asked to look at a fintech backend where something small but persistent kept happening. Every month when the finance team tried to close the books, the internal ledger and the payment processor's own records were off by a small amount. Not a dramatic amount, sometimes a few transactions, sometimes a few dollars here and there, but enough that nobody could confidently say the numbers were correct. And in a fintech, correct is not optional.
This is called reconciliation drift, and it is one of the quieter problems a payments backend can have. It does not throw an error. It does not crash anything. It just slowly stops being trustworthy, one small mismatch at a time, until someone finally notices during a monthly close and has to spend hours figuring out where things went wrong.
What was actually happening
Once I started digging, the cause was not one single bug. It was a pattern that shows up a lot in systems that grow organically over time, multiple different places in the code were allowed to change a transaction's status or a balance, and none of them were talking to each other.
A webhook handler updated a transaction when the payment provider confirmed it. A retry job also updated the same transaction if the original webhook was ever missed. And an admin tool, built later for handling support tickets, allowed a staff member to manually mark a transaction as settled if a customer complained. Three different code paths, three different points where the same number could be changed, and no single place that recorded why a change happened or which of these three paths actually caused it.
@Injectable()
export class TransactionsService {
async markSettled(transactionId: string) {
const transaction = await this.transactionRepo.findOne({
where: { id: transactionId },
});
transaction.status = 'settled';
await this.transactionRepo.save(transaction);
}
}
This method looked fine everywhere it was called. The problem was that it was called from three different places, and the transaction record itself kept no history of that. Once a transaction was marked settled, there was no way to tell whether it had happened once, correctly, or twice, incorrectly.
The fix, treat every change as an event, not an overwrite
The real fix was not patching each of those three code paths individually. It was changing how transactions record their own history in the first place. Instead of updating a transaction's status directly, every change becomes its own recorded entry, and the transaction's current state is something you calculate from that history, not something you overwrite.
@Entity()
export class TransactionEvent {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
transactionId: string;
@Column()
eventType: string;
@Column()
source: string;
@CreateDateColumn()
createdAt: Date;
}
@Injectable()
export class TransactionsService {
constructor(
private readonly eventRepo: Repository<TransactionEvent>,
) {}
async recordSettlement(transactionId: string, source: string) {
await this.eventRepo.save({
transactionId,
eventType: 'settled',
source,
});
}
async getCurrentStatus(transactionId: string): Promise<string> {
const events = await this.eventRepo.find({
where: { transactionId },
order: { createdAt: 'ASC' },
});
const latest = events[events.length - 1];
return latest ? latest.eventType : 'pending';
}
}
The source field is what makes debugging this kind of issue possible going forward. Once every change records where it came from, the webhook, the retry job, or the admin tool, you can immediately see if the same transaction was settled twice by two different paths, instead of just seeing a status that quietly changed with no explanation attached.
The fix, an automated job that actually compares your records against theirs
Recording history properly stops new mistakes from being invisible, but it does not catch drift that already exists, or drift caused by something entirely outside your own system, like a webhook that never arrived at all. For that, the backend needs to actively compare its own records against the payment provider's records on a regular schedule, rather than assuming they always agree.
NestJS's schedule module makes this straightforward to set up as a recurring job.
@Injectable()
export class ReconciliationService {
constructor(
private readonly transactionRepo: Repository<TransactionEvent>,
private readonly discrepancyRepo: Repository<ReconciliationDiscrepancy>,
private readonly providerClient: PaymentProviderClient,
) {}
@Cron('0 2 * * *')
async runDailyReconciliation() {
const providerRecords = await this.providerClient.getSettledTransactions();
const internalStatuses = await this.getAllInternalStatuses();
for (const record of providerRecords) {
const internalStatus = internalStatuses.get(record.transactionId);
if (internalStatus !== 'settled') {
await this.discrepancyRepo.save({
transactionId: record.transactionId,
issue: 'provider shows settled but internal record does not',
});
}
}
}
}
This job runs quietly every night, comparing what the provider says actually happened against what your own system believes happened. Anything that does not line up gets written down as a discrepancy, rather than staying hidden until someone stumbles onto it during a manual review weeks later.
@Entity()
export class ReconciliationDiscrepancy {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
transactionId: string;
@Column()
issue: string;
@Column({ default: false })
resolved: boolean;
@CreateDateColumn()
detectedAt: Date;
}
Keeping discrepancies in their own table, separate from transactions themselves, means the finance team has a single place to look every morning, a short list of exactly what needs attention, instead of having to compare two entire data sets by hand.
Making the discrepancies actually get noticed
A table full of unresolved mismatches is only useful if someone actually sees it. Once the discrepancy records existed, the next step was making sure the job did not just write quietly to a database that nobody checked.
@Injectable()
export class ReconciliationService {
constructor(
private readonly notificationService: NotificationService,
) {}
private async notifyIfDiscrepanciesFound(count: number) {
if (count > 0) {
await this.notificationService.alertFinanceTeam(
`Reconciliation found ${count} unresolved mismatches overnight`,
);
}
}
}
This is a small addition, but it is the difference between a system that quietly protects itself and one that quietly accumulates a growing pile of unresolved discrepancies nobody remembers to check.
The bigger picture
None of this required exotic tooling. A recorded event history instead of silent overwrites, a scheduled job that compares your truth against the provider's truth, and a clear place for discrepancies to land where someone will actually see them. NestJS made this easy to structure cleanly, a dedicated service for events, a dedicated service for reconciliation, and a built in scheduler that made the recurring job simple to wire up without reaching for a separate tool.
What this kind of review usually reveals is not one dramatic bug. It is a handful of small, reasonable seeming decisions made at different points in time, each one fine on its own, that quietly stopped agreeing with each other. Catching that early is a lot cheaper than discovering it during an audit.
If your team suspects your own numbers might not fully line up with what your payment provider believes happened, that gap is worth closing before it grows, and it is exactly the kind of review I would be glad to help with.
I am Peace Melodi, a backend software engineer. If you want your business to scale big, comfortably handling millions of users without breaking, with strong scalability and security in place, feel free to reach out.
LinkedIn: https://www.linkedin.com/in/melodi-peace-406494368
GitHub: https://github.com/PeaceMelodi
