Why we check the database before serving traffic: schema updates at schukai

Imagine deploying a new version of your application. The database migration finishes successfully. Then the application fails before it can serve traffic.

You still have the previous binary, so switching back seems like the obvious recovery step.

But the database has already changed. Can the previous version safely use it?

That question shaped how we handle schema updates. At schukai, we track the structure that has been applied, the application versions allowed to use it, and the evidence behind that compatibility decision.

Here is how that works, and why we keep those concerns separate.

A migration number only answers part of the question

A schema version tells us how far a database has progressed through its structural changes. It does not explain whether an older application can still read and write safely.

Consider two hypothetical updates.

One adds a nullable column that the previous application never touches. Another tightens a constraint in a way that rejects writes the previous application still produces.

Both advance the schema. Their recovery options differ considerably.

We therefore maintain separate values for:

The schema contract and the migration baseline have independent version numbers. We also record the application version for diagnosis, but it cannot establish database compatibility on its own.

This gives the runtime enough information to make a concrete decision before accepting normal traffic.

Our models describe the structure, with explicit rules where needed

schukai Workspace uses Go models and GORM as part of its schema definition and migration process.

For ordinary structural work, that gives us a useful connection between application models and database tables. We supplement it with explicit SQL and dedicated ensure functions for database contracts that need more precise handling, including particular indexes, constraints, and triggers.

The migration path brings those pieces together.

This matters because successfully running an ORM migration does not prove that every database requirement has been satisfied. We need to check the resulting structure as well.

Startup is where we enforce compatibility

When schukai Workspace starts, it reads the persisted schema state and checks whether the current binary supports it.

A database may be too old for the available upgrade path. It may have been advanced by a newer runtime. Its compatibility metadata may be incomplete or inconsistent.

Those conditions need an explicit outcome before business requests reach the application.

The startup path is roughly:

Read persisted schema state
            |
Check runtime compatibility
            |
Apply supported structural changes if needed
            |
Verify structure and database contracts
            |
Continue toward normal application readiness

Schema verification is one requirement for readiness. Product release checks remain separate: a structurally compatible database does not establish that every application update step has completed.

When blocking schema drift is detected, the server stays in a maintenance state. It can report that the process is alive while readiness remains false.

That distinction helps deployment infrastructure avoid sending business traffic to a process that has started but cannot safely serve it yet.

We check what is actually in the database

Suppose the metadata says the database is at the expected baseline.

That is useful, but we still inspect the database.

Our drift checks compare the expected model structure with the actual tables, column types, nullability, and primary keys. Additional checks cover contracts such as important indexes and triggers.

This catches a different category of problem from migration version tracking.

A baseline records an applied transition. A drift check examines the structure that exists now. If someone changes a column manually, the baseline number alone cannot reveal that difference.

We use both because they answer different operational questions.

The same reasoning applies after migration: we verify the result before treating the database as ready.

Every transition declares its recovery boundary

Our schema transitions use two compatibility classes.

compatible_expand

This class covers an additive transition whose compatibility with the declared previous runtime has been tested.

The physical schema baseline advances, while the minimum runtime baseline can remain at the approved older value.

That leaves a defined recovery option: the older runtime may still serve against the updated database.

Even an apparently harmless addition needs that evidence. For example, an added column can affect older writers if its constraints introduce requirements they cannot satisfy.

forward_only

This class covers a transition after which an incompatible older runtime must no longer serve.

Removing a structure that old code uses, renaming it, or tightening a constraint on an existing write path can create this boundary.

Once that transition commits, the recovery options are the target runtime, a compatible forward fix, or a complete restore.

Switching the binary back does not reverse the database change.

We persist the transition class, the minimum compatible runtime baseline, and a digest of the transition manifest. The compatibility decision belongs to the build’s transition contract; operators cannot redefine it through a configuration flag.

The migration and its success record must commit together

There is an uncomfortable failure window in any migration system: the structure changes, but recording success fails.

On the next attempt, the system must work out what happened.

For the automatic PostgreSQL migration path, we keep supported structural changes and their state updates in the same transaction. The applied baseline, run evidence, and applicable point-of-no-return information commit with the structural change.

If the transaction fails, it must not leave behind a record claiming that the target baseline was successfully applied.

Nontransactional DDL falls outside this standard path. It requires its own recovery and checkpoint contract.

That restriction limits what can happen automatically during an update. We accept it because an automated path needs predictable failure behavior.

Creating tables and repairing historical data are separate jobs

A new database and a database with years of business records need different treatment.

We separate structural migration from bootstrap and tenant initialization. We also keep explicit legacy data upgrades outside the regular startup path.

Creating a table has a technical definition. Deciding how to populate missing business values in historical records may require a business decision, backup evidence, and a reviewable plan.

Putting that work into an ordinary startup migration would hide those decisions inside a deployment.

Our tooling reflects the separation. A running server exposes schema drift inspection through nucli. Privileged local offline checks and supported maintenance procedures belong to numin, including schema status and migration preflight.

The preflight examines pending transitions without changing the database. It gives us a chance to understand blockers and recovery implications before starting the update.

What we want to know before deploying

This design adds metadata, verification work, and compatibility tests. It also makes some updates explicitly forward only, which affects how we schedule and recover them.

In exchange, we can ask more useful questions before deployment:

• Which structure will this update apply?
• Can the previous runtime still use it?
• What evidence supports that decision?
• At what point does recovery require a forward fix or restore?

For schukai Workspace, those answers belong in the update contract before the migration runs. When an application fails after a structural change, we want the recovery boundary to be something we already checked.