# Backstage 1.56 Requires a Deliberate Catalog Database Cutover

- Link: https://thestackobserver.com/backstage-1-56-requires-a-deliberate-catalog-database-cutover/
- Published: 2026-11-02T10:58:18-05:00
- Author: Stackxx

Backstage 1.56 arrives with a catalog optimization that is easy to misread as an
ordinary dependency upgrade. The incremental-ingestion module now writes ingestion
marks with a native database upsert, reducing round trips and improving the path
for large catalogs. To enable that change, however, the release adds a `source_key`
column, creates a unique constraint, and renames `ref` to `entity_ref` in the `ingestion_mark_entities`
table. Backstage’s own release notes are explicit: this migration cannot be applied
during a rolling, zero-downtime upgrade.

That constraint is the operational headline. Platform teams should treat the upgrade
as a coordinated database cutover, not let each pod migrate opportunistically while
old and new application versions overlap. The right objective is not merely to install
Backstage 1.56. It is to preserve catalog integrity, bound the interruption, and
prove that ingestion resumes correctly after the schema and application move together.

## Why this migration breaks the normal rolling-upgrade contract

A rolling deployment assumes adjacent application versions can safely share the 
same database for a period. New pods start, old pods continue serving traffic, and
the load balancer gradually shifts work. For that to be safe, schema changes normally
need an expand-and-contract shape: first add structures understood by the new code
without invalidating the old code, deploy compatible application versions, backfill
if necessary, and remove obsolete structures later.

The incremental-ingestion change in Backstage 1.56 does not offer that compatibility
window. Renaming the column changes the schema expected by the module, while the
new source key and uniqueness rule alter how ingestion marks are identified and 
written. If old and new backends run together, they do not share one stable contract.
An old process may still address `ref` while a new process expects `entity_ref`;
allowing either version to race through migrations or writes creates failure modes
that a deployment controller cannot resolve.

This does not make the release unsafe. It makes the upgrade procedure different.
The optimization replaces a more chatty write path with a single native upsert, 
which can matter when catalogs ingest many providers or large entity sets. But the
performance benefit is purchased with a one-time coordination requirement. Platform
operators must make that requirement visible in the change plan.

## First determine whether the breaking path applies

The warning is specifically tied to `@backstage/plugin-catalog-backend-module-incremental-
ingestion` and its marks table. Do not infer that every Backstage 1.56 deployment
requires downtime. Before scheduling a cutover, establish whether the module is 
installed, enabled in the deployed backend, and backed by a database that contains
the affected table.

Use evidence from the running system rather than relying only on a lockfile. A package
can exist in a monorepo without being loaded in production, and deployment configuration
may differ by environment. The change record should capture all of the following:

 * The deployed backend image digest and current Backstage package versions.
 * The module registration path in backend code and the environments where it is
   active.
 * The database engine, schema owner, migration permissions, and approximate row
   count for `ingestion_mark_entities`.
 * The number of backend replicas and every process that can run catalog migrations
   or ingestion work.
 * The entity providers that use incremental ingestion and the expected duration
   of a full reconciliation.

If the module is absent, teams can follow their usual Backstage upgrade process 
while still testing the release’s other breaking changes. If it is present, the 
safe default is a maintenance window with a complete stop of writers before migration.
That decision should be made before a deployment manifest is merged, not while a
rollout is already in progress.

## Build the cutover around one schema owner

The core control is simple: only one deliberate migration action should own the 
schema transition. Do not let every application replica start with permission to
race for it. In Kubernetes, that can mean a dedicated Job using the same immutable
image as the new backend, with application Deployments scaled to zero. In another
environment, it may be a separately invoked migration command in the release pipeline.
The mechanism matters less than the guarantees: old writers are stopped, only one
migration actor runs, and new writers begin only after the schema has been verified.

### Prepare a recovery point

Take a database backup or a provider-supported point-in-time recovery marker immediately
before the change. Confirm that it covers the relevant Backstage database and that
operators know the restore procedure. A backup is not a rollback plan by itself;
record expected restore time, data-loss window, and who has authority to initiate
recovery. If the portal is business-critical, rehearse the restore in a non-production
environment with a copy of representative catalog data.

### Drain work, then stop all old backends

Announce a short portal maintenance period or route users to a clear maintenance
response. Stop scheduled ingestion triggers if they run outside the backend. Scale
the Backstage backend to zero and verify that no old process, worker, or ad hoc 
job retains database access. Merely removing pods from the load balancer is insufficient:
a process that is not receiving HTTP traffic can still perform background ingestion.

### Run and inspect the migration

Execute the migration once with the target 1.56 image and capture its exit status
and structured logs. After it succeeds, inspect the table rather than assuming success
from process completion alone. Confirm that `source_key` and `entity_ref` exist,
the obsolete column is gone as expected, and the intended uniqueness constraint 
is present. Compare row counts with the pre-change baseline and inspect for nulls
or duplicates that would undermine the upsert key.

### Start one canary before restoring the fleet

Bring up a single new backend instance and keep the remaining replicas stopped. 
Validate startup migrations, database connectivity, catalog reads, provider activity,
and ingestion-mark writes. Trigger a controlled provider refresh if the environment
allows it. Only after the canary proves that entity updates progress should the 
deployment scale to normal capacity. This sequence narrows the source of any failure:
schema, application startup, or multi-replica behavior.

## Test catalog behavior, not just pod health

A green readiness endpoint proves that the process can answer its configured probe.
It does not prove that incremental ingestion is correct. The verification plan should
follow one or more known entities from provider input to catalog output. Choose 
a small sample that includes an update, a new entity, and, where safe, a deletion
or orphaning path. Record entity references and timestamps before the cutover so
the test is repeatable.

After the canary starts, verify that provider runs complete, ingestion marks advance,
and changed entities appear in the catalog within the expected service-level objective.
Watch catalog processing errors, database constraint violations, provider mutation
duration, queue or backlog signals, and the age of the last successful refresh. 
Compare these with the baseline rather than relying on an absence of alerts.

Backstage 1.56 also includes catalog reliability improvements: stitching work is
less likely to be lost across deletion and re-addition, successful entities should
no longer remain unavailable after interrupted scheduling, and final entities plus
search-index rows are written in one transaction. Those improvements strengthen 
the post-upgrade system, but they do not remove the need to validate the ingestion
migration itself. Treat them as expected benefits to observe after the cutover, 
not as substitutes for checks.

## Rollback means restoring code and schema together

A routine application rollback replaces the new image with the old image. That is
unsafe after an incompatible schema migration because the old version may not understand
the renamed column and new key structure. The rollback decision therefore has two
branches.

 * **Before the migration begins:** aborting is straightforward. Keep or restore
   the old application version because the database contract has not changed.
 * **After the migration commits:** do not simply redeploy old pods. Either fix 
   forward with the 1.56-compatible application or restore the pre-migration database
   recovery point and then restore the old application version.

Define the decision threshold in advance. Examples include a migration that does
not complete inside the maintenance budget, failed canary startup, constraint errors
during provider refresh, or a growing catalog-processing backlog. Also define the
point of no return for new catalog writes. If users or providers can mutate data
after the upgrade, restoring an earlier database snapshot can discard those changes.
Keeping the portal in maintenance mode through canary validation makes rollback 
less ambiguous.

## Use the disruption to improve future platform upgrades

This release exposes a broader platform-engineering lesson: version automation cannot
determine rollout safety from semantic version numbers alone. Backstage releases
aggregate many packages and plugins, while an organization’s portal adds local modules,
database topology, and operational policy. A successful dependency update is not
proof that two adjacent runtime versions can coexist.

Add a migration-compatibility checkpoint to the portal’s release template. Require
the upgrade owner to classify every database change as backward-compatible, staged
expand-and-contract, or coordinated cutover. Make CI produce the migration plan 
for review, but keep production execution as a separately observable step. Record
which image owns the migration and prevent the application service account from 
making schema changes if your deployment model supports that separation.

The same template should require a data-path canary. For a developer portal, that
means catalog ingestion, template execution, authentication, and search as appropriate—
not only HTTP availability. Over time, these checks become reusable release evidence
and shorten maintenance windows because operators no longer improvise verification.

## The practical decision

Teams using Backstage’s incremental-ingestion module should upgrade to 1.56 through
a brief, planned stop-the-world cutover: inventory the affected module, establish
a recovery point, halt every old writer, run the migration once, inspect the resulting
schema, and validate ingestion through a single canary before scaling out. Teams
not using the module should document that finding and proceed with normal compatibility
testing.

The important distinction is between availability as a goal and rolling deployment
as a technique. For this migration, insisting on a rolling rollout would increase
risk without preserving a valid shared database contract. A bounded maintenance 
window, supported by rehearsed recovery and catalog-level verification, is the more
reliable path—and reliability is the outcome a platform team is supposed to optimize.

## Sources

 * [Backstage 1.56.0 release notes](https://github.com/backstage/backstage/releases/tag/v1.56.0)
 * [Backstage pull request #35177: incremental ingestion mark table upsert](https://github.com/backstage/backstage/pull/35177)