Backstage 1.56 arrives with a catalog optimization that is easy to misread as an ordinary dependency upgrade. The incremental-ingestion module now writes ingestion marks with a native database upsert, reducing round trips and improving the path for large catalogs. To enable that change, however, the release adds a source_key column, creates a unique constraint, and renames ref to entity_ref in the ingestion_mark_entities table. Backstage’s own release notes are explicit: this migration cannot be applied during a rolling, zero-downtime upgrade.
That constraint is the operational headline. Platform teams should treat the upgrade as a coordinated database cutover, not let each pod migrate opportunistically while old and new application versions overlap. The right objective is not merely to install Backstage 1.56. It is to preserve catalog integrity, bound the interruption, and prove that ingestion resumes correctly after the schema and application move together.
Why this migration breaks the normal rolling-upgrade contract
A rolling deployment assumes adjacent application versions can safely share the same database for a period. New pods start, old pods continue serving traffic, and the load balancer gradually shifts work. For that to be safe, schema changes normally need an expand-and-contract shape: first add structures understood by the new code without invalidating the old code, deploy compatible application versions, backfill if necessary, and remove obsolete structures later.
The incremental-ingestion change in Backstage 1.56 does not offer that compatibility window. Renaming the column changes the schema expected by the module, while the new source key and uniqueness rule alter how ingestion marks are identified and written. If old and new backends run together, they do not share one stable contract. An old process may still address ref while a new process expects entity_ref; allowing either version to race through migrations or writes creates failure modes that a deployment controller cannot resolve.
This does not make the release unsafe. It makes the upgrade procedure different. The optimization replaces a more chatty write path with a single native upsert, which can matter when catalogs ingest many providers or large entity sets. But the performance benefit is purchased with a one-time coordination requirement. Platform operators must make that requirement visible in the change plan.
First determine whether the breaking path applies
The warning is specifically tied to @backstage/plugin-catalog-backend-module-incremental-ingestion and its marks table. Do not infer that every Backstage 1.56 deployment requires downtime. Before scheduling a cutover, establish whether the module is installed, enabled in the deployed backend, and backed by a database that contains the affected table.
Use evidence from the running system rather than relying only on a lockfile. A package can exist in a monorepo without being loaded in production, and deployment configuration may differ by environment. The change record should capture all of the following:
- The deployed backend image digest and current Backstage package versions.
- The module registration path in backend code and the environments where it is active.
- The database engine, schema owner, migration permissions, and approximate row count for
ingestion_mark_entities. - The number of backend replicas and every process that can run catalog migrations or ingestion work.
- The entity providers that use incremental ingestion and the expected duration of a full reconciliation.
If the module is absent, teams can follow their usual Backstage upgrade process while still testing the release’s other breaking changes. If it is present, the safe default is a maintenance window with a complete stop of writers before migration. That decision should be made before a deployment manifest is merged, not while a rollout is already in progress.
Build the cutover around one schema owner
The core control is simple: only one deliberate migration action should own the schema transition. Do not let every application replica start with permission to race for it. In Kubernetes, that can mean a dedicated Job using the same immutable image as the new backend, with application Deployments scaled to zero. In another environment, it may be a separately invoked migration command in the release pipeline. The mechanism matters less than the guarantees: old writers are stopped, only one migration actor runs, and new writers begin only after the schema has been verified.
Prepare a recovery point
Take a database backup or a provider-supported point-in-time recovery marker immediately before the change. Confirm that it covers the relevant Backstage database and that operators know the restore procedure. A backup is not a rollback plan by itself; record expected restore time, data-loss window, and who has authority to initiate recovery. If the portal is business-critical, rehearse the restore in a non-production environment with a copy of representative catalog data.
Drain work, then stop all old backends
Announce a short portal maintenance period or route users to a clear maintenance response. Stop scheduled ingestion triggers if they run outside the backend. Scale the Backstage backend to zero and verify that no old process, worker, or ad hoc job retains database access. Merely removing pods from the load balancer is insufficient: a process that is not receiving HTTP traffic can still perform background ingestion.
Run and inspect the migration
Execute the migration once with the target 1.56 image and capture its exit status and structured logs. After it succeeds, inspect the table rather than assuming success from process completion alone. Confirm that source_key and entity_ref exist, the obsolete column is gone as expected, and the intended uniqueness constraint is present. Compare row counts with the pre-change baseline and inspect for nulls or duplicates that would undermine the upsert key.
Start one canary before restoring the fleet
Bring up a single new backend instance and keep the remaining replicas stopped. Validate startup migrations, database connectivity, catalog reads, provider activity, and ingestion-mark writes. Trigger a controlled provider refresh if the environment allows it. Only after the canary proves that entity updates progress should the deployment scale to normal capacity. This sequence narrows the source of any failure: schema, application startup, or multi-replica behavior.
Test catalog behavior, not just pod health
A green readiness endpoint proves that the process can answer its configured probe. It does not prove that incremental ingestion is correct. The verification plan should follow one or more known entities from provider input to catalog output. Choose a small sample that includes an update, a new entity, and, where safe, a deletion or orphaning path. Record entity references and timestamps before the cutover so the test is repeatable.
After the canary starts, verify that provider runs complete, ingestion marks advance, and changed entities appear in the catalog within the expected service-level objective. Watch catalog processing errors, database constraint violations, provider mutation duration, queue or backlog signals, and the age of the last successful refresh. Compare these with the baseline rather than relying on an absence of alerts.
Backstage 1.56 also includes catalog reliability improvements: stitching work is less likely to be lost across deletion and re-addition, successful entities should no longer remain unavailable after interrupted scheduling, and final entities plus search-index rows are written in one transaction. Those improvements strengthen the post-upgrade system, but they do not remove the need to validate the ingestion migration itself. Treat them as expected benefits to observe after the cutover, not as substitutes for checks.
Rollback means restoring code and schema together
A routine application rollback replaces the new image with the old image. That is unsafe after an incompatible schema migration because the old version may not understand the renamed column and new key structure. The rollback decision therefore has two branches.
- Before the migration begins: aborting is straightforward. Keep or restore the old application version because the database contract has not changed.
- After the migration commits: do not simply redeploy old pods. Either fix forward with the 1.56-compatible application or restore the pre-migration database recovery point and then restore the old application version.
Define the decision threshold in advance. Examples include a migration that does not complete inside the maintenance budget, failed canary startup, constraint errors during provider refresh, or a growing catalog-processing backlog. Also define the point of no return for new catalog writes. If users or providers can mutate data after the upgrade, restoring an earlier database snapshot can discard those changes. Keeping the portal in maintenance mode through canary validation makes rollback less ambiguous.
Use the disruption to improve future platform upgrades
This release exposes a broader platform-engineering lesson: version automation cannot determine rollout safety from semantic version numbers alone. Backstage releases aggregate many packages and plugins, while an organization’s portal adds local modules, database topology, and operational policy. A successful dependency update is not proof that two adjacent runtime versions can coexist.
Add a migration-compatibility checkpoint to the portal’s release template. Require the upgrade owner to classify every database change as backward-compatible, staged expand-and-contract, or coordinated cutover. Make CI produce the migration plan for review, but keep production execution as a separately observable step. Record which image owns the migration and prevent the application service account from making schema changes if your deployment model supports that separation.
The same template should require a data-path canary. For a developer portal, that means catalog ingestion, template execution, authentication, and search as appropriate—not only HTTP availability. Over time, these checks become reusable release evidence and shorten maintenance windows because operators no longer improvise verification.
The practical decision
Teams using Backstage’s incremental-ingestion module should upgrade to 1.56 through a brief, planned stop-the-world cutover: inventory the affected module, establish a recovery point, halt every old writer, run the migration once, inspect the resulting schema, and validate ingestion through a single canary before scaling out. Teams not using the module should document that finding and proceed with normal compatibility testing.
The important distinction is between availability as a goal and rolling deployment as a technique. For this migration, insisting on a rolling rollout would increase risk without preserving a valid shared database contract. A bounded maintenance window, supported by rehearsed recovery and catalog-level verification, is the more reliable path—and reliability is the outcome a platform team is supposed to optimize.
