Shipping to production is the job

Every engineering team we work with can demo something. Far fewer can deploy it on a Tuesday afternoon without a meeting first. That gap — between "it works" and "it is running" — is where most project risk actually lives.

Staging lies

Staging environments are optimistic by construction: smaller datasets, friendlier traffic, no long-lived rows written by a version of the code nobody remembers. The problems that end up costing weeks are the ones that only exist in production.

If deploying is a scheduled event rather than a routine one, the deploy itself is the thing to fix first.

What we do instead

• Deploy on day one, before there is anything worth deploying.
• Keep the rollback path tested, not documented.
• Make the migration reversible, or make it additive.

A migration that only goes one way is a migration you cannot deploy on a Friday:

-- Additive: safe to deploy ahead of the code that reads it.
ALTER TABLE transactions
  ADD COLUMN settled_at timestamptz;

CREATE INDEX CONCURRENTLY IF NOT EXISTS
  transactions_settled_at_idx ON transactions (settled_at)
  WHERE settled_at IS NOT NULL;

Backfill separately, behind a flag, at a rate the database is happy with. Then — and only then — start reading the new column.