How to Build a Deployment Checklist That Actually Prevents Production Incidents

Summary

A deployment checklist should split into two columns: application risk (schema changes, background worker payload compatibility, cache serialization, rollback triggers) that only your team can judge, and infrastructure risk (certificates, image provenance, autoscaling, log shipping) that should be automated rather than checked by hand. Covers the expand-and-contract pattern for safe database migrations, separating deploy from release using feature flags, setting rollback triggers before shipping, and using Kubernetes startup/readiness probes to avoid cold-start restart loops. Argues that infrastructure-column items that can't be automated point toward needing an internal delivery platform or a PaaS, and recommends tracking DORA metrics to validate whether the checklist is working.

Questions this post answers

How do I safely rename a database column without breaking production during deployment?

Use the expand and contract pattern across four separate deploys instead of one. First add the new nullable column without touching the old one. Second, write to both columns while still reading the old one, then backfill existing rows in batches. Third, switch reads to the new column with a fallback to the old one for safe rollback. Only weeks later, once nothing references the old column, drop it.

How do I prevent background worker errors when deploying a new message payload format?

Send both the old and new payload shapes in the same message for one release, so either the old or new consumer version can read it successfully. For example, keep a customer_id field alongside a new nested customer object. Deploy the new consumer, let the queue drain, then drop the old field in a later release, following the same add-migrate-remove sequence used for database columns.

Why does my Kubernetes rollout keep restarting a pod that needs a long warm-up time?

A single readiness probe with too short a failureThreshold kills instances right before they finish warming up, for example when an app needs 40 seconds to warm a connection pool but the probe gives up after 30. Split the check into a startupProbe with a generous window for first boot (e.g. failureThreshold 18 for ~90 seconds) and a separate, tighter readinessProbe for ongoing traffic checks.