MyAIDevelopment · Product · Operations notes

Release Strategies: Rolling · Blue-Green · Canary · Feature Flag · Rollback

Deploy and release are different. Deploying puts code into an environment; releasing lets users experience the change. The more you separate them, the smaller the risk.

Strategies at a Glance

Strategy How it works Benefit Cost / caution
Recreate Take the old version down, bring the new one up Simple Downtime
Rolling Replace instances a few at a time Little extra capacity Two versions briefly coexist
Blue-Green Build an identical environment and switch routing Fast switch and switch back Temporarily double resources
Canary Send part of the traffic to the new version and compare Minimal blast radius Needs metrics and a baseline
Feature Flag Deploy code; control exposure by configuration Separates deploy from release Flag debt must be managed

Rolling Update

This is the default strategy of a Kubernetes Deployment. Both maxUnavailable and maxSurge default to 25%.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1

Blue-Green

flowchart LR
    U[Users] --> R[Router / Load Balancer]
    R -->|current| B[Blue: v1]
    R -.->|after switch| G[Green: v2]

Bring up the new version (Green) in an environment identical to production, verify it, then change only the router. If something goes wrong, point the router back to Blue. This return is only truly possible if the database schema is compatible with both versions.

Canary

The Google SRE Workbook defines canarying as a partial and time-limited deployment of a change and its evaluation. A small slice of production (the canary) is compared with the rest (the control) to decide whether to proceed.

flowchart LR
    D[Deploy new version] --> C1[1% traffic]
    C1 --> E1{Compare errors / latency}
    E1 -->|healthy| C2[10% traffic]
    C2 --> E2{Compare again}
    E2 -->|healthy| C3[100% traffic]
    E1 -->|anomaly| RB[Rollback]
    E2 -->|anomaly| RB

Example comparison metrics: error rate, p95 latency, crash rate, key conversion rate. When automating, define the pass criteria before the deploy.

How traffic is split depends on the platform.

Feature Flag

Pete Hodgson's "Feature Toggles" article on martinfowler.com describes release toggles as the most common way to "separate feature release from code deployment."

Flag type Lifespan Example
Release Toggle Short Hide an unfinished feature
Experiment Toggle Duration of the experiment A/B test
Ops Toggle During operation Turn off a heavy feature during an incident (kill switch)
Permissioning Toggle Long Expose only to paid or beta users

Rollback and Roll Forward

Rollback      Return to the previous artifact. Fast and predictable.
Roll Forward  Deploy a fixed version. Chosen when data changes cannot be undone.
Flag Off      Keep the code; just turn the feature off. The fastest.

The most common thing that makes rollback hard is database migration. Split it with the Parallel Change (Expand → Migrate → Contract) pattern.

  1. Expand: add the new column and keep both old and new code working.
  2. Migrate: move the data and switch to the new code.
  3. Contract: remove the unused column (once things are stable enough).

Bad Example / Good Example

Bad:  Rename a column and deploy code in one step. If the deploy fails, the old code breaks on the new schema.
Good: Split add column → write both → switch reads → drop old column across several releases.

How Mobile Differs

A mobile binary that is already installed cannot be rolled back. That makes store-level gradual releases (Apple Phased Release, Google Play Staged Rollout) and server-side feature flags more important. Document 05 covers the details. The only way to retire a broken build already on devices is a server-checked minimum supported version with a force-update screen, built and tested before launch (see "Running a Staged Rollout" in document 05).

References