MyAIDevelopment · Product · Operations notes

CI/CD Pipeline: From Commit to Production

CI/CD is not "a script that deploys automatically"; it is the quality gates a change passes on its way to production, written as code.

Terms

Term Meaning
CI (Continuous Integration) Merge changes often and run build and tests automatically on every merge
Continuous Delivery Keep the system always releasable to production. A human may approve the final deploy
Continuous Deployment Changes that pass the gates reach production without human intervention
Artifact The build output: a container image, an app bundle, a set of static files, and so on
Environment A place where artifacts run, such as Dev, Staging, Production

The Standard Flow

flowchart LR
    P[Push / PR] --> B[Build]
    B --> T[Test / Lint]
    T --> S[Security Scan]
    S --> A[Artifact + Attestation]
    A --> ST[Deploy Staging]
    ST --> V{Checks passed?}
    V -->|Yes| PR[Deploy Production]
    V -->|No| X[Stop / Notify]
    PR --> M[Observe / Ready to roll back]

The core principle is promoting an artifact built once through every environment. Rebuilding per environment means what you verified in staging may differ from what reached production.

Configuration Lives Outside the Code

The Twelve-Factor App recommends keeping values that differ per deploy (database addresses, credentials for external services, hostnames) in the environment, not in the code. The same artifact should run in many places by changing only the environment. Mounting secrets as volumes is now discussed as well.

Environment Design

Item Preview (per PR) Staging Production
Purpose See and review the change Verify the deploy procedure, migrations and integrations Real users
Data Seed or fake data Synthetic data or an anonymized subset Real data
Who deploys Automatic when a PR opens Automatic on merge to main CI only, after approval
Protection rules Short-lived, deleted when the PR closes, payments / email / push go to sandboxes Internal access only, no production secrets Required reviewers, wait timer, restricted deployment branches

Per-Environment Configuration and Secrets

The Trap of Production-Like Data

When a Small Team Can Skip Staging

You can go straight from preview to production when all of these are true.

  1. Every PR gets a preview with the same artifact and configuration structure as production.
  2. Database migrations stay backward compatible via expand → contract (document 04).
  3. You can roll back within minutes and have actually done it.
  4. Risky features are exposed through feature flags.

If any one is false, keep staging. It is essential if you often ship irreversible migrations or changes to payments, authentication or external integrations.

GitHub Actions Example

name: deploy
on:
  push:
    branches: [main]
permissions:
  contents: read
  id-token: write
jobs:
  build-deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@<commit-sha>
      - run: npm ci && npm test
      - run: npm run build
      - run: ./scripts/deploy.sh

OIDC Instead of Long-Lived Secrets

With GitHub Actions OpenID Connect, each workflow run obtains a short-lived token from the cloud provider. AWS, Azure, Google Cloud, HashiCorp Vault and others support it.

Bad:  Store a cloud access key as a repository secret and never rotate it for years.
Good: Narrow the OIDC trust to "this repository's main branch, production environment"
      and use only tokens that expire when the job ends.

An AWS example. The workflow holds only id-token: write and a role; there is no access key.

permissions:
  id-token: write
  contents: read
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: aws-actions/configure-aws-credentials@<commit-sha>
        with:
          role-to-assume: arn:aws:iam::123456789012:role/deploy-production
          aws-region: ap-northeast-2
      - run: aws sts get-caller-identity

The IAM role's trust policy narrows which workflows may assume the role through the sub claim.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:ORG/REPO:environment:production"
        }
      }
    }
  ]
}

Incident: CI Is an Attack Surface Too

GitOps: Pull-Based Deployment

The OpenGitOps principles are four.

  1. Declarative: express the desired state declaratively.
  2. Versioned and Immutable: store the desired state where versions and full history are kept.
  3. Pulled Automatically: agents pull the desired state automatically.
  4. Continuously Reconciled: agents keep observing the actual state and bring it to the desired state.

Instead of CI pushing directly to a cluster, an agent inside the cluster watches Git and follows it, so drift (settings someone changed by hand) surfaces automatically.

Infrastructure as Code Too

Infrastructure created by hand in a console (click-ops) leaves no record of who changed what and why, and the same environment cannot be recreated. IaC (Infrastructure as Code) lets you review, version and reproduce infrastructure like application code.

Manage with IaC Fine to leave as click-ops
Networking (VPC, subnets), DNS records, databases · queues · buckets, IAM roles and policies, per-environment service settings, alerts and dashboards One-time bootstrap (state storage, management account), billing and organization settings, emergency fixes during an incident (then reflect them in code), personal exploration sandboxes

Tools in One Line Each

Remote State and Locking

State maps code to real resources. Kept on a laptop, teammates overwrite each other, and secrets held in resource attributes can sit there in plain text. Keep it in a remote backend, encrypted, with restricted access. Two people running apply at once corrupt the state, so also take a lock. Terraform's S3 backend introduced S3-native locking (use_lockfile) in 1.10; when it became generally available in 1.11, DynamoDB-based locking was deprecated.

terraform {
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "prod/network.tfstate"
    region       = "ap-northeast-2"
    encrypt      = true
    use_lockfile = true
  }
}

resource "aws_s3_bucket" "assets" {
  bucket = "acme-prod-assets"
}

plan as a PR Check

flowchart LR
    PR[Open PR] --> PL[Post plan output to the PR]
    PL --> RV{Review approved?}
    RV -->|Yes| AP[CI applies after merge]
    RV -->|No| FX[Fix the code]
    FX --> PL
    AP --> DR[Daily drift check]

Run plan on every PR and attach "what will be created, changed and destroyed" to the review. Stop if you see a destroy. Only CI runs apply, after merge, using the role obtained through OIDC (section above).

      - run: terraform init -input=false
      - run: terraform plan -input=false -out=tfplan

Drift Detection

Main should hold no unapplied code, so run terraform plan -detailed-exitcode on main every day. Exit code 0 means no changes, 1 an error, 2 differences present. A 2 means someone changed something in the console: reflect it in code or revert it. Giving people read-only production console access reduces drift in the first place.

Measuring Results: DORA Metrics

Metric Meaning
Deployment Frequency How often you deploy to production successfully
Change Lead Time Time from commit to a successful production deployment
Failed Deployment Recovery Time Time to recover from a failure caused by a deployment
Change Fail Rate Share of deployments needing immediate intervention such as a rollback or hotfix
Deployment Rework Rate Share of deployments that are unplanned (bug-fix) work

DORA uses Change Fail Rate and Deployment Rework Rate as the two measures of instability; Deployment Rework Rate was added as the fifth metric in 2024. Use the metrics not as a leaderboard between teams but as signals that show where to improve.

Designing Staged Checks

PR Check     Fast lint and unit tests (within minutes)
Main Merge   Integration tests, image build, staging deploy
Release      Smoke tests, approval, production deploy
Nightly      Heavy E2E, performance, dependency scans

Running every check on every PR is slow; running nothing is dangerous. Put fast checks first and heavy ones later.

References