Secure Deployment Best Practices: CI/CD, Containers, and Infrastructure

TL;DR

The #1 deployment security best practice is to treat your deployment pipeline with the same security rigor as production. Use OIDC for CI/CD authentication, scan dependencies and containers for vulnerabilities, run containers as non-root, implement security gates in pipelines, and have tested rollback procedures.

"Your deployment pipeline is a privileged path to production. A compromised CI/CD system can bypass every security control you've built."

Best Practice 1: Secure CI/CD Authentication 3 min

Your CI/CD pipeline needs production access. Secure it properly:

GitHub Actions with OIDC (no long-lived secrets)

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

permissions:
  id-token: write  # Required for OIDC
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Authenticate with OIDC - no secrets stored
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789:role/GitHubActions
          aws-region: us-east-1

      - name: Deploy to ECS
        run: |
          aws ecs update-service --cluster prod --service api --force-new-deployment

Authentication Method	Security	Recommendation
OIDC tokens	Excellent	Use when available
Short-lived tokens	Good	Rotate frequently
Stored secrets	Acceptable	Last resort, rotate often
Personal access tokens	Poor	Avoid in CI/CD

Best Practice 2: Implement Security Gates 5 min

Block deployments that fail security checks:

Security gates in CI/CD pipeline

# .github/workflows/security.yml
name: Security Gates
on: [pull_request, push]

jobs:
  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run npm audit
        run: npm audit --audit-level=high

      - name: Run Snyk
        uses: snyk/actions/node@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high

  sast-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Semgrep
        uses: returntocorp/semgrep-action@v1
        with:
          config: p/security-audit

  secret-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Run Gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  container-scan:
    runs-on: ubuntu-latest
    needs: [build]
    steps:
      - name: Run Trivy
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          severity: 'CRITICAL,HIGH'
          exit-code: '1'

Best Practice 3: Secure Container Configuration 4 min

Containers need hardening before production:

Secure Dockerfile

# Use specific version, not latest
FROM node:20.11-alpine3.19

# Run as non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

# Set working directory
WORKDIR /app

# Copy package files first (better caching)
COPY package*.json ./
RUN npm ci --only=production

# Copy application code
COPY --chown=nodejs:nodejs . .

# Switch to non-root user
USER nodejs

# Expose port (non-privileged)
EXPOSE 3000

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1

# Don't run as PID 1
CMD ["dumb-init", "node", "server.js"]

Use minimal base images (alpine, distroless)
Run as non-root user
Use multi-stage builds to exclude dev deps
Pin specific image versions, not
Scan images for vulnerabilities before deploy
Set read-only file system where possible

Best Practice 4: Infrastructure as Code Security 3 min

Scan your infrastructure definitions:

Terraform security scanning

# Run tfsec to find security issues
tfsec .

# Example issues it catches:
# - S3 buckets without encryption
# - Security groups with 0.0.0.0/0 access
# - RDS without encryption at rest
# - IAM policies with wildcard permissions

# checkov for comprehensive scanning
checkov -d . --framework terraform

# In CI/CD pipeline
- name: Run tfsec
  uses: aquasecurity/tfsec-action@v1.0.3
  with:
    soft_fail: false  # Fail the build

Best Practice 5: Deployment Strategies 5 min

Minimize risk with safe deployment patterns:

Strategy	Risk Level	Rollback Speed	Best For
Blue-Green	Low	Instant	Critical services
Canary	Low	Fast	High-traffic services
Rolling	Medium	Medium	Stateless services
Recreate	High	Slow	Dev/staging only

Canary deployment example

# Kubernetes canary with Argo Rollouts
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-rollout
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 10    # 10% of traffic
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 50    # 50% of traffic
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: success-rate

---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          query: |
            sum(rate(http_requests_total{status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))

Best Practice 6: Rollback Procedures 3 min

Have tested rollback procedures ready:

Rollback procedures

# Quick rollback commands
# Kubernetes
kubectl rollout undo deployment/api
kubectl rollout status deployment/api

# AWS ECS
aws ecs update-service --cluster prod --service api \
  --task-definition api:previous-version

# Vercel
vercel rollback

# Database migrations (have down migrations ready)
npm run migrate:down

# Feature flags for instant rollback
const newFeature = await featureFlags.isEnabled('new-checkout');
if (newFeature) {
  return newCheckoutFlow(req);
} else {
  return legacyCheckoutFlow(req);
}

Practice Rollbacks: Regularly test your rollback procedures in staging. A rollback you have never tested is a rollback that might fail when you need it most.

External Resources: For comprehensive deployment security guidance, see the OWASP CI/CD Security Cheat Sheet , the Docker Security Cheat Sheet , and the Kubernetes Security Documentation . These resources provide industry-standard recommendations for securing your deployment pipeline.

Should I auto-deploy to production?

For most teams, auto-deploy to staging and require manual approval for production. Mature teams with strong testing and observability can safely auto-deploy to production with canary releases.

How do I handle database migrations in deployment?

Run migrations before deploying new code (expand-contract pattern). Ensure migrations are backward compatible so the old code still works. Never run destructive migrations in the same deployment as code changes.

What should block a deployment?

High or critical security vulnerabilities, failing tests, secret detection, and compliance violations should block deployment. Have clear policies documented and enforced in CI/CD.

Audit Your Deployment Security

Check your CI/CD pipeline and deployment configuration.

Start Free Scan