TL;DR
Treat the pipeline like production, because it is production with fewer eyes on it. Authenticate CI/CD with OIDC instead of stored secrets, gate deploys on dependency, secret and container scans, run containers as non-root, and rehearse the rollback before you need it.
"Your deployment pipeline is a privileged path to production. A compromised CI/CD system can bypass every security control you've built."
Best Practice 1: Secure CI/CD Authentication 3 min
Your pipeline holds production credentials, which makes it a target. A stored AWS_SECRET_ACCESS_KEY in repo settings is the version attackers hope for. OIDC swaps it for a token minted per run and expired minutes later:
# .github/workflows/deploy.yml
name: Deploy
on:
push:
branches: [main]
permissions:
id-token: write # Required for OIDC
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Authenticate with OIDC - no secrets stored
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/GitHubActions
aws-region: us-east-1
- name: Deploy to ECS
run: |
aws ecs update-service --cluster prod --service api --force-new-deployment
| Authentication Method | Security | Recommendation |
|---|---|---|
| OIDC tokens | Excellent | Use when available |
| Short-lived tokens | Good | Rotate frequently |
| Stored secrets | Acceptable | Last resort, rotate often |
| Personal access tokens | Poor | Avoid in CI/CD |
Best Practice 2: Implement Security Gates 5 min
A scan that only warns gets ignored by the third sprint. Fail the build:
# .github/workflows/security.yml
name: Security Gates
on: [pull_request, push]
jobs:
dependency-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run npm audit
run: npm audit --audit-level=high
- name: Run Snyk
uses: snyk/actions/node@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
args: --severity-threshold=high
sast-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Semgrep
uses: returntocorp/semgrep-action@v1
with:
config: p/security-audit
secret-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Run Gitleaks
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
container-scan:
runs-on: ubuntu-latest
needs: [build]
steps:
- name: Run Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: myapp:${{ github.sha }}
severity: 'CRITICAL,HIGH'
exit-code: '1'
Best Practice 3: Secure Container Configuration 4 min
Most generated Dockerfiles are three lines and run as root. The hardened version:
# Use specific version, not latest
FROM node:20.11-alpine3.19
# Run as non-root user
RUN addgroup -g 1001 -S nodejs && \
adduser -S nodejs -u 1001
# Set working directory
WORKDIR /app
# Copy package files first (better caching)
COPY package*.json ./
RUN npm ci --only=production
# Copy application code
COPY --chown=nodejs:nodejs . .
# Switch to non-root user
USER nodejs
# Expose port (non-privileged)
EXPOSE 3000
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget --no-verbose --tries=1 --spider http://localhost:3000/health || exit 1
# Don't run as PID 1
CMD ["dumb-init", "node", "server.js"]
- Use minimal base images (alpine, distroless). Less installed, less to exploit.
- Run as a non-root user
- Use multi-stage builds so dev dependencies never reach the final image
- Pin specific image versions, never
:latest - Scan images for vulnerabilities before deploy
- Set a read-only file system where possible
Best Practice 4: Infrastructure as Code Security 3 min
Terraform is code, so scan it like code. A wide-open security group is a one-line diff that reads like every other one-line diff:
# Run tfsec to find security issues
tfsec .
# Example issues it catches:
# - S3 buckets without encryption
# - Security groups with 0.0.0.0/0 access
# - RDS without encryption at rest
# - IAM policies with wildcard permissions
# checkov for comprehensive scanning
checkov -d . --framework terraform
# In CI/CD pipeline
- name: Run tfsec
uses: aquasecurity/tfsec-action@v1.0.3
with:
soft_fail: false # Fail the build
Best Practice 5: Deployment Strategies 5 min
Your strategy decides how long a bad release stays live:
| Strategy | Risk Level | Rollback Speed | Best For |
|---|---|---|---|
| Blue-Green | Low | Instant | Critical services |
| Canary | Low | Fast | High-traffic services |
| Rolling | Medium | Medium | Stateless services |
| Recreate | High | Slow | Dev/staging only |
# Kubernetes canary with Argo Rollouts
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: api-rollout
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 10 # 10% of traffic
- pause: { duration: 5m }
- analysis:
templates:
- templateName: success-rate
- setWeight: 50 # 50% of traffic
- pause: { duration: 10m }
- analysis:
templates:
- templateName: success-rate
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
metrics:
- name: success-rate
successCondition: result[0] >= 0.99
provider:
prometheus:
query: |
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))
Best Practice 6: Rollback Procedures 3 min
Know the command before the incident. Nobody reads docs well at 2am:
# Quick rollback commands
# Kubernetes
kubectl rollout undo deployment/api
kubectl rollout status deployment/api
# AWS ECS
aws ecs update-service --cluster prod --service api \
--task-definition api:previous-version
# Vercel
vercel rollback
# Database migrations (have down migrations ready)
npm run migrate:down
# Feature flags for instant rollback
const newFeature = await featureFlags.isEnabled('new-checkout');
if (newFeature) {
return newCheckoutFlow(req);
} else {
return legacyCheckoutFlow(req);
}
Practice Rollbacks: Run the rollback in staging on a schedule, not just when something breaks. An untested rollback is a plan, not a procedure, and you learn which one you have at the worst moment.
External Resources: For comprehensive deployment security guidance, see the OWASP CI/CD Security Cheat Sheet , the Docker Security Cheat Sheet , and the Kubernetes Security Documentation . These resources provide industry-standard recommendations for securing your deployment pipeline.
Should I auto-deploy to production?
Auto-deploy to staging, require a human click for production. That's the right default for almost everyone. Teams with strong test coverage and real observability can auto-deploy straight to production, but they earn that first, usually behind canary releases that roll themselves back.
How do I handle database migrations in deployment?
Migrations go first, code second, and the migration must be backward compatible so the running version survives it. That's the expand-contract pattern. The rule that saves you is boring: never ship a destructive migration in the same deploy as the code that needs it. Split it across two releases so there's always something to roll back to.
What should block a deployment?
High and critical vulnerabilities, failing tests, and any detected secret. Compliance violations too, if relevant. The list matters less than the gate actually failing the build rather than warning.
Further Reading
Put these practices into action with our step-by-step guides.
Audit Your Deployment Security
Check your CI/CD pipeline and deployment configuration.