ServerlessBase Blog

    • Blue-Green Deployments in Kubernetes

    A comprehensive guide to implementing blue-green deployments in Kubernetes for safer, faster releases with zero downtime.

    kubernetesdeploymentblue-greenzero-downtimerelease-management

    Blue-Green Deployments in Kubernetes

    You've probably been there: a deployment goes wrong, users see errors, and you're scrambling to roll back to the previous version. The classic rolling update strategy gives you gradual rollout, but it also means you're running two versions of your application simultaneously for the entire deployment window. What if you could switch traffic instantly between two completely separate environments, with zero downtime and instant rollback capability?

    Blue-green deployments solve this problem by maintaining two identical production environments: a "blue" environment running the current version and a "green" environment running the new version. When you're ready to release, you switch all traffic from blue to green. If something goes wrong, you flip the switch back in seconds. No gradual rollout, no partial deployments, no confusion about which version users are seeing.

    Understanding the Blue-Green Pattern

    Think of blue-green deployments like a traffic light system for your application. The blue environment is the "safe" state—everything is running smoothly. The green environment is the "test" state—new code is deployed here first. Traffic flows from blue to green when you're ready to release, and back to blue if something breaks.

    The key insight is that you're not rolling out changes incrementally. You're deploying to a completely separate environment and then switching traffic. This gives you several advantages over traditional rolling updates:

    • Instant rollback: If the new version has issues, switch traffic back to blue in milliseconds
    • Zero downtime: Users never see a degraded state during the switch
    • Clean separation: You can test the new version thoroughly before exposing it to users
    • Easy rollback: No need to roll back a partial deployment or investigate which version is active

    The pattern works well for any application where you can afford to maintain two parallel environments. For stateless applications, it's straightforward. For stateful applications with databases, you need to consider data consistency and migration strategies.

    Blue-Green vs. Rolling Updates

    To understand when to use blue-green deployments, let's compare them with rolling updates, the default Kubernetes deployment strategy.

    FactorRolling UpdatesBlue-Green Deployments
    DowntimeZero (gradual)Zero (instant switch)
    Rollback speedMinutes to hoursSeconds to minutes
    Resource usageSingle version activeTwo versions active
    Testing windowDuring rolloutBefore switch
    ComplexityLowMedium
    Ideal forStateless appsCritical releases

    Rolling updates are great for stateless applications where you can tolerate a few seconds of degraded performance during the switch. They're also simpler to implement and use less resources since you're only running one version at a time.

    Blue-green deployments shine when you need guaranteed zero downtime and fast rollback. They're particularly valuable for critical systems where any downtime or user-facing errors are unacceptable. The trade-off is that you need to maintain two environments, which doubles your resource costs.

    Implementing Blue-Green Deployments in Kubernetes

    Let's walk through a practical implementation using Kubernetes Deployments and Services. We'll use a simple Node.js application as an example.

    First, create a Deployment for your current version (blue):

    # blue-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: blue
  template:
    metadata:
      labels:
        app: myapp
        version: blue
    spec:
      containers:
      - name: myapp
        image: myapp:1.0.0
        ports:
        - containerPort: 3000

    Create a Service that routes traffic to the blue deployment:

    # blue-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
    version: blue
  ports:
  - port: 80
    targetPort: 3000

    Deploy the blue environment:

    kubectl apply -f blue-deployment.yaml
kubectl apply -f blue-service.yaml

    Now deploy the new version to the green environment:

    # green-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: green
  template:
    metadata:
      labels:
        app: myapp
        version: green
    spec:
      containers:
      - name: myapp
        image: myapp:1.1.0
        ports:
        - containerPort: 3000

    Deploy the green environment:

    kubectl apply -f green-deployment.yaml

    At this point, you have both versions running, but traffic is still going to blue. Before switching traffic, verify that the green deployment is healthy:

    kubectl get pods -l app=myapp,version=green
kubectl get svc myapp-service

    Check the green pods are running and ready:

    kubectl describe pod <green-pod-name>

    Once you're confident the green version is working, switch traffic by updating the service selector:

    kubectl patch svc myapp-service -p '{"spec":{"selector":{"version":"green"}}}'

    This single command switches all traffic from blue to green. If something goes wrong, switch back:

    kubectl patch svc myapp-service -p '{"spec":{"selector":{"version":"blue"}}}'

    Automating the Switch with Scripts

    Manually patching services is error-prone. Let's create a script that automates the entire blue-green deployment process.

    Create a deployment script:

    #!/bin/bash
# deploy-blue-green.sh
 
set -e
 
BLUE_DEPLOYMENT="myapp-blue"
GREEN_DEPLOYMENT="myapp-green"
SERVICE="myapp-service"
CURRENT_VERSION=$(kubectl get svc $SERVICE -o jsonpath='{.spec.selector.version}')
 
if [ "$CURRENT_VERSION" = "blue" ]; then
  TARGET_VERSION="green"
  PREVIOUS_VERSION="blue"
else
  TARGET_VERSION="blue"
  PREVIOUS_VERSION="green"
fi
 
echo "Switching from $CURRENT_VERSION to $TARGET_VERSION..."
 
kubectl patch svc $SERVICE -p "{\"spec\":{\"selector\":{\"version\":\"$TARGET_VERSION\"}}}"
 
echo "Waiting for new pods to be ready..."
kubectl wait --for=condition=ready pod -l app=myapp,version=$TARGET_VERSION --timeout=300s
 
echo "Deployment complete. Previous version: $PREVIOUS_VERSION, Current version: $TARGET_VERSION"

    Make the script executable:

    chmod +x deploy-blue-green.sh

    Run the deployment:

    ./deploy-blue-green.sh

    This script automatically detects which version is currently active, switches traffic to the other version, and waits for the new pods to be ready before completing the deployment.

    Using Helm for Blue-Green Deployments

    Helm makes blue-green deployments even cleaner by managing both environments as separate releases. Here's a Helm chart structure:

    # Chart.yaml
apiVersion: v2
name: myapp
description: A Helm chart for blue-green deployments
type: application
version: 1.0.0
appVersion: "1.0.0"

    Create a values file for the blue environment:

    # values-blue.yaml
replicaCount: 3
image:
  repository: myapp
  tag: "1.0.0"
  pullPolicy: IfNotPresent
service:
  type: ClusterIP
  port: 80
  targetPort: 3000
selector:
  version: blue

    Create a values file for the green environment:

    # values-green.yaml
replicaCount: 3
image:
  repository: myapp
  tag: "1.1.0"
  pullPolicy: IfNotPresent
service:
  type: ClusterIP
  port: 80
  targetPort: 3000
selector:
  version: green

    Deploy the blue environment:

    helm install myapp-blue ./myapp -f values-blue.yaml

    Deploy the green environment:

    helm install myapp-green ./myapp -f values-green.yaml

    Switch traffic using a Helm template:

    helm upgrade myapp-blue ./myapp -f values-blue.yaml --reuse-values
helm upgrade myapp-green ./myapp -f values-green.yaml --reuse-values

    Or use a simple script to switch between environments:

    #!/bin/bash
# switch-helm.sh
 
CURRENT_RELEASE=$(helm list -q | grep myapp | head -1)
if [ "$CURRENT_RELEASE" = "myapp-blue" ]; then
  TARGET="myapp-green"
else
  TARGET="myapp-blue"
fi
 
helm upgrade $TARGET ./myapp --reuse-values

    Handling Database Migrations

    Blue-green deployments work seamlessly for stateless applications, but database migrations require careful consideration. You have several options:

    1. Run migrations in both environments: Apply the same migration to both blue and green databases. This ensures both environments are in sync before switching traffic.

    2. Run migrations after the switch: Apply the migration to the green database after traffic has switched. This is faster but risks downtime if the migration fails.

    3. Use database versioning: Tag your database schemas with version numbers and ensure both environments use compatible versions.

    For most applications, option 1 is safest. Apply the migration to both environments, verify the green database is healthy, then switch traffic. If the migration fails, you can roll back to blue without affecting the database.

    Example migration script:

    #!/bin/bash
# migrate-database.sh
 
DB_HOST="postgres-service"
DB_NAME="myapp"
DB_USER="postgres"
 
echo "Running database migration..."
 
# Apply migration to green database
kubectl exec -it $(kubectl get pod -l app=postgres,version=green -o jsonpath='{.items[0].metadata.name}') -- \
  psql -h $DB_HOST -U $DB_USER -d $DB_NAME -f migration.sql
 
echo "Migration complete. Verify the green database is healthy before switching traffic."

    Monitoring and Validation

    Before switching traffic to green, you need to validate that everything is working correctly. Here's a checklist:

    1. Check pod health: Ensure all green pods are running and ready

      kubectl get pods -l app=myapp,version=green

    2. Verify service endpoints: Confirm the service is routing to green pods

      kubectl get svc myapp-service

    3. Test API endpoints: Make manual requests to verify functionality

      curl http://myapp-service

    4. Check logs: Review green pod logs for any errors

      kubectl logs -l app=myapp,version=green --tail=100

    5. Monitor metrics: Verify performance metrics are acceptable

      kubectl top pods -l app=myapp,version=green

    6. Test critical user flows: Manually test the most important user journeys

    If any check fails, investigate and fix the issue before switching traffic. Remember, you can always roll back to blue if green has problems.

    Common Pitfalls and Best Practices

    Pitfall 1: Forgetting to Clean Up Old Deployments

    After switching traffic, you'll have both blue and green deployments running. This doubles your resource usage. Clean up the old version after a successful deployment:

    kubectl delete deployment myapp-blue

    Or use a script that automatically cleans up:

    #!/bin/bash
# cleanup-old-deployment.sh
 
CURRENT_VERSION=$(kubectl get svc myapp-service -o jsonpath='{.spec.selector.version}')
OLD_VERSION=$(kubectl get svc myapp-service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
 
if [ "$CURRENT_VERSION" = "blue" ]; then
  kubectl delete deployment myapp-green
else
  kubectl delete deployment myapp-blue
fi

    Pitfall 2: Not Testing the Rollback

    Always test your rollback process before deploying to production. Create a script that simulates a failure and verifies the rollback works:

    #!/bin/bash
# test-rollback.sh
 
echo "Testing rollback process..."
 
# Switch to green
kubectl patch svc myapp-service -p '{"spec":{"selector":{"version":"green"}}}'
 
# Wait a few seconds
sleep 10
 
# Simulate a failure by switching back to blue
kubectl patch svc myapp-service -p '{"spec":{"selector":{"version":"blue"}}}'
 
# Verify blue is active
sleep 5
CURRENT_VERSION=$(kubectl get svc myapp-service -o jsonpath='{.spec.selector.version}')
 
if [ "$CURRENT_VERSION" = "blue" ]; then
  echo "Rollback successful!"
else
  echo "Rollback failed!"
  exit 1
fi

    Pitfall 3: Ignoring Resource Limits

    Running two environments doubles your resource requirements. Ensure your cluster has enough capacity:

    # deployment.yaml
resources:
  limits:
    cpu: "500m"
    memory: "512Mi"
  requests:
    cpu: "250m"
    memory: "256Mi"

    Monitor resource usage during deployments:

    kubectl top pods -l app=myapp

    Pitfall 4: Not Using Labels Consistently

    Consistent labeling is critical for blue-green deployments. Always use the same labels across deployments, services, and pods:

    metadata:
  labels:
    app: myapp
    version: blue  # or green

    Inconsistent labels will cause routing issues and make it difficult to manage multiple environments.

    Conclusion

    Blue-green deployments provide a powerful pattern for safe, fast releases in Kubernetes. By maintaining two parallel environments and switching traffic instantly, you achieve zero downtime and rapid rollback capability. The key to success is:

    1. Consistent labeling across all resources
    2. Automated deployment scripts to reduce human error
    3. Thorough validation before switching traffic
    4. Regular cleanup of old deployments to manage costs
    5. Database migration strategies that work with the blue-green pattern

    Platforms like ServerlessBase simplify blue-green deployments by handling the complex infrastructure details, allowing you to focus on releasing your application safely and quickly.

    Ready to implement blue-green deployments in your Kubernetes cluster? Start with a simple stateless application, automate the process with scripts, and gradually add validation and monitoring as you become more comfortable with the pattern.

    Leave comment