ServerlessBase Blog

    • Containers in Production: Lessons Learned

    A 150-160 character meta description containing 'containers in production'

    containersdockerkubernetesdevopsproduction

    Containers in Production: Lessons Learned

    You've built your first Docker image, run it locally, and it works. Now you're trying to deploy it to production, and suddenly everything breaks. Container orchestration looks easy on paper, but the reality is messier than you expected. This article covers the practical lessons I've learned deploying containers at scale, focusing on what actually matters when moving from development to production.

    Understanding Container Orchestration

    Containers solve the problem of consistent runtime environments, but they introduce new challenges when you have multiple instances running across different machines. Container orchestration platforms like Kubernetes manage these complexities automatically. They handle service discovery, load balancing, scaling, and self-healing without you needing to write custom scripts for each of these concerns.

    The core concept is that you define your desired state in declarative manifests, and the orchestrator works to make the actual state match your desired state. If a container crashes, Kubernetes restarts it. If you need more replicas, it spins up additional pods. If a node fails, it reschedules your workloads to healthy nodes. This automation is what makes containers viable for production workloads.

    Why Orchestration Matters

    Without orchestration, you're manually managing each container instance. This approach scales poorly. Adding a new container requires SSH access to each server, updating configuration files, restarting services, and verifying everything works. Any mistake in this process introduces risk. Orchestration removes the manual steps and provides consistency across your entire fleet.

    Resource Management and Limits

    One of the most common mistakes is not setting resource limits. Containers are isolated processes, but they still consume CPU and memory. If you don't specify limits, a single runaway container can consume all available resources on a node, affecting other containers running on the same machine.

    # Example: Setting resource limits in Kubernetes
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"

    The requests field tells Kubernetes how much resources to reserve for your container, while limits define the maximum it can use. Setting appropriate limits prevents resource contention and ensures predictable performance.

    Image Size and Optimization

    Large container images slow down deployment times and increase storage costs. Every layer in your image adds to the download size, and users download the entire image when pulling from a registry. A 1GB image takes significantly longer to pull than a 100MB image, especially over slow networks.

    # Check image size
docker images myapp:latest
 
# Optimize by using multi-stage builds
# First stage: build the application
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
 
# Second stage: copy only the built artifacts
FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/index.js"]

    Multi-stage builds allow you to separate the build environment from the runtime environment, keeping the final image small. Using Alpine Linux as a base image reduces size further, though it requires careful handling of dependencies.

    Security Best Practices

    Containers run with the privileges of the user who created them. If you're running as root, any vulnerability in your application could compromise the entire host. The principle of least privilege means running containers as non-root users whenever possible.

    # Example: Running as non-root user
FROM node:18-alpine
 
# Create a non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001
 
# Switch to non-root user
USER nodejs
 
WORKDIR /app
COPY --chown=nodejs:nodejs package*.json ./
RUN npm ci --only=production
COPY --chown=nodejs:nodejs . .
EXPOSE 3000
CMD ["node", "dist/index.js"]

    You should also scan images for vulnerabilities before deploying. Tools like Trivy can automatically detect known security issues in your dependencies and base images.

    Networking and Service Discovery

    Containers need to communicate with each other, but they don't have fixed IP addresses. Service discovery mechanisms handle this automatically. In Kubernetes, services provide stable network endpoints for pods, regardless of their changing IP addresses.

    # Example: Kubernetes service definition
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000
  type: ClusterIP

    The selector matches pods with the app: my-app label, and the service routes traffic to those pods on port 3000. Other services can access this service using its name, my-service, without needing to know the individual pod IPs.

    Configuration Management

    Hardcoding configuration in your application makes deployment difficult. Different environments require different settings—database URLs, API keys, feature flags. Kubernetes ConfigMaps and Secrets provide mechanisms for managing this configuration separately from your application code.

    # Example: ConfigMap for configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  NODE_ENV: "production"
  LOG_LEVEL: "info"
  DATABASE_URL: "postgres://db:5432/mydb"
    # Mount ConfigMap as environment variables
kubectl create configmap app-config --from-file=config.yaml
kubectl apply -f deployment.yaml

    Secrets should be used for sensitive data like passwords and API keys. Kubernetes handles secret encryption at rest and provides mechanisms for rotation.

    Monitoring and Logging

    Containers are ephemeral, which means logs and metrics can disappear if the container crashes. You need centralized logging and monitoring to understand what's happening in your production environment.

    # Example: Sending logs to a log aggregation system
docker run -d \
  --name my-app \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  myapp:latest

    The max-size and max-file options prevent log files from growing indefinitely. For production workloads, you should use a dedicated logging solution like Loki, ELK, or Fluentd to collect and analyze logs from all containers.

    Practical Deployment Walkthrough

    Let's walk through deploying a containerized application to production using Kubernetes. This example uses a simple Node.js application that serves HTTP requests.

    Step 1: Build and Push the Image

    First, build your Docker image and push it to a container registry. For this example, we'll use Docker Hub.

    # Build the image
docker build -t myapp:v1.0 .
 
# Tag the image
docker tag myapp:v1.0 username/myapp:v1.0
 
# Push to registry
docker push username/myapp:v1.0

    Step 2: Create Kubernetes Deployment

    Create a deployment manifest that defines how many replicas to run and how to configure the container.

    # deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: username/myapp:v1.0
        ports:
        - containerPort: 3000
        resources:
          requests:
            memory: "256Mi"
            cpu: "250m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5

    Step 3: Create a Service

    Create a service to expose your application to the network.

    # service.yaml
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000
  type: LoadBalancer

    Step 4: Apply the Manifests

    Apply the manifests to create the deployment and service.

    # Apply deployment
kubectl apply -f deployment.yaml
 
# Apply service
kubectl apply -f service.yaml
 
# Verify deployment
kubectl get pods
kubectl get services

    Step 5: Verify the Deployment

    Check that your application is running and accessible.

    # Get the service URL
kubectl get service my-app-service
 
# Test the application
curl http://<service-url>/health

    Common Pitfalls and Solutions

    1. Image Pull Errors

    If pods fail to start with image pull errors, check your image name, tag, and registry credentials. Ensure your Kubernetes cluster has access to the registry.

    # Check pod events for errors
kubectl describe pod <pod-name>
 
# Verify image exists in registry
docker pull username/myapp:v1.0

    2. Resource Exhaustion

    If your application crashes due to OOM errors, increase the memory limits or optimize your application to use less memory.

    # Check pod events for OOM errors
kubectl describe pod <pod-name>
 
# Update deployment with increased limits
kubectl set resources deployment my-app --limits=memory=1Gi

    3. Configuration Issues

    If your application isn't using the correct configuration, verify that ConfigMaps and Secrets are properly mounted and that your application reads them correctly.

    # Check pod environment variables
kubectl exec <pod-name> -- env
 
# Verify ConfigMap contents
kubectl get configmap app-config -o yaml

    Conclusion

    Deploying containers in production requires careful planning and attention to detail. Resource management, image optimization, security, networking, and monitoring are all critical aspects of a successful containerized deployment. The practical walkthrough demonstrates the basic steps for deploying an application, but real-world deployments will have additional complexity.

    Platforms like ServerlessBase simplify many of these challenges by providing managed container orchestration, automated scaling, and built-in monitoring. They handle the operational overhead so you can focus on building your application rather than managing infrastructure.

    The key takeaway is that containers are a powerful tool, but they require proper configuration and management to be effective in production. Start with simple deployments, learn the basics, and gradually add more advanced features as your needs grow. The learning curve is steep, but the benefits of containerization—consistency, scalability, and portability—make it worth the effort.

    Leave comment