Kubernetes DaemonSets: Running Pods on Every Node

You've deployed a monitoring agent, a log collector, or a node-level daemon to your Kubernetes cluster. You expect it to run on every single node, but somehow it's only on a few. You check the pods, and they're scattered across nodes randomly. This is where DaemonSets come in.

A DaemonSet ensures that a single pod runs on each node in your cluster. Unlike Deployments, which manage a replica set of identical pods, DaemonSets guarantee that every node participates in your workload. This is critical for node-level agents that need to be present on every machine.

How DaemonSets Differ from Deployments

Understanding the distinction between DaemonSets and Deployments is essential before you start using them.

The key difference lies in placement strategy. Deployments use a scheduler to distribute pods across nodes based on resource availability and constraints. DaemonSets, however, enforce a strict rule: one pod per node. This deterministic placement is what makes DaemonSets ideal for node-level workloads.

When to Use DaemonSets

DaemonSets are perfect for scenarios where every node needs to run a specific component. Common use cases include:

• Node monitoring agents: Prometheus exporters, Node Exporter, or custom monitoring scripts
• Log collectors: Fluentd, Fluent Bit, or other log aggregation agents that need to be on every node
• Storage daemons: CSI (Container Storage Interface) drivers that must run on each node
• Network plugins: CNI (Container Network Interface) plugins like Calico or Flannel
• Security agents: Falco runtime security monitors or other node-level security tools
• Custom node utilities: Any custom script or service that needs to run on every node

If you're building a monitoring stack, DaemonSets are your best friend. They ensure your Prometheus Node Exporter is running on every node, giving you complete visibility into your cluster's health.

DaemonSet Architecture

A DaemonSet controller works by maintaining a desired number of pod replicas on each node. When a new node joins the cluster, the DaemonSet automatically creates a pod on that node. When a node fails or is removed, the DaemonSet deletes its pod from that node.

The controller achieves this through node selectors and taints/tolerations. By default, DaemonSets place pods on all nodes, but you can configure them to target specific node types using node selectors, node affinity, or taints and tolerations.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100

This simple DaemonSet ensures that a Node Exporter pod runs on every node in your cluster, exposing metrics for monitoring.

Node Selection Strategies

You often need more control over where DaemonSet pods run. Kubernetes provides several strategies for node selection:

Node Selectors

Node selectors are the simplest way to target specific nodes. You define a key-value pair that must match the node's labels.

spec:
  template:
    spec:
      nodeSelector:
        disktype: ssd
        environment: production

This DaemonSet will only run on nodes labeled with disktype=ssd and environment=production.

Node Affinity

Node affinity provides more powerful matching rules, including required and preferred relationships.

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-west-1a

This DaemonSet runs on nodes with amd64 architecture and prefers nodes in the us-west-1a zone.

Taints and Tolerations

Sometimes you need to run pods on nodes with specific characteristics, like dedicated GPU nodes or master nodes. You can use taints to mark nodes and tolerations to allow DaemonSet pods to schedule on them.

# Node taint
kubectl taint nodes node1 dedicated=gpu:NoSchedule
 
# DaemonSet toleration
spec:
  template:
    spec:
      tolerations:
      - key: dedicated
        operator: Equal
        value: gpu
        effect: NoSchedule

This DaemonSet can now run on node1, which has the dedicated=gpu taint.

Rolling Updates and Rollbacks

DaemonSets support rolling updates, allowing you to update pods one node at a time. This is crucial for maintaining cluster stability during updates.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

The maxUnavailable field controls how many pods can be unavailable during the update. Setting it to 1 ensures that only one node's pod is updated at a time, minimizing disruption.

If something goes wrong with your update, you can roll back to the previous version using standard Kubernetes commands:

kubectl rollout undo daemonset/node-exporter

Practical Example: Setting Up a Log Collector

Let's walk through a practical example of deploying a log collector using a DaemonSet.

First, create a namespace for your monitoring stack:

kubectl create namespace logging

Next, create the DaemonSet manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: logging
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch.logging.svc.cluster.local"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        - name: FLUENT_ELASTICSEARCH_SCHEME
          value: "http"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Apply the DaemonSet:

kubectl apply -f fluentd-daemonset.yaml

Verify that pods are running on all nodes:

kubectl get pods -n logging -l app=fluentd

You should see one pod per node in your cluster.

Managing DaemonSet Pods

Once your DaemonSet is running, you'll need to manage its pods. Here are common operations:

View DaemonSet status

kubectl get daemonsets -n monitoring
kubectl describe daemonset node-exporter -n monitoring

Scale a DaemonSet

DaemonSets scale automatically when nodes are added or removed. However, you can manually scale them:

kubectl scale daemonset node-exporter --replicas=5 -n monitoring

This forces the DaemonSet to maintain 5 pods, which may cause some nodes to have multiple pods.

Delete a DaemonSet

To remove all pods and the DaemonSet:

kubectl delete daemonset node-exporter -n monitoring

Update a DaemonSet

To update the image or configuration:

kubectl set image daemonset/node-exporter node-exporter=prom/node-exporter:v1.0.0 -n monitoring

Or edit the manifest and apply it again:

kubectl edit daemonset node-exporter -n monitoring

Best Practices

Use Resource Limits

Always set resource requests and limits for DaemonSet pods to prevent them from consuming all node resources:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 200m
    memory: 256Mi

Handle Node Drain Gracefully

When you need to drain a node for maintenance, DaemonSet pods will be evicted. Configure graceful termination to give your agents time to flush logs or metrics:

terminationGracePeriodSeconds: 30

Monitor DaemonSet Health

Keep an eye on DaemonSet pod status. Failed pods can indicate configuration issues or resource constraints:

kubectl get pods -n monitoring -l app=node-exporter

Use Node Affinity for Critical Workloads

For critical node-level agents, use node affinity to ensure they run on specific node types or zones:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-role.kubernetes.io/control-plane
          operator: In
          values:
          - "true"

Common Pitfalls

Overloading Nodes

DaemonSets can overload nodes if you're not careful. If you have 10 DaemonSet pods on a node with 2 CPUs and 4GB RAM, and each pod requests 1 CPU and 1GB RAM, you'll run out of resources.

Solution: Monitor node resource usage and adjust pod requests accordingly.

Ignoring Pod Disruption Budgets

DaemonSet pods can be evicted during node maintenance, which might interrupt critical services. Use Pod Disruption Budgets to ensure minimum availability:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: fluentd-pdb
  namespace: logging
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: fluentd

Forgetting to Update Images

When updating DaemonSet images, ensure you're using the correct version. A simple typo can break your entire monitoring stack.

Solution: Use version tags and automate image updates with CI/CD pipelines.

Conclusion

DaemonSets are a powerful Kubernetes primitive for ensuring node-level workloads run consistently across your cluster. They're essential for monitoring, logging, and other infrastructure components that need to be present on every node.

The key takeaways are:

• DaemonSets guarantee one pod per node, unlike Deployments which distribute pods randomly
• Use DaemonSets for node-level agents like monitoring, logging, and storage drivers
• Configure node selection strategies using selectors, affinity, and tolerations
• Implement rolling updates and rollbacks to maintain cluster stability
• Always set resource limits and use Pod Disruption Budgets for production workloads

Platforms like ServerlessBase simplify DaemonSet deployment by providing a user-friendly interface for managing these workloads, handling the complex Kubernetes configuration automatically.

Ready to deploy your first DaemonSet? Start with a simple monitoring agent like Node Exporter and gradually build out your node-level infrastructure.