Introduction to Kubernetes Admission Controllers

You've probably seen the error message: "admission webhook "validating-webhook-configuration" denied the request: ...". It's frustrating when you're trying to deploy something and the cluster says no, but you don't know why. That's admission controllers at work.

Admission controllers are the gatekeepers of Kubernetes. They intercept API requests before the objects are persisted to the etcd database. If an admission controller says no, the request fails. If it says yes, the object gets created. They're the first line of defense for cluster security and consistency.

How Admission Controllers Work

Think of admission controllers as a series of filters placed between the API server and etcd. When you send a request to create or update a resource, it goes through these filters in order. Each controller can either allow, deny, or modify the request.

The admission process happens in two phases:

1. Validating admission controllers - Check if the request is valid. They don't modify the request, they just say yes or no.

2. Mutating admission controllers - Can modify the request before it's processed. They can add default values, inject sidecars, or apply other transformations.

Here's a simplified flow:

Client Request → API Server → Admission Controllers (in order) → etcd

If any admission controller returns an error, the request fails immediately. The order of controllers matters because earlier controllers can modify the request, affecting later controllers.

Built-in Admission Controllers

Kubernetes has several built-in admission controllers. You can see the full list in the Kubernetes documentation, but here are the most commonly used ones:

Validating Admission Controllers

Validating controllers are the simplest type. They just check if a request is valid and reject it if it's not. They don't modify the request in any way.

Example: LimitRanger

LimitRanger ensures that pods don't exceed resource limits. If you try to create a pod without specifying CPU or memory limits, LimitRanger will reject it:

# This will be rejected by LimitRanger
apiVersion: v1
kind: Pod
metadata:
  name: no-limits
spec:
  containers:
  - name: app
    image: nginx

# Error message
Error from server (Forbidden): error when creating "no-limits.yaml": admission webhook "limitranger.admission.k8s.io" denied the request: pod "no-limits" is forbidden: pod must have memory limits and requests

To fix it, you need to specify limits:

# This will be accepted
apiVersion: v1
kind: Pod
metadata:
  name: with-limits
spec:
  containers:
  - name: app
    image: nginx
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
      requests:
        memory: "64Mi"
        cpu: "250m"

Mutating Admission Controllers

Mutating controllers are more powerful. They can modify the request before it's processed. This is useful for applying defaults, injecting sidecars, or applying other transformations.

Example: ServiceAccount

When you create a pod without specifying a service account, the ServiceAccount admission controller automatically creates one for you:

# No service account specified
apiVersion: v1
kind: Pod
metadata:
  name: auto-sa
spec:
  containers:
  - name: app
    image: nginx

After admission, the pod will have this service account automatically:

# What gets stored in etcd
apiVersion: v1
kind: Pod
metadata:
  name: auto-sa
spec:
  serviceAccountName: default  # Added by admission controller
  containers:
  - name: app
    image: nginx

Example: DefaultStorageClass

If you create a PersistentVolumeClaim without specifying a storage class, the DefaultStorageClass admission controller will set a default storage class:

# No storage class specified
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: auto-storage
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

After admission, the PVC will have a storage class:

# What gets stored in etcd
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: auto-storage
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: fast-ssd  # Added by admission controller

Custom Admission Controllers

Kubernetes also supports custom admission controllers using webhooks. You can write your own admission controller as a web service and register it with the cluster.

How Webhooks Work

Custom admission controllers are implemented as HTTP webhooks. When a request comes in, the API server sends it to your webhook service. Your service validates or mutates the request and returns a response.

Here's the flow:

Client Request → API Server → Custom Webhook → Your Service → Response → etcd

Example: Validating Webhook

Let's create a simple validating webhook that rejects pods with the label "restricted". First, you need a webhook service. Here's a simple Node.js example:

// webhook-server.js
const express = require('express');
const app = express();
 
app.use(express.json());
 
app.post('/validate', (req, res) => {
  const { request } = req.body;
 
  // Check if the pod has the restricted label
  if (request.object.metadata.labels && request.object.metadata.labels.restricted === 'true') {
    return res.status(403).json({
      apiVersion: 'admission.k8s.io/v1',
      kind: 'AdmissionReview',
      response: {
        allowed: false,
        status: {
          code: 403,
          message: 'Pods with label "restricted=true" are not allowed'
        }
      }
    });
  }
 
  // Allow the request
  res.json({
    apiVersion: 'admission.k8s.io/v1',
    kind: 'AdmissionReview',
    response: {
      allowed: true
    }
  });
});
 
app.listen(8443, () => {
  console.log('Webhook server running on port 8443');
});

To register this webhook, you create a ValidatingWebhookConfiguration:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: restricted-pod-webhook
webhooks:
- name: restricted-pod.example.com
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE", "UPDATE"]
    resources: ["pods"]
  clientConfig:
    service:
      name: webhook-service
      namespace: default
      path: /validate
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 5

Now when you try to create a pod with the restricted label, it will be rejected:

# This will be rejected
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: restricted-pod
  labels:
    restricted: "true"
spec:
  containers:
  - name: app
    image: nginx
EOF

# Error message
Error from server (Forbidden): admission webhook "restricted-pod.example.com" denied the request: Pods with label "restricted=true" are not allowed

Example: Mutating Webhook

Mutating webhooks work similarly but can modify the request. Here's an example that adds a sidecar container to all pods:

// webhook-server.js (mutating example)
const express = require('express');
const app = express();
 
app.use(express.json());
 
app.post('/mutate', (req, res) => {
  const { request } = req.body;
 
  // Clone the object to avoid mutating the original
  const object = JSON.parse(JSON.stringify(request.object));
 
  // Add a sidecar container
  if (!object.spec.containers) {
    object.spec.containers = [];
  }
 
  object.spec.containers.push({
    name: 'sidecar',
    image: 'fluentd',
    volumeMounts: [
      {
        name: 'varlog',
        mountPath: '/var/log'
      }
    ]
  });
 
  // Add volume for the sidecar
  if (!object.spec.volumes) {
    object.spec.volumes = [];
  }
 
  object.spec.volumes.push({
    name: 'varlog',
    hostPath: {
      path: '/var/log'
    }
  });
 
  res.json({
    apiVersion: 'admission.k8s.io/v1',
    kind: 'AdmissionReview',
    response: {
      allowed: true,
      patchType: 'JSONPatch',
      patch: Buffer.from(JSON.stringify([
        {
          op: 'replace',
          path: '/spec',
          value: object.spec
        }
      ])).toString('base64')
    }
  });
});
 
app.listen(8443, () => {
  console.log('Mutating webhook server running on port 8443');
});

Admission Controller Order

The order of admission controllers matters. Earlier controllers can modify the request, affecting later controllers. This is why Kubernetes has a fixed order for built-in controllers.

For custom webhooks, you can control the order by setting the failurePolicy and matchPolicy:

• failurePolicy: Fail - The webhook fails the request if it's not reachable or returns an error.
• failurePolicy: Ignore - The webhook is ignored if it's not reachable or returns an error.
• matchPolicy: Exact - Only matches requests that exactly match the rules.
• matchPolicy: Equivalent - Matches requests that are equivalent to the rules.

Best Practices

1. Use Validating Webhooks for Security

Validating webhooks are best for security policies. They don't modify the request, so they're less likely to cause unexpected side effects.

2. Use Mutating Webhooks for Defaults

Mutating webhooks are great for applying defaults. Instead of requiring every developer to specify the same values, let the admission controller do it for them.

3. Test Your Webhooks Thoroughly

Webhooks can be tricky to test. Use tools like kubectl apply --dry-run=client to test your webhooks locally before deploying them to production.

4. Handle Webhook Failures Gracefully

If your webhook is down, the API server will fail the request (if failurePolicy: Fail). Make sure your webhook is highly available and has proper error handling.

5. Use SideEffects: None When Possible

The sideEffects field tells the API server whether your webhook has side effects. If your webhook doesn't modify external resources, set sideEffects: None to improve performance.

Common Use Cases

1. Enforce Resource Limits

Use LimitRanger or a custom webhook to ensure all pods have resource limits. This prevents runaway pods from consuming all cluster resources.

2. Inject Sidecars

Use mutating webhooks to inject monitoring agents, logging sidecars, or security agents into pods automatically.

3. Apply Security Policies

Use validating webhooks to enforce security policies, such as requiring specific labels, preventing certain container images, or enforcing pod security standards.

4. Set Defaults

Use mutating webhooks to set default values for resources, such as storage classes, service accounts, or environment variables.

5. Implement Multi-Tenancy

Use admission controllers to enforce multi-tenancy policies, such as preventing pods from accessing resources in other namespaces.

Troubleshooting

Webhook Not Working

If your webhook isn't working, check these common issues:

1. Webhook service is not running - Make sure your webhook service is running and accessible.
2. Wrong service name or namespace - Check the clientConfig.service.name and namespace in your webhook configuration.
3. Wrong path - Make sure the path in your webhook configuration matches the endpoint in your service.
4. Certificate issues - If you're using TLS, make sure the certificate is valid and trusted by the API server.
5. Timeout - If your webhook takes too long to respond, the API server will timeout. Set an appropriate timeoutSeconds.

Debugging Webhooks

You can debug webhooks by enabling debug logging in the API server:

kubectl logs -n kube-system -l component=kube-apiserver

You can also use kubectl apply --dry-run=client to see if your webhook is working:

kubectl apply --dry-run=client -f pod.yaml

Conclusion

Admission controllers are a powerful way to enforce policies and apply defaults in Kubernetes. They're the first line of defense for cluster security and consistency. Whether you use built-in controllers or custom webhooks, understanding how they work is essential for managing a Kubernetes cluster effectively.

If you're using a platform like ServerlessBase, many of these admission controller patterns are handled for you automatically, so you can focus on your application code instead of cluster configuration.

Next Steps

• Learn about Pod Security Standards in Kubernetes
• Explore how to implement Pod Security Admission
• Check out the Kubernetes documentation on admission controllers
• Build your first custom admission webhook