Security in Kubernetes - overview of admission webhooks

This blog post is a continuation of two previous posts on security mechanisms in Kubernetes. If you have not yet read them, click here for part 1 and part 2 to see how you can provide an adequate level of security in Kubernetes deployments.

Existing admission controllers are very useful, as they allow you to validate or modify requests to a Kubernetes API server. However, they have two limitations: They have to be compiled into an API server and can be configured only on the API server startup.

The flexibility of admission webhooks helps solve these problems.  Once enabled, their behavior depends on the special application running inside the Kubernetes cluster. In this way they can be modified/reconfigured at any time.

Types of admission webhooks

Admission webhooks are HTTP callbacks that receive requests and operate on them. There are two types:

• validating admission webhooks
• mutating admission webhooks

Validating admission webhooks allow you to reject a request for any reason that break cluster policy. Mutating admission webhooks modify a request and add/remove or modify its contents.

Mutating admission webhooks are executed before validation and right after a request is authorized. Whether a user is authorized to perform an operation on the cluster with the request must be checked before the mutation. You can then perform the requisite cluster admin operations (eg. injecting a sidecar container that has to be run as privileged or specifying a service account that a user cannot act as) while at the same time not granting regular access to those operations. Here the order of the execution matters, as some modification may depend on the previously performed operations. Validating admission webhooks are run after the mutating ones and after the general Kubernetes object validation. As there are no modifications, several validation webhooks can be executed simultaneously on the request to decrease the time to process a request.

How to run webhooks?

It is not only webhooks that are executed, as the above diagram shows, but also admission controllers that are enabled on the cluster. The requirements for running webhooks are as follows:

• Kubernetes cluster has to be in at least 1.9 version
• admissionregistration.k8s.io/v1beta1 API is enabled
• validation webhooks are enabled in the API server configuration*:

--enable-admission-plugins=ValidatingAdmissionWebhook,MutatingAdmissionWebhook

Admission webhooks can be configured only for particular operations, resources etc. The example below defines the configuration for webhooks to be executed only on create and update operations, and only for pods in namespaces marked with specific labels.

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  generation: 1
  name: mywebhookconfiguration
webhooks:
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    service:
      name: mywebhook1
      namespace: webhooks
      path: /blog/mutate1
  failurePolicy: Fail
  name: mywebhook1.webhooks.svc
  namespaceSelector:
    matchLabels:
      iwanttoenablemutation1here: "true"
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - pods
- admissionReviewVersions:
  - v1beta1
  clientConfig:
    service:
      name: mywebhook2
      namespace: webhooks
      path: /blog/mutate2
  failurePolicy: Fail
  name: mywebhook2.webhooks.svc
  namespaceSelector:
    matchLabels:
      iwanttoenablemutation2here: "true"
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods

In this example we deploy the configuration of webhooks to the Kubernetes cluster and define for each webhook a different label created in a namespace. The webhooks are deployed in the webhooks’ namespace and they could be called by http request to service with a specified path, eg. http://mywebhook1.webhooks.svc/mutate1/.

The admission webhook is identified by URL and does not necessarily have to be deployed on the K8s cluster. The admission request is sent by the API server to the admission webhook as an HTTPS POST request with JSON-encoded AdmissionRequest object containing UID. This UID is used to track the request/response cycle and to identify the proper entries in the API server logs. The admissionRequest structure is described here.

Admission response has to contain the same UID as the request. This UID is used to track the Request-Response round-trip. Further, the “Allowed” field containing the Boolean value of the result has to be included. If the request is not allowed, there is also a Result dictionary. Even though it is optional, it is advisable to set it properly. As users might be confused by altered behavior of Kubernetes, it is recommended that the reason  a request to a user has been rejected be passed. Full AdmissionResponse structure is available here.

If, during the execution of the webhook, HTTP 500 code is returned, or webhook processing times out, the failurePolicy setting in the webhook configuration defines whether the request should be accepted or rejected.

Webhooks - use cases

Common use cases for admission webhooks include:

• Customizing default values (eg. resources limits)
• Validation more sophisticated than in admission controllers delivered with Kubernetes
• Preventing users from creating/modifying/deleting particular resources that should be immutable (eg. RBAC policies can grant access to all pods in namespace and defining exception on that level is not possible)
• Modifying deployments to inject sidecar containers like log collectors, monitoring and auditing applications, and additional traffic controllers like Istio (this process is well documented here)

Conclusion

Admission webhooks offer a convenient and standardised way to extend cluster manageability and increase security. They help with monitoring or auditing user actions and application events before they’ve even been scheduled. With admission webhooks as a default security setting to the applications environment, the desired restrictions can be imposed on the actions performed by a user in a Kubernetes cluster. These restrictions are otherwise very difficult to impose. In this way you can create a reliable security mechanism that will help you to avoid security breaches and resource exhaustion. For these reasons, admission webhooks can be considered one of Kubernetes best practices in security.

* Please note that this parameter is valid from the 1.10 version. In version 1.9 you should use the admission-control flag instead of enable-admission-plugins.