This post is one of the K8s API and Controllers series
- K8s CustomResourceDefinitions (CRD) internals
- best practice for K8s api multi-version conversion
- simple K8s apiserver from scratch
- K8s apiserver aggregation internals
- The most thorough K8s code generation tutorial
- Implement K8s apiserver using library
- Why custom K8s apiserver should avoid runtime?
- K8s API Admission Control and Policy (this post)
🎟️ What is Admission Control in K8s
Admission control is a module in kube-apiserver that intercepts requests before an object is persisted but after the request is authenticated and authorized. Any API request coming to kube-apiserver follows these steps:
- Firstly, it passes filterchain, in which kube-apiserver does authn/authz to it
- Then, it is dispatched by kube-aggregator to the corresponding sub apiserver’s HTTP mux
- After decoding/conversion/defaulting with runtime.Scheme
- The corresponding sub apiserver performs admission control on the request
- RESTStorage strategy will be executed, and it finally is persisted to etcd
The admission control is performed by Admission Controllers.
An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.
Admission Controllers are quite common in K8s, For example (in v1.27.3 and most previous versions), during the creation of a Pod, it will be mutated by ServiceAccount, Priority, and DefaultTolerationSeconds Admission Controller. As show below
When attempting to create a Pod within a namespace that does not exist in the cluster, a 404 response will be returned. This is handled by the NamespaceLifecycle Admission Controller.
Admission controllers perform deep inspection of a given request (including content). Some controllers, such as the NamespaceLifecycle Admission Controller, validate the request and determine its admissibility. Others, like the ServiceAccount, Priority, and DefaultTolerationSeconds Admission Controllers, may mutate the content of the request.
All of the Admission Controllers are compiled-in kube-apiserver. For more detailed information, please click the link provided.
🧿 Admission internals
Admission is a module in k8s.io/apiserver pkg/admission. In which these interfaces are exposed to apiserver developer
- An admission controller must implement Interface.Handles to decide whether to handle the incoming operation (CREATE, UPDATE, DELETE, or CONNECT), and MutationInterface or ValidationInterface to make an admission decision.
- The MutationInterface is also an admission Interface (by nesting). Its function
Admitmakes an admission decision, and is allowed mutate the request object
- The ValidationInterface is quite similar to MutationInterface, but its function
Validateis not allowed to mutate the request object
All admission controllers are implementation of MutationInterface or ValidationInterface. MutationInterface implementation can mutate and validate request content, but ValidationInterface implementation can only validate request content.
The official kube-apiserver has many built-in admission plugins on top of the admission package, and here’s the detail of built-in admission plugins register. These built-in plugins are commonly known as Admission Controllers.
✍️ Write an Admission Controller in apiserver
As admission control is performed after the dispatch of kube-aggregator, the custom apiserver should implement admission controllers by itself.
👻👻👻 If interested in K8s apiserver aggregation, you can read this post: K8s apiserver aggregation internals.
Implementing the admission interfaces, then registering it to apiserver, that’s all for writing an Admission Controller in apiserver.
As x-kubernetes commit: add admission shows
Firstly write the Admission Controller implements admission.ValidationInterface under pkg/admisssion
Then register Admission Controller to Admission options and enactive admission module in apiserver
Things happen in kube-apiserver are quite similar to it in custom apiserver
- built-in admission plugins are where the Admission Controller codes in
- built-in admission plugins register are codes about plugin register
k8s.io/apiserver pkg/server/options.NewAdmissionOptions provides a convinient way for admission module initialization, which register these necessary admission controllers for custom apiserver
NamespaceLifecycle is mostly used for custom object creation. The others are intended for dynamic policies.
🔮 Webhook and Policy
All of the Admission Controllers are compiled-in kube-apiserver, with three notable guys
MutatingAdmissionWebhook and ValidatingAdmissionWebhook controller make third-party admission controller possible. This two controller dynamically load and unload webhook configurations at runtime. The actual admisson controls are delegated to webhooks configured.
Applying the MutatingWebhookConfiguration below will install a mutating webhook to cluster. Matching requests (any request with action create or update of any resources) will be forwarded to and reviewed by this webhook.
During admission control, MutatingAdmissionWebhook controller sends a admission/v1 AdmissionReview to this mutating webhook, with comming request in AdmissionReview.Request. The webhook decodes the object from request, provides its mutation as JSONPatch in AdmissionReview.Response, and returns an admission/v1 AdmissionReview back to the apiserver.
If mutation and validation succeed, return AdmissionReview with Response field
allowed: true, otherwise with
Request| webhook-server webhook-server --> |AdmissionReview
result:...jsonpatch| apiserver webhook-server --> |mutate/validate
MutatingAdmissionWebhook admission controller calls any mutating webhooks which match the request. Matching webhooks are called in serial; each one may modify the object if it desires.
ValidatingAdmissionWebhook admission controller calls any validating webhooks which match the request. Matching webhooks are called in parallel; if any of them rejects the request, the request fails.
Validating webhooks must not mutate the object. So mutating webhooks can work as validating webhooks, but not vice versa.
Mutating webhooks are similar to the built-in controller implements admission.MutationInterface, while validating webhooks are similar to the built-in controller implements admission.ValidationInterface.
Admission webhooks are super powerful in K8s. A well-known example of mutating webhook is istio’s automatic sidecar injection.
Once OPA/Gatekeeper or Kyverno deployed, user simply need to provide they policies as custom resources. Webhook module of OPA/Gatekeeper and Kyverno will load user-defined policies and apply mutations/validations to the comming reuqest.
A Policy management system can enforce best practice for K8s usage, thereby improving cluster security, among other benefits. Out-of-the-box policies widely used in K8s community can be found at OPA/Gatekeeper Library and Kyverno Policies. Some of them are:
- Restrict Image Registries
- Pod Security Policies, such as enforce Pod spec.allowPrivilegeEscalation to false
- Automount Service Account Token for Pod
- Disallows all Services with type LoadBalancer
- Enforce Container Resources
- Add Certificates as a Volume
- Add Tolerations
- Disallow CRI socket mounts
For custom resources by custom apiserver, apply admisson control is simple and straightforward. For custom resources by CRDs, the only option is the admission webhook. Does any constraits, like validate resource field, deserve to implement and deploy a admission webhook?
In v1.26 Kubernetes imports Validating Admission Policy (powered by controller ValidatingAdmissionPolicy).
Validating admission policies offer a declarative, in-process alternative to validating admission webhooks.
ValidatingAdmissionPolicy is the official policy engine that use the Common Expression Language (CEL) to declare the validation rules of a policy. Compared to OPA/Gatekeeper and Kyverno, Validating Admission Policy are really at early stage now (2023-7). Out-of-the-box policies are rare.
Usage details can be found at Kubernetes Documentation: Validating Admission Policy.
Repo douglasmakey/k8s-validating-admission-policy also provides good example.
Repo kubescape/cel-admission-library contains out-of-the-box policies.
✅ The relationship between Validating Admission Policy and popular policy engines (such as OPA/Gatekeeper, Kyverno) is not a matter of choosing one over the other, it can be a complementary relationship. OPA/Gatekeeper have integration with K8s Validating Admission Policy.
✍️ Write an Admission Webhook
My repo denyenv-validating-admission-webhook is a validating webhook example.
//todo add mutating webhook example
📖 Further Reading
Author Zeng Xu
LastMod 2023-06-19 23:19
License 本作品采用 知识共享署名-非商业性使用-禁止演绎 4.0 国际许可协议 进行许可，转载时请注明原文链接。