Every admission webhook you install is a new single point of failure sitting on the critical path of every API write. Gatekeeper, Kyverno, OPA — they all work the same way: the kube-apiserver makes a synchronous HTTPS call to a pod, waits for an answer, and decides what to do when that call fails. ValidatingAdmissionPolicy (VAP) removes that call entirely. The policy is a CEL expression compiled and evaluated inside the API server. No pod, no service, no certificate, no timeout.

This article covers how to write real VAP policies, how to validate them with --dry-run=server and audit annotations, and how to move from Audit to Deny without taking production down on a Friday afternoon.

Why the webhook is the weakest link in your admission chain

A ValidatingWebhookConfiguration forces you into an uncomfortable trade-off through a single field: failurePolicy.

With failurePolicy: Fail, if the Gatekeeper pods are unschedulable, if the service's endpoints are empty, if the CA bundle has rotated out of sync, or if the webhook takes longer than timeoutSeconds, then every matching API write is rejected cluster-wide. That includes writes from controllers trying to repair the cluster. I have seen clusters where the Gatekeeper namespace was drained during a node upgrade, and the webhook then blocked the recreation of its own pods. Classic bootstrap deadlock.

With failurePolicy: Ignore, you get the opposite problem: your security control silently becomes optional. An attacker who can disrupt the webhook — or simply a noisy neighbour saturating a node — gets a free pass. For a compliance auditor, Ignore means the control does not exist.

On top of that you inherit the operational tax: TLS certificate rotation, webhook availability in control-plane upgrade paths, 2–5 ms of latency added to every single request, memory for the policy engine, and a Rego or JSON-schema dialect that nobody on the team reads fluently.

VAP eliminates all of it. The expression is compiled once, type-checked against the OpenAPI schema of the targeted resource, and evaluated in-process with a cost budget. No network hop means no failure mode to engineer around.

ValidatingAdmissionPolicy in two objects

Availability

VAP was beta (admissionregistration.k8s.io/v1beta1) in Kubernetes 1.28 and reached GA (admissionregistration.k8s.io/v1) in 1.30. On OpenShift, 4.15/4.16 ship the beta API, and OpenShift 4.17 (Kubernetes 1.31) ships the GA API. Check what your cluster serves before writing anything:

kubectl api-resources --api-group=admissionregistration.k8s.io oc version

The policy

The ValidatingAdmissionPolicy is the definition: what to match, what to evaluate. It does nothing on its own.

apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: require-container-limits spec: failurePolicy: Ignore matchConstraints: resourceRules: - apiGroups: ["apps"] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["deployments", "statefulsets", "daemonsets"] matchConditions: - name: skip-platform-namespaces expression: >- !request.namespace.startsWith('openshift-') && !request.namespace.startsWith('kube-') - name: skip-system-serviceaccounts expression: >- !request.userInfo.username.startsWith('system:serviceaccount:openshift-') variables: - name: containers expression: "object.spec.template.spec.containers" validations: - expression: >- variables.containers.all(c, has(c.resources) && has(c.resources.limits) && 'memory' in c.resources.limits && 'cpu' in c.resources.requests) message: "each container must set resources.limits.memory and resources.requests.cpu" reason: Invalid

Key points for an advanced reader:

  • matchConstraints is the coarse filter, evaluated by the API server's existing matching machinery (same semantics as a webhook's rules).
  • matchConditions is a CEL-level filter evaluated before validations. Exclusions belong here, not inside your validation expression. A request that fails a match condition is simply not subject to the policy.
  • variables are lazily evaluated and memoized. Use them to keep expressions readable and to stay under the cost budget.
  • reason maps to the HTTP status reason returned to the client (Invalid, Forbidden, Unauthorized, RequestEntityTooLarge).

The binding

The ValidatingAdmissionPolicyBinding is the activation: where the policy applies and what happens on failure. This separation is what makes a safe rollout possible — you ship one definition and change its blast radius with bindings.

apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: require-container-limits-audit spec: policyName: require-container-limits validationActions: ["Audit", "Warn"] matchResources: namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: NotIn values: ["default"]

validationActions accepts Deny, Warn and Audit, and they are additive. Warn returns a Warning: header that kubectl prints. Audit adds an annotation to the API server audit event. Deny rejects the request.

Writing CEL that survives review

The variables available

Inside a validation expression you get object (the incoming object, null on DELETE), oldObject (null on CREATE), request (the AdmissionRequest attributes: namespace, name, operation, userInfo, resource…), params (the bound parameter object), namespaceObject (the full Namespace object — labels and annotations included), and authorizer.

namespaceObject alone replaces a large chunk of Gatekeeper constraint boilerplate: you can read a namespace label directly inside the expression instead of threading it through a constraint selector.

Parameterized policies

Hardcoding an allowlist in a policy means editing the policy for every change — and policies are cluster-scoped objects usually owned by the platform team. Use paramKind with a CRD instead.

apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: registryallowlists.policy.acme.io spec: group: policy.acme.io scope: Cluster names: kind: RegistryAllowList plural: registryallowlists singular: registryallowlist versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object required: ["allowedPrefixes"] properties: allowedPrefixes: type: array items: type: string 
apiVersion: policy.acme.io/v1 kind: RegistryAllowList metadata: name: corporate-registries spec: allowedPrefixes: - "image-registry.openshift-image-registry.svc:5000/" - "quay.io/acme-prod/" - "registry.redhat.io/" 
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: allowed-image-registries spec: failurePolicy: Ignore paramKind: apiVersion: policy.acme.io/v1 kind: RegistryAllowList matchConstraints: resourceRules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE"] resources: ["pods"] matchConditions: - name: skip-platform-namespaces expression: >- !request.namespace.startsWith('openshift-') && !request.namespace.startsWith('kube-') variables: - name: images expression: >- object.spec.containers.map(c, c.image) + (has(object.spec.initContainers) ? object.spec.initContainers.map(c, c.image) : []) - name: denied expression: >- variables.images.filter(i, !params.spec.allowedPrefixes.exists(p, i.startsWith(p))) validations: - expression: "size(variables.denied) == 0" messageExpression: >- "images from untrusted registries: " + variables.denied.join(", ")  reason: Forbidden auditAnnotations: - key: images valueExpression: "variables.images.join(', ')"

Note that this policy matches pods, not Deployments. That is deliberate: matching workload controllers only catches what users submit, while matching pods catches everything that actually runs — including pods created by controllers, Jobs, and the oc debug path. The downside is that the rejection surfaces in a ReplicaSet event instead of at kubectl apply time, which is exactly why you also want a Deployment-level policy for fast developer feedback.

apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: name: allowed-image-registries-deny spec: policyName: allowed-image-registries paramRef: name: corporate-registries parameterNotFoundAction: Deny validationActions: ["Deny"] matchResources: namespaceSelector: matchLabels: policy.acme.io/registries: "enforced"

parameterNotFoundAction: Deny is important: if someone deletes the RegistryAllowList, the policy fails closed instead of silently allowing everything.

Break-glass with the authorizer

The authorizer variable lets a policy ask the cluster's RBAC: "is this user allowed to do X?". That is a clean, auditable exemption mechanism — no annotation that any developer can set.

 matchConditions: - name: not-break-glass expression: >- !authorizer.group('policy.acme.io').resource('policyexemptions') .namespace(request.namespace).check('create').allowed()

Grant create on the virtual policyexemptions resource to the SRE on-call group, and the policy stops applying to them — while every use shows up in the RBAC audit trail.

Rolling out without blocking prod

The whole point of the policy/binding split is that you never have to guess the blast radius.

Step 1 — Create the policy with no binding 

kubectl apply -f policy-require-limits.yaml kubectl get validatingadmissionpolicy require-container-limits \ -o jsonpath='{.status.typeChecking}' | jq .

The API server type-checks every expression against the schema of each matched resource and reports warnings in status.typeChecking.expressionWarnings. A typo such as object.spec.template.spec.container shows up here, at apply time, not at 3 a.m. Treat a non-empty expressionWarnings as a hard failure in CI.

Step 2 — Bind in Audit mode and measure

Apply the binding with validationActions: ["Audit"] and failurePolicy: Ignore. Nothing is rejected; failures are recorded as audit annotations under the key validation.policy.admission.k8s.io/validation_failure.

oc adm node-logs --role=master --path=kube-apiserver/audit.log \ | jq -r 'select(.annotations["validation.policy.admission.k8s.io/validation_failure"] != null) | [.objectRef.namespace, .objectRef.resource, .objectRef.name, .annotations["validation.policy.admission.k8s.io/validation_failure"]] | @tsv' \ | sort | uniq -c | sort -rn | head -30

This is your real violation inventory. Run it for a full business cycle — at minimum one week, ideally across a release window and a cluster upgrade, so that batch jobs and operators get a chance to trip the policy.

Step 3 — Test individual objects with server-side dry-run

--dry-run=server runs the complete admission chain, including VAP, without persisting anything. This is the fastest feedback loop and belongs in your CI pipeline.

cat <<'EOF' > /tmp/bad-pod.yaml apiVersion: v1 kind: Pod metadata: name: vap-canary namespace: team-payments spec: containers: - name: app image: docker.io/library/nginx:latest EOF kubectl apply --dry-run=server -f /tmp/bad-pod.yaml # Error from server (Forbidden): error when creating "/tmp/bad-pod.yaml": # pods "vap-canary" is forbidden: ValidatingAdmissionPolicy 'allowed-image-registries' # with binding 'allowed-image-registries-deny' denied request: # images from untrusted registries: docker.io/library/nginx:latest

Do the same from a developer's identity to validate matchConditions:

kubectl auth can-i --list --as=system:serviceaccount:team-payments:deployer kubectl apply --dry-run=server --as=system:serviceaccount:team-payments:deployer -f /tmp/bad-pod.yaml

Step 4 — Warn, then Deny, namespace by namespace

Add Warn to the binding so developers see the message in their own kubectl output before anything breaks. Then create a second binding scoped with a namespaceSelector and validationActions: ["Deny"], and switch failurePolicy to Fail on the policy. Enforcement is now opt-in per namespace via a label:

oc label namespace team-payments policy.acme.io/registries=enforced

Roll the label out tier by tier. If something breaks, removing one label is an instant, scoped rollback — far safer than deleting a cluster-wide policy.

Migrating from Gatekeeper

The mapping is mechanical:

  • ConstraintTemplate → ValidatingAdmissionPolicy (the Rego/CEL logic plus the parameter schema).
  • Constraint CR (e.g. K8sAllowedRepos) → ValidatingAdmissionPolicyBinding + a param object.
  • spec.match.kinds → matchConstraints.resourceRules.
  • spec.match.excludedNamespaces → matchConditions or binding namespaceSelector.
  • enforcementAction: dryrun → validationActions: ["Audit"].
  • violation[{...}] Rego blocks → one validations entry each, with messageExpression.

Run both engines side by side: keep Gatekeeper in dryrun for the migrated constraints while VAP is in Audit, then compare violation counts. Identical counts means the CEL translation is faithful. Delete the Gatekeeper constraint only after VAP is in Deny.

Gatekeeper still wins in three areas: mutation (MutatingAdmissionPolicy is only progressing through alpha/beta in recent releases), referential constraints that need to list other cluster objects (VAP has no cluster read access — only authorizer checks), and external data providers. Keep a slim Gatekeeper install for those, and move the 80 % of simple structural checks to VAP.

Security best practices

  • Fail closed in production. Once a policy is validated, set failurePolicy: Fail and parameterNotFoundAction: Deny. A policy that cannot evaluate must not become a free pass.
  • Protect the policy objects with RBAC. validatingadmissionpolicies and validatingadmissionpolicybindings are cluster-scoped. Anyone with update on them can disable your controls. Restrict write access to the platform team, and consider a meta-policy that denies DELETE on bindings for non-admin users.
  • Protect the param CRs too. If a namespace admin can edit the RegistryAllowList, the allowlist is decorative. Keep param objects cluster-scoped and RBAC-restricted, or use paramRef with a namespaced kind in a namespace only platform owns.
  • Never trust user-controlled fields for exemptions. Annotations are set by whoever creates the object. Use authorizer checks or namespace labels (which require namespace-edit rights) instead.
  • Match pods, not just controllers, for anything that is a real runtime security control (privileged, hostPath, hostNetwork, image provenance). Controller-created pods bypass Deployment-level policies.
  • Do not exclude openshift-* blindly in your own namespaces policy set. Platform exclusions are necessary, but keep them explicit and reviewed; an attacker who can create a namespace named openshift-foo would otherwise inherit the exemption. Combine the prefix check with a check on request.userInfo.
  • Keep policies in Git and type-check them in CI with kubectl apply --dry-run=server plus a status.typeChecking assertion against a disposable cluster (kind, CRC).
  • Keep the audit query as a detection rule. Even in Deny mode, bindings with Audit still record every rejection. Ship validation.policy.admission.k8s.io/validation_failure events to your SIEM — repeated violations from one service account is a useful signal.

Key takeaways

ValidatingAdmissionPolicy removes a network call, a pod, a certificate and a timeout from the hot path of every API write. That is one less outage class and one less thing to upgrade.

The operational model to remember: the policy defines the rule, the binding defines the risk. Ship the policy first and read status.typeChecking. Bind it in Audit, mine the API server audit log for a full business cycle, add Warn so developers see the message before it hurts, then enable Deny through a namespace label you can remove in one command. Use --dry-run=server and --as at every step to test exactly what the API server will do.

CEL is not Rego, and that is the point: expressions are type-checked against the real OpenAPI schema, cost-bounded, and readable by any engineer on the team. Move your structural checks there, keep a reduced policy engine for mutation and referential logic, and delete the webhook configurations you no longer need.