If you apply the Pod Security Admission Baseline label with --dry-run=server before enforcing it on the namespace that runs node-exporter (a namespace is the unit Kubernetes uses to partition resources), node-exporter's Pod shows up as a violation.

For node-exporter installed from the prometheus-node-exporter Helm chart with default settings, the warning looks like this.1

$ kubectl label --dry-run=server --overwrite ns ne-base pod-security.kubernetes.io/enforce=baseline Warning: existing pods in namespace "ne-base" violate the new PodSecurity enforce level "baseline:latest" Warning: prometheus-node-exporter-rrrlc: host namespaces, hostPath volumes, hostPort namespace/ne-base labeled (server dry run) 

host namespaces, hostPath volumes, and hostPort, listed after the Pod name, are the settings Baseline does not allow. Remove all three from the DaemonSet and the dry-run no longer warns about node-exporter's Pod.

With the three removed, though, some of node-exporter's network metrics keep their names but report the node-exporter Pod's values instead of the node's. Filesystem values also come from the container's mounts instead of the node's disks.

For example, even when the node's conntrack table (the list of connections the kernel is tracking) is above the threshold, the existing alert on its usage does not fire.

The sections below take each item in the warning and describe what node-exporter reads through that setting and which metrics stop being node values when it is removed.

Next, for environments that have already removed them and run under Baseline, a section shows how to tell from the DaemonSet's settings which metrics are no longer node values. The last section describes how to keep the items and place node-exporter's namespace outside Baseline.

The two setups, one with the items removed to run inside Baseline and one with node-exporter's namespace moved outside Baseline, look like this.

Setup with the items removed, running inside Baseline

Setup with node-exporter's namespace moved outside Baseline

What is happening

This section goes through the three items in the warning in order, covering what node-exporter reads through each setting and how to remove it. It then shows, for each removal, which values stop being node values and what that does to alerts.

host namespaces

Baseline's host namespaces forbids hostNetwork, hostPID, and hostIPC.2 The chart enables hostNetwork and hostPID by default.

hostNetwork makes the Pod use the node's network (the node's IP addresses and interfaces) as is. The traffic and connection values node-exporter reads are then the node's.

Even when read from the node's /proc mounted into the Pod, these values come from the network namespace of the process that reads them.3 So removing hostNetwork turns them into the Pod's values.

Values that exist once for the whole kernel (socket memory, the conntrack limit) and interface attributes read from /sys stay as the node's.

hostPID lets the Pod see every process on the node. node-exporter reads node values from the node's /proc mount, so hostPID does not change where it reads from.4 Removing it changes none of the values in the table shown later.

To clear host namespaces from the warning, remove both. In the settings passed to the chart (values), write:

hostNetwork: false hostPID: false 

hostPath volumes

hostPath volumes mounts a directory on the node into the Pod. By default the chart mounts the node's /proc and /sys and the node's root filesystem.5

node-exporter reads kernel values from the mounted /proc and /sys. From the root filesystem it reads per-mount-point values and the disks' udev information (/run/udev/data).

Only the root filesystem mount can be removed through values.6

hostRootFsMount: enabled: false 

The values hostProcFsMount and hostSysFsMount only have a mountPropagation setting, so the /proc and /sys mounts cannot be removed and stay. Removing the root mount therefore still leaves hostPath volumes in the warning.

hostPort

hostPort binds a port on the node's IP to a port on the Pod.

A Pod with hostNetwork enabled listens directly on the node's IP, so Kubernetes gives that Pod a hostPort with the same number as the container port.7 That is why hostPort appears in the warning even though the values have no hostPort setting.

node-exporter listens on this port on the node's IP, and Prometheus scrapes metrics from there.

Removing hostNetwork clears hostPort from the warning. node-exporter then listens on the Pod's IP, and Prometheus scrapes the Pod's IP.

Metrics after removal

The table columns refer to these four states.

  • Default: the chart's default settings
  • hostNetwork: only hostNetwork set to false
  • Root: only the root filesystem mount removed through values
  • Baseline: hostNetwork, hostPID, and the root filesystem mount removed through values, with the /proc and /sys mounts also removed

In the table, "Node" means the node's value and "Pod" means the node-exporter Pod's value. The nodename and instance rows show whether the label holds the node's or the Pod's name, and the node's or the Pod's IP.

Value Default hostNetwork Root Baseline
CPU, memory, disk Node Node Node Node
Network traffic Node Pod Node Pod
Connections, sockets Node Pod Node Pod
Socket memory Node Node Node Node
conntrack count Node Pod Node Pod
conntrack limit Node Node Node Node
Interface attributes Node Node Node Pod
Filesystem Node Node Partial errors Container mounts
udev labels Node Node Empty labels Empty labels
nodename Node name Pod name Node name Pod name
instance Node IP Pod IP Node IP Pod IP

These are the main metrics checked for each row.

Row Main metrics
CPU, memory, disk node_cpu_seconds_total, node_memory_MemTotal_bytes, node_disk_read_bytes_total
Network traffic node_network_receive_bytes_total
Connections, sockets node_netstat_Tcp_CurrEstab, node_sockstat_TCP_inuse, node_sockstat_sockets_used
Socket memory node_sockstat_TCP_alloc, node_sockstat_UDP_mem
conntrack count and limit node_nf_conntrack_entries, node_nf_conntrack_entries_limit
Interface attributes node_network_info
Filesystem node_filesystem_device_error, node_filesystem_size_bytes
udev labels Labels such as model and serial on node_disk_info
nodename The nodename label on node_uname_info
instance The address Prometheus scraped, which it attaches to every metric

The following three cell terms mean these states.

Cell term State
Partial errors The node's mount points are listed, but for mount points other than /, such as /boot and /run, node_filesystem_device_error is 1 and node_filesystem_size_bytes is missing
Container mounts Instead of the node's mount points, only the container's own mounts such as /etc/hosts are listed, and / is missing as well
Empty labels node_disk_info is present, but labels such as model and serial are empty

Alerts built on rows that become Pod values evaluate the Pod's values, not the node's. For example, the kube-prometheus-stack alert on conntrack usage uses this expression.8

(node_nf_conntrack_entries{job="node-exporter"} / node_nf_conntrack_entries_limit) > 0.75 

With hostNetwork removed, the count becomes the Pod's value, 0, while the limit stays the node's value.9 Even with the node's table filled past the threshold, the expression returns nothing and the alert does not fire.

Telling which state your environment is in

You can tell which state in the previous section's table your node-exporter is in from two things in the DaemonSet's Pod settings: hostNetwork and the path of the hostPath volumes.

Print the DaemonSet as YAML with the following command and look under spec.template.spec. Replace the namespace and DaemonSet name with your own.

$ kubectl get daemonset -n ne-base prometheus-node-exporter -o yaml 

In a DaemonSet installed with the chart defaults, the relevant lines look like this (... marks omitted lines).

 ...  hostNetwork: true hostPID: true ...  volumes: - hostPath: path: /proc type: "" name: proc - hostPath: path: /sys type: "" name: sys - hostPath: path: / type: "" name: root ... 

A removed setting shows up as a missing line here. Which lines are missing decides which column of the previous section's table applies. Whether hostPID: true is present affects no column, so you can ignore it.

Lines missing from the DaemonSet Column in the previous section's table
hostNetwork: true hostNetwork
path: / Root
hostNetwork: true and path: / Both hostNetwork and Root
The two above plus path: /proc and path: /sys Baseline

When both columns apply, every row that is not a node value in either the hostNetwork column or the Root column is no longer a node value. Combinations not in this table cannot be judged from the previous section's table.

In the column that applies, the rows whose cell is anything other than "Node", "Node name", or "Node IP" are the ones that are no longer node values. Which metrics those rows cover is in the row-to-metric table in the previous section.

What to do

node-exporter that uses the settings in the warning cannot run in a namespace where Baseline is enforced.

PSA sets the enforced level per namespace through a label and cannot exempt individual Pods.10 So put node-exporter alone in a separate namespace and place no restrictions on that namespace.

The level that restricts nothing is called privileged. Create the namespace for node-exporter first and give it the privileged label.11

$ kubectl create namespace monitoring-ne $ kubectl label ns monitoring-ne pod-security.kubernetes.io/enforce=privileged 

If you have already removed items, restore the chart defaults when you move node-exporter into this namespace. For items removed through values, delete those values. For the /proc and /sys mounts, revert the changes you made outside the chart.

If you install the prometheus-node-exporter chart on its own, delete the current release with helm uninstall and reinstall it with -n monitoring-ne.

kube-prometheus-stack puts node-exporter in the same namespace as Prometheus.

Pass the name of the namespace you created to prometheus-node-exporter.namespaceOverride in values and apply it with helm upgrade. This separates node-exporter's namespace from Prometheus's.

prometheus-node-exporter: namespaceOverride: monitoring-ne 

node-exporter's Pod then runs in monitoring-ne.12 If kube-prometheus-stack is installed in monitoring, a Baseline dry-run on monitoring shows no warning.

$ kubectl label --dry-run=server --overwrite ns monitoring pod-security.kubernetes.io/enforce=baseline namespace/monitoring labeled (server dry run) 

After the move, check the DaemonSet with the method from the previous section and confirm that all the default lines are present.

monitoring-ne, where node-exporter now runs, remains an exception to Baseline.

In this namespace, any Pod can use every setting Baseline forbids, including ones that were not in the warning (privileged containers, for example). Anyone who can create Pods there can run a Pod that reaches the node's network, processes, and directories.

So keep only node-exporter in monitoring-ne.

The two setups compare as follows.

Remove the items and run under Baseline Place node-exporter's namespace outside Baseline
Network-related values Pod's values Node's values
conntrack usage alert Does not fire when over the threshold Fires when over the threshold
Filesystem values Container mounts Node's mount points
Scope Baseline does not cover None node-exporter's namespace

If you can restrict who can create Pods, and resources that create Pods such as Deployments, in monitoring-ne, you can place node-exporter's namespace outside Baseline.

In the setup with the items removed, the metric names stay the same, so on a dashboard it is hard to notice that the values have become the Pod's. I would rather keep one namespace as an exception to Baseline than have alerts stop firing without anyone knowing.

  1. Verified on k3s v1.37.1+k3s1, Linux 6.17, and prometheus-node-exporter chart 4.60.0 (node_exporter v1.12.1). The code is in show-your-work and the results are in the GitHub Actions run. ↩

  2. The "Host Namespaces" control under Baseline in Pod Security Standards. Sharing the node's namespaces lets a Pod reach the node's network, processes, and shared memory, which breaks the isolation between the Pod and the node, so it is forbidden. ↩

  3. /proc/net and /proc/sys/net show the contents of the reading process's network namespace. ↩

  4. Reading the node's process information needs only one of hostPID or a /proc mount. The chart uses both, so with hostPID removed it can still read through the /proc mount (helm-charts #5507). ↩

  5. The chart's DaemonSet template. The paths to read from are passed in args. ↩

  6. The chart's values. ↩

  7. Kubernetes defaulting for Pods. ↩

  8. The expression in kube-prometheus-stack's alerting rules. ↩

  9. node-exporter reads the count and the limit from /proc/sys/net/netfilter. In Linux, the count is separate per network namespace and the limit is not. ↩

  10. Pod Security Admission exemptions are set by username, RuntimeClass, or namespace. There is no per-Pod setting. ↩

  11. If the warn and audit labels are also set to baseline, set them to privileged as well. When labeling all namespaces at once, leave this namespace out. ↩

  12. In kube-prometheus-stack chart 92.11.0, namespaceOverride appears in the node-exporter chart's template and kube-prometheus-stack's values. ↩