Prometheus Architecture in Kubernetes

• Prometheus is a monitoring system that collects metrics from applications and infrastructure, stores them as time-series data, and evaluates queries and alerting rules.
• Prometheus is primarily a Pull-Based Monitoring System so No agent credentials in targets, No push client libraries needed for most services.

• Pull model disadvantages:

  • Short-lived jobs: Batch jobs that run and exit in under one scrape interval can't be scraped. Solution: Pushgateway
  • Firewall/NAT traversal: Prometheus needs network access to all targets. In complex multi-datacenter setups this can be complex. Solution: remote_write + federation

• Its architecture consists of several components:

  1. Prometheus Server - Scrapes and stores metrics.
  2. TSDB - Stores time-series data.
  3. Service Discovery - Finds monitoring targets automatically.
  4. Exporters / Applications - Expose metrics.
  5. PromQL / Rule Engine - Queries metrics and evaluates rules.
  6. Alertmanager - Handles alert notifications.
  7. Grafana - Visualizes metrics in dashboards.

Pull model (Prometheus):

 +--------------------+ | Prometheus | | | | Scrape Manager | +--------------------+ | | HTTP GET /metrics | Every 15 seconds v +--------------------+ | Application | | | | Metrics :8080 | +--------------------+ | | HTTP Response | Metrics data v +--------------------+ | Prometheus | | TSDB | +--------------------+ 

Kubernetes deployment architecture:

monitoring namespace | +-- prometheus Deployment / StatefulSet | +-- prometheus-config ConfigMap | +-- prometheus-rules ConfigMap | +-- prometheus-data PersistentVolumeClaim | +-- node-exporter DaemonSet | +-- one Pod per node | +-- kube-state-metrics Deployment | +-- alertmanager Deployment / StatefulSet | +-- grafana Deployment +-- grafana-datasources ConfigMap +-- grafana-dashboards ConfigMap 

What a Metric Is: Reading a Pod's /metrics

The shop app is running. Is it busy? Is it failing? Instead of guessing from logs, the app counts what it does and shows the numbers on a web page, usually at /metrics. Each of those numbers is a metric. Prometheus reads that page every few seconds and stores the numbers over time.

Reading a Service's page with kubectl

kubectl get --raw can fetch any Service's URL through the API server:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/shop:8080/proxy/metrics # HELP http_requests_total Requests handled by the shop, by path and status code. # TYPE http_requests_total counter http_requests_total{path="/",code="200"} 1284 http_requests_total{path="/cart",code="200"} 311 http_requests_total{path="/cart",code="404"} 9 http_requests_total{path="/checkout",code="200"} 87 http_requests_total{path="/checkout",code="500"} 4 http_requests_total{path="/checkout",code="502"} 2 http_requests_total{path="/pay",code="503"} 1 # HELP shop_orders_in_progress Orders being processed right now. # TYPE shop_orders_in_progress gauge shop_orders_in_progress 3 

The path follows one pattern: /api/v1/namespaces/NAMESPACE/services/NAME:PORT/proxy/PATH.

How to read it

Every line is one of three things:

So http_requests_total{path="/cart",code="404"} 9 , reads: requests to /cart that got a 404= 9 so far.

Counter or gauge

The # TYPE line tells you how a number behaves:

Labels make series

Series (time series) is a metric with a unique combination of labels, whose value is tracked over time.
Imagine your application exposes these metrics:

http_requests_total{path="/cart",code="200"} 311 http_requests_total{path="/cart",code="404"} 9 http_requests_total{path="/checkout",code="200"} 87 

All three belong to the same metric: http_requests_total
But they represent 3 different series because their label combinations are different.

Why is it called a time series?
Prometheus scrapes your application's /metrics endpoint repeatedly, collecting a new value for each series.
For example, this is one series: http_requests_total{path="/cart",code="200"}

http_requests_total is one metric but 7 is a series: one per path and code combination. To count them, count the series lines and skip the comments:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/shop:8080/proxy/metrics | grep -c '^http_requests_total{' 7 # ^ means the line must start with http_requests_total{. # -c counts matching lines. 

To see a slice, filter on a label:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/shop:8080/proxy/metrics | grep 'path="/cart"' http_requests_total{path="/cart",code="200"} 311 http_requests_total{path="/cart",code="404"} 9 

Common mistakes

• Counting the # HELP and # TYPE lines as series. They're comments. Count only lines that start with the metric name and {.
• A typo in the proxy path. You get Error from server (NotFound): the server could not find the requested resource. Check the namespace, the Service name and the port against kubectl get svc -n monitoring.
• Reading a counter's value as "per second". It's a running total since the app started.

Deploying Prometheus: ConfigMap, Deployment, Service

Prometheus: every few seconds it scrapes (fetches) each app's /metrics page and stores every value with a timestamp. Then you can ask how many errors in the last hour?

On Kubernetes, Prometheus is just three objects: a ConfigMap with its settings, a Deployment that runs it, and a Service to reach it.

The configuration file

Prometheus reads one file, prometheus.yml:

global: scrape_interval: 5s evaluation_interval: 5s rule_files: - /etc/rules/*.yml 

• scrape_interval: Controls how frequently Prometheus collects metrics from your applications. Real setups often use 15–30 seconds.

• evaluation_interval: Controls how frequently Prometheus evaluates its configured rules.

  • There are two main types of rules:
    1. Recording rules: Precalculate PromQL expressions and store the results as new metrics.
    2. Alerting rules: Evaluate conditions that can trigger alerts.

• rule_files: Tells Prometheus where to find the recording and alerting rule definitions. In this course they come from a ConfigMap called prometheus-rules, mounted at /etc/rules.

Turn the file into a ConfigMap:

kubectl create configmap prometheus-config -n monitoring --from-file=prometheus.yml 

The Deployment and the Service

apiVersion: apps/v1 kind: Deployment metadata: name: prometheus namespace: monitoring spec: replicas: 1 selector: matchLabels: app: prometheus template: metadata: labels: app: prometheus spec: containers: - name: prometheus image: prom/prometheus:v3.13.4 args: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus ports: - containerPort: 9090 volumeMounts: - name: config mountPath: /etc/prometheus - name: rules mountPath: /etc/rules volumes: - name: config configMap: name: prometheus-config - name: rules configMap: name: prometheus-rules --- apiVersion: v1 kind: Service metadata: name: prometheus namespace: monitoring spec: selector: app: prometheus ports: - port: 9090 targetPort: 9090 

The ConfigMap is mounted where the image looks for its config, /etc/prometheus. Apply it and wait:

$ kubectl get pods -n monitoring NAME READY STATUS RESTARTS AGE prometheus-6689c68d6b-2lf6b 1/1 Running 0 12s shop-76cb9f89b9-sgwgg 1/1 Running 0 28s $ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/-/ready Prometheus Server is Ready. 

Ready, but scraping nothing

Ask Prometheus for its targets. Nothing comes back:

kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}' 

Nothing in the file says what to scrape. A scrape job does, under scrape_configs in prometheus.yml :

global: scrape_interval: 5s evaluation_interval: 5s rule_files: - /etc/rules/*.yml scrape_configs: - job_name: shop static_configs: - targets: - shop.monitoring.svc:8080 

The target is the shop Service's DNS name and port. After changing the ConfigMap, restart Prometheus: it reads its file only when it starts.

kubectl create configmap prometheus-config -n monitoring --from-file=prometheus.yml --dry-run=client -o yaml | kubectl replace -f - kubectl rollout restart deployment/prometheus -n monitoring 

Now the target is there, and healthy:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}' {"job":"shop","health":"up","scrapeUrl":"http://shop.monitoring.svc:8080/metrics","lastError":""} 

In Prometheus, up is a built-in metric that tells you whether Prometheus successfully scraped a target's /metrics endpoint.
It has only two values:

• up = 1 → Prometheus successfully collected metrics from the target.
• up = 0 → Prometheus tried to collect metrics, but the scrape failed.

kubectl get --raw \ '/api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/query?query=up' up{job="shop",instance="shop.monitoring.svc:8080"} 1 # job="shop" => The scrape job configured in prometheus.yml # instance="shop.monitoring.svc:8080" => The target Prometheus scrapes # 1 => Last scrape succeeded 

Common mistakes

• Changing the ConfigMap and stopping there. The running Prometheus still uses the old file. Restart the Deployment.
• Mounting the config at the wrong path. Prometheus then starts with its built-in default file. Check the loaded config at /api/v1/status/config.
• kubectl create configmap on a ConfigMap that exists fails with "already exists". Pipe --dry-run=client -o yaml into kubectl replace -f - instead.

A Target That Is DOWN

Empty graphs, running Prometheus

Monday, 9 a.m. The shop dashboard is blank. Prometheus is up, so the problem is somewhere between Prometheus and the app. Prometheus already knows what went wrong: it records the result of every scrape. You only have to ask.

Ask the targets page

/api/v1/targets lists every target with its health and the last error. jq trims it to what matters:

kubectl get --raw \ /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets \ | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}' # kubectl get --raw, This sends a request through the Kubernetes API server to the Prometheus Service. # jq filters the JSON response. # .data.activeTargets[] selects each active target. # job: .labels.job gets its job name. # health gets its scrape status. # scrapeUrl gets the URL Prometheus is scraping. # lastError gets the latest scrape error. # -c prints each JSON result on one line. {"job":"prometheus","health":"up","scrapeUrl":"http://localhost:9090/metrics","lastError":""} {"job":"shop","health":"down","scrapeUrl":"http://shop.monitoring.svc:8081/metrics","lastError":"Get \"http://shop.monitoring.svc:8081/metrics\": context deadline exceeded"} # http://localhost:9090/metrics => Scraping its own metrics successfully # http://shop.monitoring.svc:8081/metrics => Prometheus did not complete the scrape before its timeout. 

Two fields tell the story:

• scrapeUrl: the exact address Prometheus is calling, here port 8081.
• lastError: what happened. context deadline exceeded means nothing answered in time.

The up metric says the same thing as a number, 0 for the shop:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=up' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}' {"metric":{"__name__":"up","instance":"localhost:9090","job":"prometheus"},"value":[1792188858.578,"1"]} {"metric":{"__name__":"up","instance":"shop.monitoring.svc:8081","job":"shop"},"value":[1792188858.578,"0"]} 

Compare with the truth

What port does the shop actually listen on? The Service says:

$ kubectl get svc shop -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE shop ClusterIP 10.109.73.210 <none> 8080/TCP 15s 

The Service is on 8080, and Prometheus calls 8081. The app is fine, and the scrape configuration is wrong. So the fix belongs in prometheus-config, not in the app.

Fix, reload, confirm

Correct the target in the ConfigMap, then restart Prometheus so it reads the file again (lesson 2). Ask the targets page once more:

$ kubectl get --raw /api/v1/namespaces/monitoring/services/prometheus:9090/proxy/api/v1/targets | jq -c '.data.activeTargets[] | {job: .labels.job, health, scrapeUrl, lastError}' {"job":"prometheus","health":"up","scrapeUrl":"http://localhost:9090/metrics","lastError":""} {"job":"shop","health":"up","scrapeUrl":"http://shop.monitoring.svc:8080/metrics","lastError":""} 

This is the loop for every scrape problem: read lastError, compare the address with the Service, fix the config, reload, then check again.

Common mistakes

• "Fixing" the Service to match the wrong port. The app was healthy. Changing it breaks everything else that uses port 8080.
• Editing the ConfigMap and checking right away. Without a restart the old address stays, and the target stays DOWN.
• Reading only health. down says that it failed. lastError says why: a timeout, connection refused, or an HTTP error such as 404 each point somewhere different.

In production

• An alert on up == 0 is often the first one a team writes. A target that silently stops reporting is worse than one that reports errors, because every graph that depends on it just goes flat.
• up = 0 means scraping failed, not necessarily that the application itself is down

Node Metrics: node-exporter as a DaemonSet

The app is fine. Is the machine?

• The shop looks healthy, but checkout is slow. Maybe a node is out of memory, or its CPU is pegged. Apps don't report that; the machine does. node-exporter reads the node's own counters (CPU, memory, disk, network) and serves them as metrics on port 9100.
• You need exactly one per node, so the right object is a DaemonSet: Kubernetes runs one pod on every node, including nodes added later

Measuring the node, not the pod

A normal pod sees its own little world: its own network, its own /proc. To report on the machine, node-exporter needs the machine's view:

apiVersion: apps/v1 kind: DaemonSet metadata: name: node-exporter namespace: monitoring spec: selector: matchLabels: app: node-exporter template: metadata: labels: app: node-exporter spec: hostNetwork: true tolerations: - operator: Exists # you haven't specified a key or effect, it matches all taint keys, values, and effects. containers: - name: node-exporter image: prom/node-exporter:v1.12.1 args: - --path.procfs=/host/proc - --path.sysfs=/host/sys ports: - containerPort: 9100 volumeMounts: - name: proc mountPath: /host/proc readOnly: true - name: sys mountPath: /host/sys readOnly: true volumes: - name: proc hostPath: path: /proc - name: sys hostPath: path: /sys --- apiVersion: v1 kind: Service metadata: name: node-exporter namespace: monitoring spec: clusterIP: None selector: app: node-exporter ports: - port: 9100 targetPort: 9100 

Note

1. Understand Taints and Tolerations

• A Taint is applied to a Node to restrict which Pods can run on it.
• For example, Kubernetes control-plane nodes commonly have this taint: Taints: node-role.kubernetes.io/control-plane:NoSchedule
• This means ordinary Pods cannot be scheduled onto this node unless they have a matching toleration.

What does operator: Exists mean?
Normally, you could write a specific toleration:

tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule 

This means:

• key: Match the control-plane taint.
• operator: Exists: Match the taint based on its key, regardless of its value.
• effect: NoSchedule: Match only taints with the NoSchedule effect.

But your configuration is:

tolerations: - operator: Exists 

Since you haven't specified a key or effect, it matches all taint keys, values, and effects.

2. Normal Service (ClusterIP) and a Headless Service
Normal Service (ClusterIP):

apiVersion: v1 kind: Service metadata: name: shop namespace: monitoring spec: type: ClusterIP selector: app: shop ports: - port: 8080 targetPort: 8080 

Imagine you have three Shop Pods, When a client resolves shop.monitoring.svc.cluster.local, DNS returns the Service's ClusterIP.
shop.monitoring.svc.cluster.local → 10.96.0.100

 +----------------------+ | Client | | shop:8080 | +----------------------+ | v +----------------------+ | ClusterIP Service | | 10.96.0.100 | +----------------------+ | +-------------+-------------+ | | | v v v +---------------+ +---------------+ +---------------+ | Shop Pod 1 | | Shop Pod 2 | | Shop Pod 3 | | 10.244.1.2 | | 10.244.1.3 | | 10.244.1.4 | +---------------+ +---------------+ +---------------+ 

Kubernetes Service routing then directs connections to one of the available Pods.

Headless Service:

apiVersion: v1 kind: Service metadata: name: node-exporter namespace: monitoring spec: clusterIP: None selector: app: node-exporter ports: - port: 9100 targetPort: 9100 

 +------------------------+ | Prometheus | | DNS lookup | +------------------------+ | v +------------------------+ | Headless Service | | Returns endpoint IPs | +------------------------+ | +---------------+---------------+ | | | v v v +----------------+ +----------------+ +----------------+ | Exporter 1 | | Exporter 2 | | Exporter 3 | | 192.168.1.11 | | 192.168.1.12 | | 192.168.1.13 | +----------------+ +----------------+ +----------------+ 

Imagine Node Exporter is running on three nodes, DNS can return multiple IP addresses:

node-exporter.monitoring.svc.cluster.local → 192.168.1.11 → 192.168.1.12 → 192.168.1.13 

Because your Node Exporter uses hostNetwork: true, these endpoint IPs can be the nodes' IP addresses.

One pod per node

$ kubectl get ds -n monitoring NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE node-exporter 2 2 2 2 2 <none> 26s  $ kubectl get pods -n monitoring -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES node-exporter-5jpsp 1/1 Running 0 27s 192.168.67.3 cb-mon-m02 <none> <none> node-exporter-gksbh 1/1 Running 0 27s 192.168.67.2 cb-mon <none> <none> prometheus-78c96876b5-h98jj 1/1 Running 0 44s 10.244.1.12 cb-mon-m02 <none> <none> 

DESIRED 2 matches the 2 nodes. The node-exporter pods have the nodes' IPs (192.168.67.x), while Prometheus has a pod IP (10.244.x.x). That's hostNetwork at work.

Why headless

• Normal Service: Provides one virtual IP and load-balances traffic across backend Pods.
• Headless Service: Does not provide a virtual ClusterIP. DNS can return the IP addresses of individual Pods instead.
• Prometheus must scrape every node, not whichever pod a normal Service picks. The job in this lab's config asks DNS for the headless Service's name and gets every pod's address back, one target each:

- job_name: node-exporter dns_sd_configs: - names: ['node-exporter.monitoring.svc.cluster.local'] type: A port: 9100 refresh_interval: 5s 

What you get

Hundreds of metrics per node:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- node-exporter.monitoring.svc:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes|network_receive_bytes_total)' | head -6 node_load1 7.14 node_load15 8.94 node_memory_MemAvailable_bytes 3.824058368e+09 node_network_receive_bytes_total{device="docker0"} 0 node_network_receive_bytes_total{device="eth0"} 5.8914394e+07 node_network_receive_bytes_total{device="lo"} 2.9718904e+07 

Common mistakes

• A Deployment instead of a DaemonSet. You get some number of pods, placed anywhere, not one per node.
• No tolerations. The control-plane node gets no pod, and its target is missing.
• A normal (non-headless) Service. DNS returns one virtual IP, so Prometheus sees one target and scrapes a random node each time.
• Forgetting the hostPath mounts. The numbers come from the container, not the machine, and look plausible but are wrong.

In production

node-exporter is the standard way to watch Linux machines, inside Kubernetes or not. On real nodes its memory and disk metrics feed some of the most important alerts, such as a disk filling up.

PromQL Selectors and a Recording Rule

PromQL, Prometheus's query language, picks the right series and adds them up.

Asking Prometheus a question

Queries go to /api/v1/query. In the labs, run them from inside the Prometheus pod with wget:

$ kubectl exec deploy/prometheus -n monitoring -- wget -qO- --post-data 'query=http_requests_total' localhost:9090/api/v1/query | jq -c '.data.result[] | {metric, value}' {"metric":{"__name__":"http_requests_total","code":"200","instance":"shop.monitoring.svc:8080","job":"shop","path":"/"},"value":[1792188952.116,"1284"]} # 1792188952.116 => Timestamp - Unix timestamp {"metric":{"__name__":"http_requests_total","code":"200","instance":"shop.monitoring.svc:8080","job":"shop","path":"/cart"},"value":[1792188952.116,"311"]} {"metric":{"__name__":"http_requests_total","code":"404","instance":"shop.monitoring.svc:8080","job":"shop","path":"/cart"},"value":[1792188952.116,"9"]} {"metric":{"__name__":"http_requests_total","code":"200","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.116,"87"]} {"metric":{"__name__":"http_requests_total","code":"500","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.116,"4"]} {"metric":{"__name__":"http_requests_total","code":"502","instance":"shop.monitoring.svc:8080","job":"shop","path":"/checkout"},"value":[1792188952.116,"2"]} {"metric":{"__name__":"http_requests_total","code":"503","instance":"shop.monitoring.svc:8080","job":"shop","path":"/pay"},"value":[1792188952.116,"1"]} 

Notice two labels the app never wrote: job and instance. Prometheus adds them to every series it scrapes, so you always know where a number came from.

Selectors: picking series by label

Put conditions in braces after the name:

5.. means "a 5 and...