Vault is a dependency you notice only when it is down. A sealed node still has a running process and an open port, so a simple "service is up" check stays green while applications fail to read secrets. This post covers the metrics that show those states and five alert rules in PromQL.

The examples use a free Grafana dashboard, HashiCorp Vault Lite: https://grafana.com/grafana/dashboards/26039/ . Every metric name below comes from the dashboard JSON.

Expose the metrics

Vault serves Prometheus metrics at /v1/sys/metrics?format=prometheus. Set prometheus_retention_time (for example 60s or more) in the telemetry stanza, otherwise there is nothing to scrape. Either give Prometheus a token with read access to sys/metrics, or enable unauthenticated_metrics_access in the listener's telemetry block (this exposes metrics to anyone who can reach the port).

scrape_configs: - job_name: vault metrics_path: /v1/sys/metrics params: format: ["prometheus"] authorization: credentials: YOUR_VAULT_METRICS_TOKEN static_configs: - targets: ["vault-1:8200", "vault-2:8200", "vault-3:8200"]

Scrape every node, not just the load balancer, or you cannot tell a sealed standby from a healthy one.

What to watch

Seal state. vault_core_unsealed is 1 when a node is unsealed and 0 when sealed. If a node is gone entirely its series disappears instead of dropping to zero, so compare the number of reporting nodes with the number you expect.

Leader and standby. vault_core_active is 1 on the active node and 0 on standbys. A healthy HA cluster has exactly one active node. Zero means the cluster is not serving requests; a leader that keeps moving between nodes points at network or storage trouble.

Tokens and leases. vault_token_count and vault_expire_num_leases show how many objects Vault has to track until they expire. Steady growth often means clients that log in on every request, or dynamic secrets that are never revoked while the TTL is long. These values come from the active node, so the JSON uses max rather than summing across nodes.

Request latency. vault_core_handle_request is a summary in milliseconds with quantile labels (0.5, 0.9, 0.99) plus _sum and _count. Quantiles are computed inside Vault and cannot be aggregated exactly across nodes, so the JSON takes max as a conservative "worst node" view. Read it next to requests per second: rate(vault_core_handle_request_count[5m]).

Audit failures. vault_audit_log_request_failure and vault_audit_log_response_failure. By default Vault refuses to serve a request if no audit device can record it, so audit failures can mean unavailability, not just missing logs. The healthy value is zero.

Five alert rules

The thresholds are starting points. Tune them to your own traffic.

groups: - name: vault rules: - alert: VaultSealed expr: vault_core_unsealed == 0 for: 1m labels: {severity: critical} - alert: VaultActiveNodeCount expr: sum(vault_core_active) != 1 for: 2m labels: {severity: critical} - alert: VaultHighLatencyP99 expr: max(vault_core_handle_request{quantile="0.99"}) > 500 for: 10m labels: {severity: warning} - alert: VaultAuditFailures expr: | sum(increase(vault_audit_log_request_failure[5m])) + sum(increase(vault_audit_log_response_failure[5m])) > 0 labels: {severity: critical} - alert: VaultLeaseCountHigh expr: max(vault_expire_num_leases) > 100000 for: 15m labels: {severity: warning}

Why these:

  1. Sealed pages on any node at zero; for: 1m skips the short window of a planned restart.
  2. Active node count catches both no leader and more than one. If every node vanishes from the scrape, sum returns nothing and the rule stays silent, so add a separate up{job="vault"} == 0 rule.
  3. p99 latency is in milliseconds; 500 is a placeholder. for: 10m filters isolated slow requests.
  4. Audit failures has no for, because a single failed write already matters.
  5. Lease count uses a placeholder of 100000. Look at your usual level on the "Leases over time" panel and set the threshold well above it. The same idea works for vault_token_count.

I left out alerts on vault_runtime_alloc_bytes, vault_runtime_sys_bytes and vault_runtime_num_goroutines. Those panels in the JSON help during incident analysis, but sensible thresholds depend on cluster size.

Install the dashboard

In Grafana go to Dashboards, New, Import, enter ID 26039 (https://grafana.com/grafana/dashboards/26039/) and pick your Prometheus data source. Grafana 10.0 or newer is required. The job and instance variables let you narrow the view to one cluster.

I also keep ready-made alert rules and Telegram routing in a bot: https://t.me/devkit_shop_bot?start=gf_vault_article

Summary

Four questions on one screen cover most of it: are all nodes unsealed, is exactly one active, are tokens and leases growing, and are audit writes failing. Everything else is detail for debugging.