Prometheus alerting rules turn a PromQL expression into an alert: if the expression returns series, an alert becomes active for each one. Writing the rule is quick. Making it reliable takes longer: choosing the threshold, setting the for duration, attaching a runbook, then proving in continuous integration that the rule fires when it should and only then. This article follows that path with a rule file, a promtool test and the checkpoints an SRE team needs.

It builds on the article on recording rules, whose error-ratio series is the basis of the alert here. The alerts described are evaluated by Prometheus itself, not by Dynatrace.

In brief

  • An alerting rule has alert, expr and, optionally, for, keep_firing_for, labels and annotations.
  • With for, a series must stay active for the stated duration before it moves from pending to firing.
  • promtool check rules validates the files; promtool test rules checks that an alert fires (or does not) on fabricated series.
  • Prometheus does not send notifications itself: it passes alert state to Alertmanager.

Prometheus alerting rules: the structure of a rule

According to the Prometheus documentation on alerting rules, rules are declared in a group, like recording rules. Each alerting rule has:

  • alert: the name of the alert;
  • expr: the PromQL expression, an alert being active for each label set it returns;
  • for: how long an element must stay active before it fires;
  • keep_firing_for: how long the alert keeps firing after its condition was last met;
  • labels and annotations: extra information attached to the alert, which can be templated ($labels, $value, $externalLabels).

The file below defines two alerts. The first reuses the failure ratio from the previous article; the second watches for a target that stops responding.

File alerts.yml: HighErrorRatio alert with for 10m and keep\_firing\_for 5m, and TargetDown alert with for 5m

The 0.05 threshold, the api job name and the runbook URL are example values in this article. The severity labels are likewise a free convention, not a field Prometheus requires.

Tuning for and keep_firing_for

The documentation describes two states. An active element that has not yet reached the for duration is pending. Without for, the alert becomes active on the first evaluation. That alone explains why a bare threshold wakes the on-call engineer for a spike of a few seconds.

for: filter out brief spikes

The page states that for suppresses alerts that only briefly match the condition. A 10-minute duration on an error ratio computed over 5 minutes therefore asks for a lasting problem. The choice is a trade-off: the longer for is, the fewer false positives, but the longer the detection delay (and so the MTTR). Relate it to your SLO. For an alert built on the speed at which the error budget is consumed, see burn rate alerts on SLOs.

keep_firing_for: avoid flapping

According to the same page, keep_firing_for helps prevent flapping and false resolutions caused by data gaps. With 5 minutes, an alert that drops below the threshold for one minute is not declared resolved straight away. Without this field, the alert deactivates on the first evaluation where the condition is no longer met, once any for duration is satisfied.

What does Prometheus store?

For each pending or firing alert, Prometheus records an ALERTS series with the labels alertname, alertstate (pending or firing) and the alert’s labels, at the value 1. The series is marked stale when the alert leaves that state. You can therefore query alert history with PromQL, for example to count firing alerts by name over a week and spot the noisy rules.

Prometheus does not handle notifications itself. According to the documentation, it sends alert state to an Alertmanager, which handles summarisation, rate limiting, silencing and dispatch. The quality of an alert therefore also depends on routing. For the Dynatrace side of noise, see reducing Davis alert noise.

Designing a useful alert

The alerting best-practice page recommends alerting on symptoms rather than causes, that is, on what hurts the user: latency and error rate, as high in the stack as possible. It also advises leaving slack for small blips, and having alerts point to the consoles that help identify the faulty component. It treats nearing capacity as a warning, not a page.

Applied to the example: HighErrorRatio is a symptom alert with severity page, while TargetDown is closer to a ticket. This distinction cuts on-call toil, because every page should demand human action.

Testing alerting rules with promtool

The Unit testing for rules page describes the promtool test rules test.yml command. The test file declares the rule_files, the evaluation_interval (1 minute by default) and a list of tests. Each test supplies input_series and, for an alert, an alert_rule_test block:

  • eval_time: the moment, counted from the start of the test, at which alerts are checked;
  • alertname: the alert under test;
  • exp_alerts: the expected alerts, with exp_labels (the series labels, without __name__ or alertname) and exp_annotations. An empty list checks that the alert does not fire.

The notation 0.08+0x20 starts at 0.08 and adds 0 for 20 steps, so the series stays constant above the threshold.

File test.yml for promtool test rules: error-ratio series constant at 0.08, no alert at 5 minutes and an expected alert at 15 minutes

The test checks two things: at 5 minutes, the 10-minute for duration has not been reached, so there is no alert; at 15 minutes, the alert is active with the expected labels and annotations. This file was not run for this article, because promtool was not available in the writing environment. Run it before relying on it and adjust it if your version reports something different.

Validating rules in CI

According to the promtool reference, check rules checks whether rule files are valid. Its --lint option accepts all, duplicate-rules or none (default duplicate-rules) and --lint-fatal makes the command exit with code 3 on a lint error. For tests, --run limits execution to the groups whose name matches a regular expression, and --diff (marked experimental) prints the differences between expected and received output.

A typical pipeline step chains the two commands:

promtool check rules alerts.yml rules.yml --lint=all --lint-fatal promtool test rules test.yml --diff

The page gives no general table of exit codes, apart from code 3 for --lint-fatal. In most CI systems a non-zero code fails the step: check that behaviour on your own tool before relying on it.

One possible review rule: every change to alerts.yml must come with a test that fails without it. An alert that is never tested soon goes stale. To place this practice within observability as a whole, see the difference between observability and monitoring.

Best practices

  • Alert on symptoms visible to the user, with a severity that decides between a page and a ticket.
  • Set a for consistent with the query window and your SLO, and add keep_firing_for if the alert flaps.
  • Put a runbook link in the annotations and test them with exp_annotations.
  • Version rules and their tests together, and validate them in CI.
  • Also test the case where the alert must not fire.

Frequently asked questions

What happens if for is missing?

According to the documentation, the alert becomes active on the first evaluation where the expression returns series. On a noisy metric this produces short-lived alerts; a for avoids this.

Do I need a recording rule before an alert?

No. An alert can carry a direct expression. Using a recorded series avoids duplicating an expensive query between dashboards and alerts, and makes the test simpler, as in the example.

Can these alerts be sent to Dynatrace?

The rules are still evaluated by Prometheus and passed to an Alertmanager. This article documents no direct integration with Dynatrace: to carry Prometheus metrics across, see Prometheus and Kubernetes with Dynatrace, and check the target platform’s documentation for alerting.

Conclusion

An alerting rule consists of an expression, a for duration, labels and annotations. It is only worth having if it is tested: promtool check rules and promtool test rules in CI prevent both silent alerts and pointless pages. Start with one symptom rule, one test that fires and one test that does not.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of its owner. Prometheus is a Cloud Native Computing Foundation project. Procedures change: always check the official documentation before a production change.

Originally published on span.observer. French version: https://span.observer/fr/alerting-rules-prometheus-promtool-ci/.