OpenSearch Alerting sends one notification for three matching lines

What you see: several log lines match within a minute and you get one email or webhook call, not one per line.

Why: a per query monitor runs one search per run and evaluates its trigger once, on the whole result, however many documents it found. In our test, 3 matching lines 5 seconds apart produced 1 notification.

What to use: a per document monitor, with the action set to per_alert and actionable_alerts: ["NEW"]. It created one alert per matching document and sent 3 notifications for the 3 lines, each carrying that document's id, and none again in the next 3 minutes.

Measured on OpenSearch 3.6.0, the indexer that ships with Wazuh 5.0.0-rc1, in test containers: one indexer node, security plugin off, monitors created through the REST API, schedule 1 minute, one webhook listener, throttle off. Test documents were written by us, not real logs. The per document setup was also run on stock OpenSearch 3.8.0 with the same result; the comparison of the three monitor types was run on 3.6.0 only.

Check your monitor

Set two variables first. Our cluster had the security plugin off; if yours has it on, use the form in the comment:

IDX=http://localhost:9200 # your cluster. Ours had the security plugin off: plain HTTP, no login. CURL="curl -s" # with security on: CURL="curl -sk -u admin:YOUR_PASSWORD" and IDX=https://... 

Both checks are read-only. The first shows which monitor type you have and which execution policy is on each action:

$CURL "$IDX/_plugins/_alerting/monitors/MONITOR_ID" | python3 -m json.tool \ | grep -E '"monitor_type"|"interval"|"unit"|per_alert|per_execution|actionable_alerts|throttle_enabled' 

query_level_monitor is a per query monitor, bucket_level_monitor per bucket, doc_level_monitor per document. If yours is query_level_monitor and you want one notification per line, that is the cause.

The second shows how many alerts exist and for which documents:

$CURL "$IDX/_plugins/_alerting/monitors/alerts?size=200" | python3 -c ' import json,sys for a in json.load(sys.stdin).get("alerts",[]): print(a["monitor_name"], a["id"], a["state"], a["start_time"], a.get("related_doc_ids"))' 

In our test the per document monitor had one ACTIVE alert per matching document, each with one entry in related_doc_ids. The per query monitor kept no alert per document: it reports on the search as a whole.

What we measured

Three documents containing "error", written at 04:04:07.7, 04:04:12.9 and 04:04:18.0, plus one control document with none of the words. All three monitors ran at 04:04:52. We counted what the webhook listener received, then kept watching until 04:07:46.

No monitor sent anything again from 04:04:52 to 04:07:46, about 3 minutes and at least 2 more runs. The control document produced nothing. Per bucket gave 3 only because the three lines had different text: it groups by the exact message, so two lines with identical text should share one bucket. We did not measure that case, so for one notification per line, per document is the type that matches the goal by design.

Why it happens

A per query monitor runs your search once per run and evaluates the trigger once, on the whole result. Three hits or three hundred, the trigger is either true or false for that run. That is useful when you want one message per run, and the wrong shape when you want one message per line.

A per document monitor checks each new document against its queries and keeps an alert per matching document. With the action on per_alert and ["NEW"], each new alert sent one notification: 3 documents, 3 alerts, 3 notifications, 0 repeats.

Fix: the monitor we ran

A per document monitor with a whole condition in one doc-level query: any of ten words, except six exclusion strings and one service. Replace the index and the channel id; the query and the action policy are the ones we ran.

$CURL -XPOST "$IDX/_plugins/_alerting/monitors" -H 'Content-Type: application/json' -d '{ "type": "monitor", "monitor_type": "doc_level_monitor", "name": "critical-lines", "enabled": true, "schedule": { "period": { "interval": 1, "unit": "MINUTES" } }, "inputs": [ { "doc_level_input": { "description": "", "indices": [ "YOUR-INDEX" ], "queries": [ { "id": "q-critical", "name": "q-critical", "tags": [], "query": "message:(error OR warning OR alert OR failed OR unavailable OR lost OR wrong OR unable OR invalid OR failure) AND NOT message:\"disabling inotify\" AND NOT message:\"user config - ignore\" AND NOT message:\"_put_session\" AND NOT message:\"PVE ticket\" AND NOT message:\"guest-ping\" AND NOT message:\"Cyberprotect\" AND NOT service:\"sshd-session\"" } ] } } ], "triggers": [ { "document_level_trigger": { "name": "critical-line", "severity": "1", "condition": { "script": { "source": "query[name=q-critical]", "lang": "painless" } }, "actions": [ { "name": "notify", "destination_id": "YOUR-CHANNEL-ID", "subject_template": { "source": "Critical log line", "lang": "mustache" }, "message_template": { "source": "Critical log line, run ending {{ctx.periodEnd}}", "lang": "mustache" }, "throttle_enabled": false, "action_execution_policy": { "action_execution_scope": { "per_alert": { "actionable_alerts": [ "NEW" ] } } } } ] } } ] }' 

With six documents five seconds apart (three that should notify, two excluded, one with no keyword), the run notified exactly 3 times, and the 3 alerts' related_doc_ids were exactly the three that should notify. Nothing more was sent in the next 3 minutes.

Doc-level queries use query string syntax, so the exclusion list is one string of AND NOT clauses. In our index, service was a keyword field, so service:"sshd-session" matched it exactly. If yours is text, test that the clause still excludes those lines, or point it at the keyword subfield. Only two of the six exclusions were tested with a real line; for the other four we only know the query was accepted.

If your channel is a webhook to a private address, notifications can fail silently: the default host_deny_list refuses private ranges.

Limits of what we measured

One node, security plugin off, plain HTTP, monitors created through the REST API and not the Dashboards editor, one webhook listener. We did not test email, Slack or other channel types. Throttle was off. We watched for repeats for about 3 minutes after each run, and the alerts were still ACTIVE at the end. Not measured: per bucket with identical message text, a per query monitor actually running with per_alert (the API accepted and stored it), a service field mapped as text, and heavy load.

Full write-up with the lab output: https://atkvn.com/fix-opensearch-alerting-one-notification-per-document.html