instruction: Configure AlertManager for Alerting rules | Prometheus Prometheus Docs Download Community Support &amp; Training Blog Ctrl + K ...
output: Alerting rules | Prometheus Prometheus Docs Download Community Support &amp; Training Blog Ctrl + K Show nav Introduction Overview First steps Comparison to alternatives FAQ Roadmap Design documents Media Glossary Long-term support Concepts Data model Metric types Jobs and instances Prometheus Server Getting started Installation Configuration Configuration Recording rules Alerting rules Template examples Template reference Unit testing for rules HTTPS and authentication Querying Basics Operators Functions Examples HTTP API Remote Read API Storage Federation HTTP SD Management API Command Line prometheus promtool Migration API stability Feature flags Visualization Expression browser Grafana Perses Console templates Instrumenting Client libraries Writing client libraries Pushing metrics Exporters and integrations Writing exporters Exposition formats UTF-8 escaping schemes Content negotiation Operating Security Integrations Alertmanager Alerting overview Alertmanager Configuration Clients Notification template reference Notification template examples Management API HTTPS and authentication Best practices Metric and label naming Consoles and dashboards Instrumentation Histograms and summaries Alerting Recording rules When to use the Pushgateway Remote write tuning Guides Basic auth Monitoring Docker container metrics using cAdvisor Use file-based service discovery to discover scrape targets Instrumenting a Go application Understanding and using the multi-target exporter pattern Monitoring Linux host metrics with the Node Exporter OpenTelemetry UTF-8 in Prometheus Docker Swarm Query log TLS encryption Tutorials Getting started with Prometheus Understanding metric types Instrumenting HTTP server written in Go Visualizing metrics using Grafana Alerting based on metrics Specifications Native Histograms [EXPERIMENTAL] OpenMetrics 1.0 Remote Write 1.0 2.0 Alerting rules Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service. Whenever the alert expression results in one or more vector elements at a given point in time, the alert counts as active for these elements&#x27; label sets. Defining alerting rules Alerting rules are configured in Prometheus in the same way as recording rules . An example rules file with an alert would be: groups : - name : example labels : team : myteam rules : - alert : HighRequestLatency expr : job:request_latency_seconds:mean5m{job=&quot;myjob&quot;} &gt; 0.5 for : 10m keep_firing_for : 5m labels : severity : page annotations : summary : High request latency The optional for clause causes Prometheus to wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element. In this case, Prometheus will check that the alert continues to be active during each evaluation for 10 minutes before firing the alert. Elements that are active, but not firing yet, are in the pending state. Alerting rules without the for clause will become active on the first evaluation. There is also an optional keep_firing_for clause that tells Prometheus to keep this alert firing for the specified duration after the firing condition was last met. This can be used to prevent situations such as flapping alerts, false resolutions due to lack of data loss, etc. Alerting rules without the keep_firing_for clause will deactivate on the first evaluation where the condition is not met (assuming any optional for duration described above has been satisfied). The labels clause allows specifying a set of additional labels to be attached to the alert. Any existing conflicting labels will be overwritten. The label values can be templated. The annotations clause specifies a set of informational labels that can be used to store longer additional information such as alert descriptions or runbook links. The annotation values can be templated. Templating Label and annotation values can be templated using console templates . The $labels variable holds the label key/value pairs of an alert instance. The configured external labels can be accessed via the $externalLabels variable. The $value variable holds the evaluated value of an alert instance. # To insert a firing element&#x27;s label values: {{ $labels.&lt;labelname&gt; }} # To insert the numeric expression value of the firing element: {{ $value }} Examples: groups : - name : example rules : # Alert for any instance that is unreachable for &gt;5 minutes. - alert : InstanceDown expr : up == 0 for : 5m labels : severity : page annotations : summary : &quot;Instance {{ $labels.instance }} down&quot; description : &quot;{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.&quot; # Alert for any instance that has a median request latency &gt;1s. - alert : APIHighRequestLatency expr : api_http_request_latencies_second{quantile=&quot;0.5&quot;} &gt; 1 for : 10m annotations : summary : &quot;High request latency on {{ $labels.instance }}&quot; description : &quot;{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)&quot; Inspecting alerts during runtime To manually inspect which alerts are active (pending or firing), navigate to the &quot;Alerts&quot; tab of your Prometheus instance. This will show you the exact label sets for which each defined alert is currently active. For pending and firing alerts, Prometheus also stores synthetic time series of the form ALERTS{alertname=&quot;&lt;alert name&gt;&quot;, alertstate=&quot;&lt;pending or firing&gt;&quot;, &lt;additional alert labels&gt;} . The sample value is set to 1 as long as the alert is in the indicated active (pending or firing) state, and the series is marked stale when this is no longer the case. Sending alert notifications Prometheus&#x27;s alerting rules are good at figuring what is broken right now , but they are not a fully-fledged notification solution. Another layer is needed to add summarization, notification rate limiting, silencing and alert dependencies on top of the simple alert definitions. In Prometheus&#x27;s ecosystem, the Alertmanager takes on this role. Thus, Prometheus may be configured to periodically send information about alert states to an Alertmanager instance, which then takes care of dispatching the right notifications. Prometheus can be configured to automatically discover available Alertmanager instances through its service discovery integrations. Previous Recording rules Edit Edit this page Next Template examples On this page © Prometheus Authors 2014- 2025 | Documentation Distributed under CC-BY-4.0 © 2025 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.