instruction: How do you set up a complete Prometheus monitoring stack?
output: Metric and label naming | Prometheus Prometheus Docs Download Community Support &amp; Training Blog Ctrl + K Show nav Introduction Overview First steps Comparison to alternatives FAQ Roadmap Design documents Media Glossary Long-term support Concepts Data model Metric types Jobs and instances Prometheus Server Getting started Installation Configuration Configuration Recording rules Alerting rules Template examples Template reference Unit testing for rules HTTPS and authentication Querying Basics Operators Functions Examples HTTP API Remote Read API Storage Federation HTTP SD Management API Command Line prometheus promtool Migration API stability Feature flags Visualization Expression browser Grafana Perses Console templates Instrumenting Client libraries Writing client libraries Pushing metrics Exporters and integrations Writing exporters Exposition formats UTF-8 escaping schemes Content negotiation Operating Security Integrations Alertmanager Alerting overview Alertmanager Configuration Clients Notification template reference Notification template examples Management API HTTPS and authentication Best practices Metric and label naming Consoles and dashboards Instrumentation Histograms and summaries Alerting Recording rules When to use the Pushgateway Remote write tuning Guides Basic auth Monitoring Docker container metrics using cAdvisor Use file-based service discovery to discover scrape targets Instrumenting a Go application Understanding and using the multi-target exporter pattern Monitoring Linux host metrics with the Node Exporter OpenTelemetry UTF-8 in Prometheus Docker Swarm Query log TLS encryption Tutorials Getting started with Prometheus Understanding metric types Instrumenting HTTP server written in Go Visualizing metrics using Grafana Alerting based on metrics Specifications Native Histograms [EXPERIMENTAL] OpenMetrics 1.0 Remote Write 1.0 2.0 Metric and label naming The metric and label conventions presented in this document are not required for using Prometheus, but can serve as both a style-guide and a collection of best practices. Individual organizations may want to approach some of these practices, e.g. naming conventions, differently. Metric names A metric name... ...MUST comply with the data model for valid characters. ...SHOULD have a (single-word) application prefix relevant to the domain the metric belongs to. The prefix is sometimes referred to as namespace by client libraries. For metrics specific to an application, the prefix is usually the application name itself. Sometimes, however, metrics are more generic, like standardized metrics exported by client libraries. Examples: prometheus _notifications_total (specific to the Prometheus server) process _cpu_seconds_total (exported by many client libraries) http _request_duration_seconds (for all HTTP requests) ...MUST have a single unit (i.e. do not mix seconds with milliseconds, or seconds with bytes). ...SHOULD use base units (e.g. seconds, bytes, meters - not milliseconds, megabytes, kilometers). See below for a list of base units. ...SHOULD have a suffix describing the unit, in plural form. Note that an accumulating count has total as a suffix, in addition to the unit if applicable. Also note that this applies to units in the narrow sense (like the units in the table below), but not to countable things in general. For example, connections or notifications are not considered units for this rule and do not have to be at the end of the metric name. (See also examples in the next paragraph.) http_request_duration_ seconds node_memory_usage_ bytes http_requests_ total (for a unit-less accumulating count) process_cpu_ seconds_total (for an accumulating count with unit) foobar_build _info (for a pseudo-metric that provides metadata about the running binary) data_pipeline_last_record_processed_ timestamp_seconds (for a timestamp that tracks the time of the latest record processed in a data processing pipeline) ...MAY order its name components in a way that leads to convenient grouping when a list of metric names is sorted lexicographically, as long as all the other rules are followed. The following examples have their the common name components first so that all the related metrics are sorted together: prometheus_tsdb_head_truncations_closed_total prometheus_tsdb_head_truncations_established_total prometheus_tsdb_head_truncations_failed_total prometheus_tsdb_head_truncations_total The following examples are also valid, but are following a different trade-off. They are easier to read individually, but unrelated metrics like prometheus_tsdb_head_series might get sorted in between. prometheus_tsdb_head_closed_truncations_total prometheus_tsdb_head_established_truncations_total prometheus_tsdb_head_failed_truncations_total prometheus_tsdb_head_truncations_total ...SHOULD represent the same logical thing-being-measured across all label dimensions. request duration bytes of data transfer instantaneous resource usage as a percentage As a rule of thumb, either the sum() or the avg() over all dimensions of a given metric should be meaningful (though not necessarily useful). If it is not meaningful, split the data up into multiple metrics. For example, having the capacity of various queues in one metric is good, while mixing the capacity of a queue with the current number of elements in the queue is not. Why include unit and type suffixes in metric names? Some metric naming conventions (e.g. OpenTelemetry) do not recommend or even do not allow including information about a metric unit and type in the metric name. A common argument is that those pieces of information are already defined somewhere else (e.g. schema, metadata, other labels, etc.). Prometheus strongly recommends including unit and type in a metric name, even if you store that information elsewhere, because of the following practical reasons: Metric consumption reliability and UX : When interacting with a modern UI to use such a metric in PromQL, it&#x27;s possible to display rich information about the metric&#x27;s type and unit (autocompletion, overlays, pop-ups). Unfortunately, interactive, adhoc querying in a powerful UI is not the only way that users interact with metrics. Metric consumption ecosystem is vast. Majority of the consumption comes in a form of the plain YAML configuration for variety of observability tools like alerting, recording, autoscaling, dashboards, analysis, processing, etc. It&#x27;s critical , especially during monitoring/SRE incident practices to look on PromQL expressions in plain YAML and understand the underlying metric type and unit you work with. Metric collisions : With growing adoption and metric changes over time, there are cases where lack of unit and type information in the metric name will cause certain series to collide (e.g. process_cpu for seconds and milliseconds). Labels Use labels to differentiate the characteristics of the thing that is being measured: api_http_requests_total - differentiate request types: operation=&quot;create|update|delete&quot; api_request_duration_seconds - differentiate request stages: stage=&quot;extract|transform|load&quot; Do not put the label names in the metric name, as this introduces redundancy and will cause confusion if the respective labels are aggregated away. CAUTION : Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values. Base Units Prometheus does not have any units hard coded. For better compatibility, base units should be used. The following lists some metrics families with their base unit. The list is not exhaustive. Family Base unit Remark Time seconds Temperature celsius celsius is preferred over kelvin for practical reasons. kelvin is acceptable as a base unit in special cases like color temperature or where temperature has to be absolute. Length meters Bytes bytes Bits bytes To avoid confusion combining different metrics, always use bytes , even where bits appear more common. Percent ratio Values are 0–1 (rather than 0–100). ratio is only used as a suffix for names like disk_usage_ratio . The usual metric name follows the pattern A_per_B . Voltage volts Electric current amperes Energy joules Power Prefer exporting a counter of joules, then rate(joules[5m]) gives you power in Watts. Mass grams grams is preferred over kilograms to avoid issues with the kilo prefix. Previous HTTPS and authentication Edit Edit this page Next Consoles and dashboards On this page © Prometheus Authors 2014- 2025 | Documentation Distributed under CC-BY-4.0 © 2025 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.