instruction: How do you implement comprehensive microservices observability?
output: GitHub - kubernetes-monitoring/kubernetes-mixin: A set of Grafana dashboards and Prometheus alerts for Kubernetes. Skip to content Navigation Menu Toggle navigation Sign in Appearance settings Product GitHub Copilot Write better code with AI GitHub Spark New Build and deploy intelligent apps GitHub Models New Manage and compare prompts GitHub Advanced Security Find and fix vulnerabilities Actions Automate any workflow Codespaces Instant dev environments Issues Plan and track work Code Review Manage code changes Discussions Collaborate outside of code Code Search Find more, search less Explore Why GitHub All features Documentation GitHub Skills Blog Solutions By company size Enterprises Small and medium teams Startups Nonprofits By use case DevSecOps DevOps CI/CD View all use cases By industry Healthcare Financial services Manufacturing Government View all industries View all solutions Resources Topics AI DevOps Security Software Development View all Explore Learning Pathways Events &amp; Webinars Ebooks &amp; Whitepapers Customer Stories Partners Executive Insights Open Source GitHub Sponsors Fund open source developers The ReadME Project GitHub community articles Repositories Topics Trending Collections Enterprise Enterprise platform AI-powered developer platform Available add-ons GitHub Advanced Security Enterprise-grade security features Copilot for business Enterprise-grade AI features Premium Support Enterprise-grade 24/7 support Pricing Search or jump to... Search code, repositories, users, issues, pull requests... --> Search Clear Search syntax tips Provide feedback --> We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly --> Name Query To see all available qualifiers, see our documentation . Cancel Create saved search Sign in Sign up Appearance settings Resetting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} kubernetes-monitoring / kubernetes-mixin Public Notifications You must be signed in to change notification settings Fork 620 Star 2.3k A set of Grafana dashboards and Prometheus alerts for Kubernetes. License Apache-2.0 license 2.3k stars 620 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings Code Issues 13 Pull requests 4 Discussions Actions Security Uh oh! There was an error while loading. Please reload this page . Insights Additional navigation options Code Issues Pull requests Discussions Actions Security Insights kubernetes-monitoring/kubernetes-mixin master Branches Tags Go to file Code Open more actions menu Folders and files Name Name Last commit message Last commit date Latest commit History 1,380 Commits .github .github alerts alerts dashboards dashboards lib lib rules rules scripts scripts tests tests .gitignore .gitignore .lint .lint .vale.ini .vale.ini CODEOWNERS CODEOWNERS CODE_OF_CONDUCT.md CODE_OF_CONDUCT.md CONTRIBUTING.md CONTRIBUTING.md DESIGN.md DESIGN.md LICENSE LICENSE Makefile Makefile OWNERS OWNERS README.md README.md SECURITY.md SECURITY.md SECURITY_CONTACTS SECURITY_CONTACTS config.libsonnet config.libsonnet jsonnetfile.json jsonnetfile.json mixin.libsonnet mixin.libsonnet runbook.md runbook.md View all files Repository files navigation README Code of conduct Contributing Apache-2.0 license Security Prometheus Monitoring Mixin for Kubernetes NOTE: This project is pre-release stage. Flags, configuration, behaviour and design may change significantly in following releases. A set of Grafana dashboards and Prometheus alerts for Kubernetes. Releases Note: Releases up until release-0.12 are changes in their own branches. Changelogs are included in releases starting from version-0.13.0 . Release branch Kubernetes Compatibility Prometheus Compatibility Kube-state-metrics Compatibility release-0.1 v1.13 and before release-0.2 v1.14.1 and before v2.11.0+ release-0.3 v1.17 and before v2.11.0+ release-0.4 v1.18 v2.11.0+ release-0.5 v1.19 v2.11.0+ release-0.6 v1.19+ v2.11.0+ release-0.7 v1.19+ v2.11.0+ v1.x release-0.8 v1.20+ v2.11.0+ v2.0+ release-0.9 v1.20+ v2.11.0+ v2.0+ release-0.10 v1.20+ v2.11.0+ v2.0+ release-0.11 v1.23+ v2.11.0+ v2.0+ release-0.12 v1.23+ v2.11.0+ v2.0+ release-0.13 v1.23+ v2.11.0+ v2.0+ master v1.26+ v2.11.0+ v2.0+ In Kubernetes 1.14 there was a major metrics overhaul implemented. Therefore v0.1.x of this repository is the last release to support Kubernetes 1.13 and previous version on a best effort basis. Some alerts now use Prometheus filters made available in Prometheus 2.11.0, which makes this version of Prometheus a dependency. Warning: This compatibility matrix was initially created based on experience, we do not guarantee the compatibility, it may be updated based on new learnings. Warning: By default the expressions will generate grafana 7.2+ compatible rules using the $__rate_interval variable for rate functions. If you need backward compatible rules please set grafana72: false in your _config Release steps Maintainers can trigger the release workflow by pushing a git tag that matches the pattern: version-* . Checkout master branch and pull for latest. git checkout master Create a tag following sem-ver versioning for the version and trigger release. # replace MAJOR.MINOR.PATCH with e.g. 1.2.3 tag=version-MAJOR.MINOR.PATCH ; git tag $tag &amp;&amp; git push origin $tag Decisions on backfilling releases We wanted to backfill release-0.1 to release-0.12 to have a changelog, but we were not able to use a GitHub action in a newer commit to trigger a release that generates a changelog on older commits. See #489 for full discussion. Metrics Deprecation The following recording rule is marked deprecated. It will be removed in v2.0.0. node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate It will be replaced by the following recording rule to preserve data points using rate and add 5m to indicate the range of the rate query in the recording rule name. node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate5m How to use This mixin is designed to be vendored into the repo with your infrastructure config. To do this, use jsonnet-bundler : You then have three options for deploying your dashboards Generate the config files and deploy them yourself Use ksonnet to deploy this mixin along with Prometheus and Grafana Use prometheus-operator to deploy this mixin (TODO) Generate config files You can manually generate the alerts, dashboards and rules files, but first you must install some tools: $ go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest $ brew install jsonnet Then, grab the mixin and its dependencies: $ git clone https://github.com/kubernetes-monitoring/kubernetes-mixin $ cd kubernetes-mixin $ jb install Finally, build the mixin: $ make prometheus_alerts.yaml $ make prometheus_rules.yaml $ make dashboards_out The prometheus_alerts.yaml and prometheus_rules.yaml file then need to passed to your Prometheus server, and the files in dashboards_out need to be imported into you Grafana server. The exact details will depending on how you deploy your monitoring stack to Kubernetes. Dashboards for Windows Nodes There exist separate dashboards for windows resources. Compute Resources / Cluster(Windows) Compute Resources / Namespace(Windows) Compute Resources / Pod(Windows) USE Method / Cluster(Windows) USE Method / Node(Windows) These dashboards are based on metrics populated by windows-exporter from each Windows node. Running the tests make test Using with prometheus-ksonnet Alternatively you can also use the mixin with prometheus-ksonnet , a ksonnet module to deploy a fully-fledged Prometheus-based monitoring system for Kubernetes: Make sure you have the ksonnet v0.8.0: $ brew install https://raw.githubusercontent.com/ksonnet/homebrew-tap/82ef24cb7b454d1857db40e38671426c18cd8820/ks.rb $ brew pin ks $ ks version ksonnet version: v0.8.0 jsonnet version: v0.9.5 client-go version: v1.6.8-beta.0+$Format:%h$ In your config repo, if you don't have a ksonnet application, make a new one (will copy credentials from current context): $ ks init &lt;application name&gt; $ cd &lt;application name&gt; $ ks env add default Grab the kubernetes-jsonnet module using and its dependencies, which include the kubernetes-mixin: $ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb $ jb init $ jb install github.com/kausalco/public/prometheus-ksonnet Assuming you want to run in the default namespace ('environment' in ksonnet parlance), add the follow to the file environments/default/main.jsonnet : local prometheus = import "prometheus-ksonnet/prometheus-ksonnet.libsonnet" ; prometheus { _config+:: { namespace: "default" , }, } Apply your config: $ ks apply default Using prometheus-operator TODO Multi-cluster support Kubernetes-mixin can support dashboards across multiple clusters. You need either a multi-cluster Thanos installation with external_labels configured or a Cortex system where a cluster label exists. To enable this feature you need to configure the following: // Opt-in to multiCluster dashboards by overriding this and the clusterLabel. showMultiCluster: true , clusterLabel: '&lt;your cluster label&gt;' , Customising the mixin Kubernetes-mixin allows you to override the selectors used for various jobs, to match those used in your Prometheus set. You can also customize the dashboard names and add grafana tags. In a new directory, add a file mixin.libsonnet : local kubernetes = import "kubernetes-mixin/mixin.libsonnet" ; kubernetes { _config+:: { kubeStateMetricsSelector: 'job="kube-state-metrics"' , cadvisorSelector: 'job="kubernetes-cadvisor"' , nodeExporterSelector: 'job="kubernetes-node-exporter"' , kubeletSelector: 'job="kubernetes-kubelet"' , grafanaK8s+:: { dashboardNamePrefix: 'Mixin / ' , dashboardTags: [ 'kubernetes' , 'infrastucture' ], }, }, } Then, install the kubernetes-mixin: $ jb init $ jb install github.com/kubernetes-monitoring/kubernetes-mixin Generate the alerts, rules and dashboards: $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' &gt; alerts.yml $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' &gt;files/rules.yml $ jsonnet -J vendor -m files/dashboards -e '(import "mixin.libsonnet").grafanaDashboards' Customising alert annotations The steps described below extend on the existing mixin library without modifying the original git repository. This is to make consuming updates to your extended alert definitions easier. These definitions can reside outside of this repository and added to your own custom location, where you can define your alert dependencies in your jsonnetfile.json and add customisations to the existing definitions. In your working directory, create a new file kubernetes_mixin_override.libsonnet with the following: local utils = import 'lib/utils.libsonnet' ; ( import 'mixin.libsonnet' ) + ( { prometheusAlerts+:: // The specialAlerts can be in any other config file local slack = 'observability' ; local specialAlerts = { KubePodCrashLooping: { slack_channel: slack }, KubePodNotReady: { slack_channel: slack }, }; local addExtraAnnotations(rule) = rule { [ if 'alert' in rule then 'annotations' ]+: { dashboard: 'https://foo.bar.co' , [ if rule.alert in specialAlerts then 'slack_channel' ]: specialAlerts[rule.alert].slack_channel, }, }; utils.mapRuleGroups(addExtraAnnotations), } ) Create new file: lib/kubernetes_customised_alerts.jsonnet with the following: std.manifestYamlDoc (( import '../kubernetes_mixin_override.libsonnet' ).prometheusAlerts) Running jsonnet -S lib/kubernetes_customised_alerts.jsonnet will build the alerts with your customisations. Same result can be achieved by modyfying the existing config.libsonnet with the content of kubernetes_mixin_override.libsonnet . Background Alert Severities While the community has not yet fully agreed on alert severities and their to be used, this repository assumes the following paradigms when setting the severities: Critical: An issue, that needs to page a person to take instant action Warning: An issue, that needs to be worked on but in the regular work queue or for during office hours rather than paging the oncall Info: Is meant to support a trouble shooting process by informing about a non-normal situation for one or more systems but not worth a page or ticket on its own. Architecture and Technical Decisions For more motivation, see " The RED Method: How to instrument your services " talk from CloudNativeCon Austin. For more information about monitoring mixins, see this design doc . Note You can use the external tool call prom-metrics-check to validate the created dashboards. This tool allows you to check if the metrics installed and used in Grafana dashboards exist in the Prometheus instance. Please have a look at https://github.com/ContainerSolutions/prom-metrics-check . About A set of Grafana dashboards and Prometheus alerts for Kubernetes. Resources Readme License Apache-2.0 license Code of conduct Code of conduct Contributing Contributing Security policy Security policy Uh oh! There was an error while loading. Please reload this page . Activity Custom properties Stars 2.3k stars Watchers 48 watching Forks 620 forks Report repository Releases 6 version-1.2.0 Latest Jun 2, 2025 + 5 releases Packages 0 No packages published Uh oh! There was an error while loading. Please reload this page . Contributors 199 + 185 contributors Languages Jsonnet 98.9% Other 1.1% Footer &copy; 2025 GitHub,&nbsp;Inc. Footer navigation Terms Privacy Security Status Docs Contact Manage cookies Do not share my personal information You can’t perform that action at this time.