instruction: How do you implement comprehensive microservices observability?
output: Architecture | Jaeger Docs Introduction &cir; Getting Started &cir; Features Architecture &cir; APIs &cir; Sampling &cir; SPM &cir; Terminology Deployment &cir; Configuration &cir; User Interface &cir; On Kubernetes &cir; On Windows &cir; Security Storage Backends &cir; Memory &cir; Badger &cir; Cassandra &cir; ElasticSearch &cir; Kafka &cir; OpenSearch Operations &cir; Monitoring &cir; Troubleshooting &cir; Performance Tuning &cir; Tools Frequently Asked Questions External Guides &cir; Migration Guides Versions 2.9&nbsp;&nbsp;( latest ) 2.8 2.7 2.6 2.5 2.4 2.3 2.2 2.1 2.0 1.72&nbsp;&nbsp;( latest ) 1.71 1.70 1.69 1.68 1.67 1.66 1.65 1.64 1.63 1.62 1.61 1.60 1.59 1.58 1.57 Demo Download Blog Project Get involved Get in touch Mentorships Mentorships - For Mentees Mentorships - For Mentors News Roadmap Branding Report security issue Main GitHub repo Docs GitHub repo We stand with our friends and colleagues in Ukraine. To support Ukraine in their time of need visit this page . Introduction &cir; Getting Started &cir; Features Architecture &cir; APIs &cir; Sampling &cir; SPM &cir; Terminology Deployment &cir; Configuration &cir; User Interface &cir; On Kubernetes &cir; On Windows &cir; Security Storage Backends &cir; Memory &cir; Badger &cir; Cassandra &cir; ElasticSearch &cir; Kafka &cir; OpenSearch Operations &cir; Monitoring &cir; Troubleshooting &cir; Performance Tuning &cir; Tools FAQs External Guides &cir; Migration Guides Architecture Version&nbsp;&nbsp; 2.9 Latest Go to the latest 1.x version See also: APIs Sampling SPM Terminology Jaeger v2 is designed to be a versatile and flexible tracing platform. It can be deployed as a single binary that can be configured to perform different roles within the Jaeger architecture. Roles collector : Receives incoming trace data from applications and writes it into a storage backend. query : Serves the APIs and the user interface for querying and visualizing traces. ingester : Ingests spans from Kafka and writes them into a storage backend; useful when running in a split collector-Kafka-ingester configuration . all-in-one : Collector and query roles in a single process. agent : A host agent or a sidecar that runs next to the application and forwards trace data to the collector. While Jaeger can be configured for this role, we recommend using the standard OpenTelemetry Collector &#8201; external link instead because you may likely need it to process other types of telemetry (metrics & logs). Choosing between the all-in-one and the collector / query configurations is a matter of preference. When using external storage backend, both configurations are horizontally scalable, but the collector / query configuration allows to separate the read and write traffic and to scale them independently, as well as to apply different access and security policies. The all-in-one configuration with in-memory storage is most suitable for development and testing, but it is not recommended for production since the data is lost on restarts. all-in-one with the Badger backend can be used in production, but only for modest data volumes since it is limited to a single instance and cannot be scaled horizontally. Architecture choices The two most common deployment options for a scalable Jaeger backend are direct-to-storage and using Kafka as a buffer. Direct to storage In this deployment the collector s receive the data from traced applications and write it directly to storage. The storage must be able to handle both average and peak traffic. The collector s may use an in-memory queue to smooth short-term traffic peaks, but a sustained traffic spike may result in dropped data if the storage is not able to keep up. Via Kafka To prevent data loss between collector s and storage, Kafka can be used as an intermediary, persistent queue. The collector s are configured with Kafka exporters. An additional component, ingester , needs to be deployed to read data from Kafka and save it to storage. Multiple ingester s can be deployed to scale up ingestion; they will automatically partition the load across them. In practice, an ingester is very similar to a collector , only configured with a Kafka receiver instead of RPC-based receivers. With OpenTelemetry Collector You do not need to use the OpenTelemetry Collector to operate Jaeger, because Jaeger is a customized distribution of the OpenTelemetry Collector with different roles. However, if you already use the OpenTelemetry Collectors, for gathering other types of telemetry or for pre-processing / enriching the tracing data, it can be placed in front of Jaeger in the collection pipeline. The OpenTelemetry Collectors can be run as an application sidecar, as a host agent / daemon, or as a remote service cluster. The OpenTelemetry Collector supports Jaeger&rsquo;s Remote Sampling protocol and can either serve static configurations from config files directly, or proxy the requests to the Jaeger backend (e.g., when using adaptive sampling). OpenTelemetry Collector as a sidecar / host agent Benefits: The SDK configuration is simplified as both trace export endpoint and sampling config endpoint can point to a local host and not worry about discovering where those services run remotely. Collector may provide data enrichment by adding environment information, like k8s pod name. Resource usage for data enrichment can be distributed across all application hosts. Downsides: An extra layer of marshaling/unmarshaling the data. OpenTelemetry Collector as a remote cluster Benefits: Sharding capabilities, e.g., when using tail-based sampling &#8201; external link . Downsides: An extra layer of marshaling/unmarshaling the data. Jaeger Binary The Jaeger binary is build on top of the OpenTelemetry Collector framework and includes: Official upstream components, such as OTLP Receiver, Batch and Attribute Processor, etc. Upstream components from opentelemetry-collector-contrib , such as Kafka Exporter and Receiver, Tail Sampling Processor, etc. Jaeger own components, such as Jaeger Storage Exporter, Jaeger Query Extension, etc. Jaeger Components Jaeger Storage Extension &#8201; external link - Extensible hub for storage backends supported in Jaeger. It provides all other Jaeger components access to Jaeger storage implementations. Jaeger Storage Exporter &#8201; external link - Writes spans to storage backend configured in the Jaeger Storage Extension. Jaeger Query Extension &#8201; external link - Run the query APIs and the Jaeger UI. Adaptive Sampling Processor &#8201; external link - Performs probabilities calculations for adaptive sampling . Remote Sampling Extension &#8201; external link - Serves the endpoints for Remote Sampling , based on static configuration file or adaptive sampling . OpenTelemetry Components Receivers OTLP &#8201; external link - Accepts spans sent via OpenTelemetry Protocol (OTLP). Jaeger &#8201; external link - Accepts Jaeger formatted traces transported via gRPC or Thrift protocols. Kafka &#8201; external link - Accepts spans from Kafka in various formats (OTLP, Jaeger, Zipkin). Zipkin &#8201; external link - Accepts spans using Zipkin v1 and v2 protocols. No-op &#8201; external link - Used for Jaeger UI / query service deployment that does not require an ingestion pipeline. Processors Batch Processor &#8201; external link - Batches spans for better efficiency. Tail Sampling &#8201; external link - Supports advanced post-collection sampling. Memory Limiter &#8201; external link - Supports back-pressure when the collector is overloaded. Attributes Processor &#8201; external link - Allows filtering, rewriting, and enriching spans with attributes. Can be used to redact sensitive data, reduce data volume, or attach environment information. Filter Processor &#8201; external link - Allows dropping spans and span events from the collector (⚠️ may cause broken traces). Exporters OTLP &#8201; external link - Send data in OTLP format via gRPC. OTLP HTTP &#8201; external link - Sends data in OTLP format over HTTP. Kafka &#8201; external link - Sends data to Kafka in various formats (OTLP, Jaeger, Zipkin). Prometheus &#8201; external link - Sends metrics to Prometheus. Debug &#8201; external link - Debugging tool for pipelines. No-op &#8201; external link - Used for Jaeger UI / query service deployment that does not require an ingestion pipeline. Connectors Span Metrics &#8201; external link - Generates metrics from span data. Forward &#8201; external link - Redirects telemetry between pipelines in the collector (ex: span to metric / span to log) Extensions Health Check v2 &#8201; external link - Supports health checks. zPages &#8201; external link - Exposes internal state of the collector for debugging. Performance Profiler (pprof) &#8201; external link - enables Go&rsquo;s net/http/pprof endpoint, typically used by developers to collect performance profiles and investigate issues with the collector. Community Getting in touch Via chat Project video meetings Report issues on GitHub Report a security issue &copy; 2025 The Jaeger Authors. Documentation distributed under CC-BY-4.0 . &copy; 2025 The Linux Foundation. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our Trademark Usage page.