OpenTelemetry with Elastic and Grafana: how to roll it out

OpenTelemetry (OTel) is an open standard and a set of tools for generating, collecting and exporting telemetry data: traces, metrics and logs. It is not a system for storing or visualising data. Those roles are left to backends such as Elastic, Grafana, Prometheus or commercial products. This means you instrument an application once and can change where the data goes without rewriting code. Below we describe how to set this up in practice.

What OpenTelemetry consists of

• Signals. Traces show the path of a single request across services, metrics are numbers describing the state of a system over time, and logs are events from applications and infrastructure. Shared trace and span identifiers let you connect a log line to the request that caused it.
• SDKs and instrumentation. Libraries for languages such as Java, .NET, Python, Node.js and Go. Automatic instrumentation adds traces to popular libraries without code changes, while manual instrumentation lets you describe your own business logic.
• OTLP. The OpenTelemetry Protocol, which carries data between the application, the collector and the backend. It runs over gRPC (port 4317 by default) and HTTP (port 4318).
• Semantic conventions. Agreed attribute names, for example for HTTP requests, databases or hosts. They are why the same data looks the same in every tool.
• Collector. A separate process that receives data, processes it and forwards it. It is the most important part of a rollout.

OpenTelemetry is a Cloud Native Computing Foundation project, formed by merging OpenTracing and OpenCensus, and it is not tied to a single vendor.

Why a collector instead of sending straight from the application

Applications can send data directly to a backend, but in a company environment a collector in between brings several concrete benefits:

• One place for configuration. You change the backend address, credentials and filters in the collector, not in dozens of applications.
• Buffering and retries. When the backend is briefly unavailable, the collector holds the data and retries, so the application does not wait.
• Processing before export. The collector can add attributes (environment name, region), strip sensitive data, drop noise and sample traces.
• Export to several destinations. The same stream can go to Elastic and Grafana at once, for example during a migration from one tool to another.

A collector is built from receivers, processors and exporters, joined into pipelines separately for each signal. It is usually run in two roles: as an agent on every host or in a Kubernetes cluster (collecting data locally), and as a gateway, a small group of instances that accept data from agents and forward it to backends.

Example collector configuration

The sketch below receives OTLP data, protects the collector from running out of memory, batches the data and sends it to two destinations. Addresses and credentials are examples.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  resourcedetection:
    detectors: [system]
  batch: {}

exporters:
  otlp/elastic:
    endpoint: apm.example.com:443
    headers:
      Authorization: "Bearer ${env:ELASTIC_SECRET_TOKEN}"
  otlphttp/grafana:
    endpoint: https://otlp.example.com
    headers:
      Authorization: "Basic ${env:GRAFANA_OTLP_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/elastic, otlphttp/grafana]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/elastic, otlphttp/grafana]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlp/elastic, otlphttp/grafana]

The memory_limiter processor comes first in the pipeline and batch last. Pass secrets through environment variables rather than writing them into the file. Keep export to two backends only for the comparison or migration period, because it doubles traffic and storage cost.

OpenTelemetry and Elastic

Elastic accepts OTLP data, so an application instrumented with OpenTelemetry can send traces and metrics to Elastic Observability. There are three routes:

1. OTLP data from the collector (or straight from the SDK) goes to APM Server, which accepts OTLP and stores the data in Elasticsearch;
2. the same path, but using the Elastic Distribution of OpenTelemetry (EDOT), a distribution of the collector and SDKs for selected languages developed by Elastic;
3. existing Elastic APM agents, which can work with the OpenTelemetry API.

For new deployments we start with the standard OpenTelemetry SDKs and a collector, because this does not tie applications to one vendor. Check which versions and features your Elastic version supports, because OTel support in Elastic is growing quickly.

OpenTelemetry and Grafana

The Grafana stack also accepts OTLP: Tempo stores traces, Loki logs, and Mimir and Prometheus metrics. For collection, Grafana offers Grafana Alloy, its own distribution of the OpenTelemetry Collector with built-in Prometheus pipelines. Alloy can therefore play the collector role from the example above.

How to roll it out step by step

1. Start with one service. Pick an application that matters to the business but is not the most critical. Enable automatic instrumentation and see which traces appear.
2. Agree conventions. Service, environment and version names must be consistent across all teams. Without that, dashboards and alerts will be hard to read.
3. Deploy the collector. First as a gateway, then as agents on hosts. Monitor the collector itself: it has its own metrics (memory use, queue length, dropped data).
4. Add logs correlated with traces. Writing the trace ID into application logs is the cheapest way to get from an error in a log to the whole request.
5. Control cost. Trace sampling, filtering high-cardinality metrics and shorter retention usually cut the bill more than any backend optimisation.
6. Expand gradually. Add further services only once the first one is stable.

What to watch out for

Automatic instrumentation raises an application’s CPU and memory use, so measure it in a test environment after enabling it. Trace attributes can contain personal data, for example email addresses in URLs, so filter them in the collector before export. Finally, different languages have different maturity for each signal, so check the OpenTelemetry documentation for your language before you count on logs or metrics from a given library.

When you want to measure the performance of the cluster that receives this data, our article on benchmarking Elasticsearch with Rally will help. We deploy a collector-based architecture, from the choice of design to day-to-day running, as part of application monitoring.