Tracing a slow request with OpenTelemetry in a small app
Short answer: A trace shows one request's whole journey through your app as a timeline of steps called spans: the web handler, each database query, each call to another service. OpenTelemetry is the open standard for producing those traces without locking into one vendor. For a small app, start with automatic (zero-code) instrumentation, send traces to any backend that accepts OpenTelemetry data, sample sensibly, and look at your slowest requests first.
Why logs and metrics aren't enough for slow requests
Metrics tell you that the checkout endpoint got slower. Logs tell you individual things happened. Neither easily tells you that a particular checkout took four seconds because three of those seconds were spent waiting on a shipping-rate API, called twice in a row.
A trace answers that directly. You open one slow request and see a waterfall: how long each step took, which steps ran in sequence, and which ran in parallel. For an app made of a website, an API and a couple of background services, it's often the fastest way to find where time goes.
The few concepts you need
OpenTelemetry's documentation on traces describes the building blocks:
- Trace: the whole path of one request, identified by a trace ID.
- Span: one unit of work inside it, such as an HTTP request or a database query, with a start time, duration and its own span ID.
- Span context: the trace ID, span ID and flags that travel with the request so other services can join the same trace.
- Attributes: key-value details on a span, such as the route, HTTP status or order ID.
Spans have parents, so the trace forms a tree: the incoming request is the root, with database calls and outgoing requests underneath.
How traces cross service boundaries
When your web app calls your API, the API needs to know it's part of the same trace. This is called context propagation. The W3C Trace Context specification defines standard HTTP headers for it, chiefly traceparent, which carries the trace ID, the parent span ID and trace flags, plus an optional tracestate for vendor-specific data.
Because it's a W3C standard, services written in different languages, and proxies or platforms that understand the header, can all join the same trace. OpenTelemetry uses it by default.
Start with zero-code instrumentation
You don't have to add tracing calls by hand to get value. OpenTelemetry offers zero-code instrumentation for several languages, which hooks into common frameworks and libraries, such as HTTP servers, HTTP clients and database drivers, and creates spans automatically. Depending on the language, that's an agent, a package you load at start-up or an environment-variable-driven setup.
A sensible first step:
- Add the zero-code setup for your language to one service.
- Set a service name, such as
shop-web, so traces are labelled clearly. - Point the exporter at a backend that accepts OpenTelemetry Protocol (OTLP) data.
- Make a few requests and confirm traces appear.
Then repeat for the next service, so traces join up end to end.
Add a few manual spans where it matters
Automatic spans cover framework and library calls. Your own business logic, such as "calculate delivery options" or "apply discount rules", is invisible unless you add a span around it. Add manual spans only around steps you actually want to see, and attach attributes that help you filter, like cart.items or delivery.region.
Keep personal data out of span attributes, for the same reasons you'd keep it out of logs: traces are stored, searched and shared.
Sampling keeps cost and noise down
Recording every span of every request can produce a lot of data. OpenTelemetry's sampling documentation describes two broad approaches:
- Head sampling: decide at the start of a request whether to keep its trace, for example keeping a fixed percentage. It's simple and cheap, but it may drop the rare slow or failing request you care about.
- Tail sampling: decide after the trace is complete, for example keeping all errors and slow requests plus a sample of the rest. It needs more machinery, usually the OpenTelemetry Collector.
For a small app with modest traffic, keeping everything may be fine at first. As traffic grows, head sampling is the easy step, and tail sampling is the better one if your backend or collector supports it.
Choosing where traces go
Because OpenTelemetry is vendor-neutral, you can send the same data to an open-source backend you host yourself or to a commercial observability service, and switch later without re-instrumenting. Many teams also run the OpenTelemetry Collector in between, which receives data, applies sampling or filtering, and forwards it. Compare current prices and free tiers yourself; they change often.
Reading your first traces
Once traces arrive, look for these patterns:
- Many short database spans in a row: often an N+1 query problem.
- One long span on an outgoing call: a slow third-party API, which may need a timeout or caching.
- Steps in sequence that could run in parallel: two independent API calls made one after the other.
- Gaps between spans: time spent in code you haven't instrumented yet.
Start with your most important journey, such as checkout or sign-up, and its slowest requests.
A worked example
Imagine a hypothetical Wellington meal-kit app whose order page sometimes took several seconds. Logs showed nothing unusual. The team added zero-code instrumentation to the web app and the pricing service, and the first slow trace showed the pricing service calling the delivery-zone lookup once per cart item, one after another. Caching zone lookups per order and making a single call fixed most of the delay, and the trace waterfall confirmed it.
FAQ
Do I need OpenTelemetry if my host already shows request times?
Host dashboards show totals. Traces show the breakdown inside each request, which is what you need to fix a slow one.
Will tracing slow my app down?
There's some overhead, generally small with sensible sampling. Measure it in your own environment before and after.
Can I connect logs to traces?
Yes. Add the trace ID to each log entry, and many tools will let you jump between a log line and its trace.
Does this work for serverless functions?
Yes, with language-specific setup. Check OpenTelemetry's documentation for your platform.
If you want to see where your app's slow requests spend their time, tell us about your setup.