Distributed Tracing for Microservices: When Logs Aren’t Enough

A user complains that checkout is slow. You check your application logs. Nothing. You check your database logs. Nothing suspicious. You check your Redis logs. Clean. You check your payment service logs. Everything looks normal. Somewhere between the frontend and the final transaction, something is burning time. But where? How do you find it?

This is the moment when you realize that logs alone aren’t enough. You’re not looking at a single service—you’re looking at a request traveling through five services, and each one only knows about the part it touched. Logs tell you what happened inside a service. Distributed tracing tells you what happened across all of them.

If you’re running microservices in production, distributed tracing isn’t optional. It’s the difference between a five-minute debug session and a five-hour firefight at 2 a.m.

Why Logs Fail in Distributed Systems

Logs work great when you own the entire request path. A monolith processes an HTTP request, writes to a database, and returns. One service. One log stream. You grep for the request ID and you can see the entire story.

Microservices broke that. Now a single user action touches ten services. The request starts in an API gateway. It fans out to an auth service, a product service, an inventory service, an order service, a payment service, a notification service, and a warehouse service. Each one writes logs. Each one has a different log format, different timestamps, different context.

The problem: logs are local. Your payment service log doesn’t know what the order service was doing at the same millisecond. Your API gateway log doesn’t show you that a downstream service was hanging. You can correlate logs by request ID, but only if every service propagates it correctly. And even then, you’re stitching together fragments. You see that Service A called Service B, but you don’t see the latency breakdown. Was it network? Was it processing? Was it waiting for a response from Service C?

Logs also don’t capture the causal chain of a distributed request. Service A calls Service B which calls Service C which calls Service D. If the request fails, logs tell you what each service saw independently. Tracing tells you the entire path, the timing of each hop, and exactly where the failure occurred. A log says “database query took 500ms.” A trace shows you that the query took 100ms, but the request spent 400ms waiting in a connection pool because the database was overloaded.

That’s where distributed tracing comes in.

Distributed Tracing Fundamentals

Distributed tracing works by assigning each request a unique trace ID and following that ID through every service it touches. Each service creates spans—discrete units of work with a start time, end time, and metadata. A trace is a collection of spans that represent a single request’s journey.

Here’s what a trace looks like in practice. A user requests their order history:

Trace ID: 8f1e4b6a9d2c
├─ Span: HTTP GET /orders (API Gateway) [0ms - 245ms]
│  ├─ Span: Authenticate User (Auth Service) [5ms - 25ms]
│  ├─ Span: Fetch Orders (Order Service) [30ms - 180ms]
│  │  ├─ Span: Query PostgreSQL [35ms - 120ms]
│  │  ├─ Span: Redis Cache Lookup [125ms - 128ms]
│  │  └─ Span: Enrich Order Data (Product Service) [135ms - 175ms]
│  └─ Span: Serialize Response [185ms - 240ms]

Each span has tags (metadata about the operation), logs (events that happened during the span), and parent-child relationships. The span in the API gateway is the parent. The spans in downstream services are children. This creates a tree structure that shows the entire request flow.

The magic is in the timing. You can see that the Order Service took 150ms total, but only 85ms of that was actual work. The remaining 65ms was waiting for the Product Service to enrich the data. If that enrichment is slow, you know exactly where to optimize.

Spans also capture errors. If the Product Service throws an exception, the span is marked as failed, and the exception is logged to the span. You see the error in context—not just that it happened, but exactly which service threw it, how long it delayed the overall request, and what the state of the system was when it occurred.

The standard for distributed tracing is OpenTelemetry (OTel). It’s vendor-agnostic, it’s becoming the industry standard, and it works with every major tracing backend: Jaeger, Zipkin, DataDog, New Relic, Honeycomb, and others. If you’re starting fresh, use OpenTelemetry. If you’re already using something proprietary, consider migrating to it—vendor lock-in in observability is expensive.

Trace Context Propagation Across Services

For tracing to work, every service in the chain needs to know the trace ID and parent span ID. That information travels in the request headers. When Service A calls Service B, it includes the trace ID and parent span ID. Service B reads those headers, creates a new span as a child of the parent, and includes those headers when it calls Service C. The chain continues.

This is called trace context propagation, and it’s the foundation of distributed tracing. Without it, you have orphaned spans—traces that don’t connect.

The standard header format is the W3C Trace Context, which uses these headers:

traceparent: 00-8f1e4b6a9d2c5e3f7a1b4c6d-9d2c5e3f-01
tracestate: vendor-specific-data

The format is: version-trace-id-parent-span-id-flags. If you’re using OpenTelemetry, it handles this automatically. But if you’re doing manual instrumentation or integrating with legacy services, you need to propagate these headers explicitly.

Here’s what that looks like in a Node.js service using the OpenTelemetry API:

const tracer = api.trace.getTracer('my-service');
const span = tracer.startSpan('process_order', {
  attributes: {
    'http.method': 'POST',
    'http.url': '/orders',
    'order.id': orderId,
  },
});

// Call downstream service
const context = api.context.with(api.trace.setSpan(api.context.active(), span), () => {
  return fetch('http://payment-service/charge', {
    headers: {
      'traceparent': getTraceparentHeader(span),
    },
    body: JSON.stringify({ orderId, amount }),
  });
});

span.end();

The key insight: trace context propagation is not automatic across service boundaries. If you’re calling an external service via HTTP, you must manually add the trace headers. If you’re using a service mesh like Istio, it can do this transparently. If you’re not, you’re responsible for propagating context.

This is where many teams go wrong. They instrument their services beautifully, but forget to propagate context to third-party APIs, legacy systems, or external vendors. The result is traces that stop abruptly, leaving gaps in visibility.

Instrumentation Strategies: Manual vs Automatic

There are two ways to instrument your services for tracing: automatic instrumentation and manual instrumentation. Most teams use a mix of both.

Automatic instrumentation uses agents or libraries that hook into popular frameworks and libraries. An OpenTelemetry agent can automatically create spans for HTTP requests, database queries, and message queue operations. You add the agent to your application startup and get tracing for free. No code changes required.

The advantage: minimal effort, covers common operations immediately. The disadvantage: less control, potential performance overhead, and you can’t instrument business logic.

Here’s what automatic instrumentation looks like in Node.js with OpenTelemetry:

// Requires zero changes to your application code
const sdk = new NodeSDK({
  traceExporter: new JaegerExporter(),
  instrumentations: [
    new HttpInstrumentation(),
    new PgInstrumentation(),
    new RedisInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

sdk.start();

That’s it. HTTP calls, database queries, and Redis operations are now traced automatically. Every HTTP endpoint gets a span. Every database query gets a span. The overhead is usually 2-5%.

Manual instrumentation gives you full control. You create spans for specific operations and add custom attributes that matter to your business. This is essential for instrumenting business logic—order processing, payment flows, user signup workflows. Automatic instrumentation can’t trace these because they’re application-specific.

Manual instrumentation looks like this:

const tracer = api.trace.getTracer('order-service');

async function processOrder(orderId) {
  const span = tracer.startSpan('process_order', {
    attributes: {
      'order.id': orderId,
      'order.amount': order.total,
      'customer.id': order.customerId,
    },
  });

  try {
    // Charge the customer
    const chargeSpan = tracer.startSpan('charge_customer', {
      parent: span,
    });
    await chargeCustomer(order);
    chargeSpan.end();

    // Send notification
    const notifySpan = tracer.startSpan('send_notification', {
      parent: span,
    });
    await notifyCustomer(order);
    notifySpan.end();

    span.setStatus({ code: SpanStatusCode.OK });
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
}

Now you can see in your tracing backend exactly how long each step of the order process took, whether it succeeded or failed, and which orders are slow. This is where tracing becomes invaluable for product teams, not just infrastructure teams.

The best approach: start with automatic instrumentation for all your standard operations (HTTP, databases, caches), then add manual instrumentation for your core business workflows. This gives you the best of both worlds—minimal effort for baseline coverage, plus deep visibility into what matters to your business.

Sampling and Cost Trade-Offs

Here’s the harsh truth: if you trace every request, your observability costs will explode. A high-traffic service generating 100,000 requests per second, with each request creating 20 spans, means 2 million spans per second. At typical pricing ($0.50 per million spans), that’s $3,000 per day just to store traces.

Most teams can’t afford to trace 100% of traffic. So they sample.

Sampling means tracing only a subset of requests. The simplest approach is head-based sampling: at the entry point (API gateway, load balancer), you decide whether to trace this request or not. If yes, you set the trace ID and propagate the sampling decision downstream. If no, the request is never traced.

The advantage: cheap, simple, reduces overhead. The disadvantage: rare errors might not get sampled. If an error occurs in 0.1% of requests and you’re only sampling 10% of traffic, you might miss it.

Here’s how head-based sampling works:

function shouldSample(request) {
  // Sample 10% of traffic
  if (Math.random() < 0.1) return true;

  // Always sample errors
  if (request.statusCode >= 400) return true;

  // Always sample slow requests
  if (request.duration > 5000) return true;

  return false;
}

A better approach is tail-based sampling: you collect all traces, but only export the ones that matter. Export traces that have errors. Export traces that are slow. Export traces from specific customers or services. This way, you’re not paying to store routine requests, but you’re capturing the ones that need investigation.

The trade-off: tail-based sampling requires a local buffer and more compute on the agent side. But you get much better signal-to-noise ratio.

Here’s a practical sampling strategy we’ve used at scale:

// Export 100% of error traces
if (trace.hasError) {
  return true;
}

// Export 100% of slow traces (>500ms)
if (trace.duration > 500) {
  return true;
}

// Export 100% of traces from high-value customers
if (highValueCustomers.includes(trace.customerId)) {
  return true;
}

// Export 1% of successful, fast traces (baseline sampling)
if (Math.random() < 0.01) {
  return true;
}

return false;

With this strategy, you're capturing all the traces that matter (errors, slow requests, important customers) while keeping costs down on routine traffic. A typical SaaS application might end up sampling 15-25% of traffic this way, which is a massive cost reduction compared to 100% sampling, while retaining visibility into production issues.

Common Pitfalls and How to Avoid Them

We've seen distributed tracing implementations fail in predictable ways. Here's how to avoid them.

Pitfall 1: Forgetting to propagate trace context to async operations. You create a span for processing an HTTP request, then you enqueue a job to a message queue and return. The job runs asynchronously, but the trace ends when the HTTP response is sent. The job's work is invisible.

Solution: extract the trace context before enqueueing the job and inject it when the worker processes the job. This preserves the causal chain.

// In the HTTP handler
const traceContext = getTraceContext(span);
await queue.enqueue('process_order', { orderId, traceContext });

// In the worker
worker.on('job', (job) => {
  const span = tracer.startSpan('process_order_job', {
    attributes: job.data.traceContext,
  });
  // Process the job
  span.end();
});

Pitfall 2: Creating too many spans. If you create a span for every operation, you'll generate so much data that your tracing backend grinds to a halt. We've seen teams accidentally create 10,000 spans per request by instrumenting loops or recursive functions.

Solution: be intentional about what you trace. Trace service boundaries, not every function call. Trace database queries, not every SQL statement. Trace business workflows, not implementation details.

Pitfall 3: Not including enough context in spans. A span that just says "database query" is useless. Was it a SELECT or an UPDATE? How many rows? For which customer? Did it timeout?

Solution: add attributes to every span. Customer ID, user ID, feature flags, request size, response size. These attributes let you slice and dice traces by business dimensions, not just technical metrics.

Pitfall 4: Sampling too aggressively. Teams often sample at 1% or less to save costs, then complain they never see production issues. At 1% sampling, you're missing 99% of errors.

Solution: use tail-based sampling and be more aggressive with error traces. If you're sampling errors at 100% and routine traffic at 5%, you're capturing the signals you actually need while keeping costs reasonable.

Pitfall 5: Choosing a tracing backend and discovering it doesn't scale. Some backends work great for 100 traces per second but fall apart at 10,000. Some charge by the span (expensive). Some charge by the trace (better). Some charge by the gigabyte of ingested data (unpredictable).

Solution: start with an open-source backend like Jaeger or Zipkin in development. For production, choose a managed backend based on your expected volume. DataDog, Honeycomb, and New Relic all handle high volume, but their pricing models differ significantly. Do the math before committing.

A slow request buried in logs is a debugging nightmare. The same request traced across services is a story with a clear beginning, middle, and end. Every service in your architecture should be instrumented for tracing—not because it's trendy, but because when something breaks at 2 a.m., you'll want to see exactly what happened, where it happened, and how long it took. That's what distributed tracing gives you. If you're managing microservices and you haven't implemented observability beyond basic metrics, distributed tracing is the next step—it's the difference between guessing and knowing. We take three engagements a quarter, and many of them start with teams who need to debug production issues faster. Apply for an engagement if you're ready to move beyond logs.