Distributed Tracing

19 min read

Follow a single request as it hops across services, threads, and await points using the tracing crate and OpenTelemetry — the Rust equivalent of OpenTelemetry’s Node SDK plus structured logging, but with spans woven into the type system.

Quick Overview

Distributed tracing records the path of one logical request as a tree of spans — timed units of work — and stitches those spans together across process boundaries by passing a trace context in request headers. In Node you typically reach for @opentelemetry/sdk-node with auto-instrumentation; in Rust the tracing crate provides spans and structured events as a first-class facade, and tracing-opentelemetry exports them to any OpenTelemetry backend (Jaeger, Tempo, Honeycomb, Datadog).

For a TypeScript/JavaScript developer the key mental shift is that tracing unifies what you usually treat as two separate concerns — structured logging (pino/winston) and tracing (OpenTelemetry) — behind one API. A tracing event is a log line; a tracing span is a trace span; both carry typed, structured fields.

Note: This page covers tracing for observability — spans, context, and OpenTelemetry export. For numeric counters/gauges/histograms see metrics.md; for liveness/readiness probes see health-checks.md.

TypeScript/JavaScript Example

A typical Node service wires up the OpenTelemetry SDK, then relies on auto-instrumentation plus a few manual spans. Context propagation across an outgoing HTTP call is handled by the instrumentation, but you often inject/extract headers by hand at service boundaries.

1
// tracing.ts — initialize the OpenTelemetry SDK (run before anything else)
2
import { NodeSDK } from "@opentelemetry/sdk-node";
3
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
4
import { resourceFromAttributes } from "@opentelemetry/resources";
5
import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
6
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
7

8
const sdk = new NodeSDK({
9
  resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: "orders-api" }),
10
  traceExporter: new OTLPTraceExporter({ url: "http://localhost:4317" }),
11
  instrumentations: [getNodeAutoInstrumentations()],
12
});
13
sdk.start();

1
// handler.ts — create a manual span and propagate context downstream
2
import { trace, context, propagation, SpanStatusCode } from "@opentelemetry/api";
3

4
const tracer = trace.getTracer("orders-api");
5

6
export async function getOrder(orderId: number): Promise<Order> {
7
  // Manual span around a unit of work
8
  return tracer.startActiveSpan("get_order", async (span) => {
9
    span.setAttribute("order.id", orderId);
10
    try {
11
      const order = await loadOrder(orderId);
12

13
      // Inject the active trace context into outgoing request headers
14
      const headers: Record<string, string> = {};
15
      propagation.inject(context.active(), headers);
16
      await fetch(`http://billing/charge/${orderId}`, { headers });
17

18
      return order;
19
    } catch (err) {
20
      span.setStatus({ code: SpanStatusCode.ERROR });
21
      throw err;
22
    } finally {
23
      span.end(); // easy to forget — leaks the span if you do
24
    }
25
  });
26
}

The ergonomics are familiar: you start a span, set attributes, and must remember to end() it. Propagation across a fetch is a manual inject into a plain headers object.

Rust Equivalent

In Rust you annotate a function with #[instrument] and the span is created on entry and closed automatically on return — no finally, no leaked spans. Fields are typed and structured.

1
use tracing::{info, instrument, warn};
2
use tracing_subscriber::EnvFilter;
3

4
// Wrapping a function in a span: the span is named after the function and
5
// captures its arguments as fields. It is entered on call, closed on return.
6
#[instrument]
7
fn fetch_user(user_id: u64) -> String {
8
    info!("looking up user in database");
9
    if user_id == 0 {
10
        warn!("user id 0 is reserved");
11
    }
12
    format!("user-{user_id}")
13
}
14

15
// `skip(token)` keeps a secret out of the trace; we still log its length.
16
#[instrument(skip(token))]
17
fn authorize(user_id: u64, token: &str) -> bool {
18
    info!(token_len = token.len(), "checking token");
19
    !token.is_empty()
20
}
21

22
fn main() {
23
    // The "subscriber" is the global sink that collects spans and events.
24
    tracing_subscriber::fmt()
25
        .with_env_filter(EnvFilter::new("info"))
26
        .with_target(false)
27
        .init();
28

29
    let name = fetch_user(42);
30
    let ok = authorize(42, "secret-token");
31
    info!(user = %name, authorized = ok, "request handled");
32
}

1
[dependencies]
2
tracing = "0.1.44"
3
tracing-subscriber = { version = "0.3.23", features = ["env-filter"] }

Real output (colors stripped via NO_COLOR=1):

1
2026-06-02T06:37:43.077448Z  INFO fetch_user{user_id=42}: looking up user in database
2
2026-06-02T06:37:43.077529Z  INFO authorize{user_id=42}: checking token token_len=12
3
2026-06-02T06:37:43.077537Z  INFO request handled user=user-42 authorized=true

Notice how each event is prefixed with the span it occurred in (fetch_user{user_id=42}:). That span context is exactly what OpenTelemetry turns into a parent/child relationship in a trace.

Detailed Explanation

#[instrument] is the workhorse. It is a procedural macro that wraps the function body in a span. The span name defaults to the function name, and every argument is recorded as a field using its Debug representation. On async fn it does the right thing automatically — the span follows the future across every .await, which is the part that is famously easy to get wrong by hand (see Common Pitfalls).

Events vs. spans. info!, warn!, error!, debug!, trace! emit events — point-in-time records, like log lines. A span (#[instrument], info_span!, span.enter()) represents a duration of work with a start and end. Every event fires inside whatever spans are currently active, inheriting their context. This is the unification: one event is simultaneously a log line and a child of the current trace span.

Field syntax. Inside an event or #[instrument(fields(...))]:

field = value records value via its Display/Debug as configured.
%value forces the Display impl (field = %name).
?value forces the Debug impl (field = ?some_struct).
field = expr evaluates an arbitrary expression.

Unlike console.log("user:", name) where everything collapses to a string, these fields stay structured all the way to the backend, so you can query token_len > 10 in Jaeger or Tempo.

The subscriber is the sink. tracing itself only emits — a Subscriber (here tracing_subscriber::fmt()) decides what to do with spans and events. This is the same split as a logging facade (log/slf4j) versus a concrete logger. Swapping the human-readable formatter for a JSON formatter or an OpenTelemetry exporter is a one-line change; your instrumentation never moves.

EnvFilter is the equivalent of DEBUG=app:* / log-level env vars. EnvFilter::new("info") shows info and above; RUST_LOG=orders_api=debug,tower_http=warn (read via EnvFilter::from_default_env()) lets you tune per-module verbosity at runtime without recompiling.

Why no span.end()? Rust’s ownership model closes the span when its guard is dropped at the end of scope — the same RAII mechanism that frees memory. There is no finally, and there is no way to forget. This is a direct consequence of the Drop trait and ownership rules.

Key Differences

Concern	TypeScript/JavaScript (OpenTelemetry SDK)	Rust (`tracing` + `tracing-opentelemetry`)
Logging vs. tracing	Two libraries (`pino` + OTel SDK)	One facade — events are logs and trace data
Creating a span	`tracer.startActiveSpan(...)`, manual `span.end()`	`#[instrument]` — auto-closed on scope exit
Span fields	`span.setAttribute("k", v)`, stringly-typed	Typed structured fields, `Debug`/`Display` controlled
Async correctness	Relies on `AsyncLocalStorage` context	`#[instrument]` / `.instrument()` carry the span across `.await`
Forgetting to end	Leaks the span	Impossible — RAII closes it
Auto-instrumentation	Rich (http, pg, redis monkey-patched)	Library-provided spans (`tower-http`, `sqlx`) — opt in, no monkey-patching
Backend export	`NodeSDK` + exporter	A `tracing` layer + OpenTelemetry exporter
Sampling/perf when off	Runtime checks	Disabled spans compile to near-zero overhead

The two genuinely important differences for a Node developer:

There is no global monkey-patching. Node’s auto-instrumentation rewrites http, pg, etc. at load time. Rust libraries instead ship tracing spans (axum via tower-http::TraceLayer, sqlx via its own spans) that you enable explicitly. More boilerplate, zero hidden magic.
Disabled instrumentation is nearly free. A span filtered out by the subscriber costs a single atomic check; in JS, instrumentation overhead is paid regardless. You can leave #[instrument] everywhere in production.

Common Pitfalls

Pitfall 1: `#[instrument]` on a function with a non-`Debug` argument

#[instrument] records every argument by default, which requires each one to implement Debug. Passing a connection pool or other opaque type fails to compile:

1
use tracing::instrument;
2

3
// A type that does NOT implement Debug.
4
struct DbPool;
5

6
#[instrument] // does not compile (error[E0277]: `DbPool` doesn't implement `Debug`)
7
fn query(pool: DbPool, sql: &str) {
8
    println!("{sql}");
9
}
10

11
fn main() {
12
    query(DbPool, "SELECT 1");
13
}

The real compiler error:

1
error[E0277]: `DbPool` doesn't implement `Debug`
2
   --> src/main.rs:6:1
3
    |
4
  6 | #[instrument]
5
    | ^^^^^^^^^^^^^ the trait `Debug` is not implemented for `DbPool`
6
    |
7
    = note: add `#[derive(Debug)]` to `DbPool` or manually `impl Debug for DbPool`
8
    = note: required for `&DbPool` to implement `Debug`
9
note: required by a bound in `debug`
10
help: consider annotating `DbPool` with `#[derive(Debug)]`

Fix: skip the argument with #[instrument(skip(pool))] (you rarely want a pool in a trace anyway), or skip_all and re-add the fields you care about. Never put secrets (tokens, passwords, PII) in fields — skip them.

Pitfall 2: holding a span guard across `.await`

You can enter a span manually with let _guard = span.enter(). This works for synchronous code, but it is a logic bug in async code — and notably one the compiler does not catch:

1
use tracing::info_span;
2

3
async fn fetch() -> u32 { 42 }
4

5
async fn handler() {
6
    let span = info_span!("handler");
7
    let _guard = span.enter();   // compiles fine, but WRONG in async
8
    let value = fetch().await;   // span stays "entered" while the task is suspended
9
    println!("{value}");
10
}

This compiles and runs, so it is easy to ship. The problem is semantic: when the future suspends at .await, the guard is still alive, so on a multi-threaded runtime the executor may run an unrelated task while your span is marked active — attributing that task’s events to the wrong span. Fix: use #[instrument] on the async fn, or future.instrument(span).await, both of which correctly exit the span on every suspension and re-enter on resume:

1
use tracing::{info, info_span, Instrument};
2

3
async fn fetch() -> u32 { 42 }
4

5
async fn handler() {
6
    let span = info_span!("handler");
7
    async {
8
        let value = fetch().await;
9
        info!(value, "fetched");
10
    }
11
    .instrument(span) // correct: span follows the future across .await
12
    .await;
13
}

Pitfall 3: forgetting to flush the exporter on shutdown

OpenTelemetry batches spans and exports them in the background. If your process exits without shutting the provider down, the last batch of spans is lost — traces just before a crash or a normal exit go missing. Always call provider.shutdown() before main returns (or wire it into your graceful shutdown path). Unlike a forgotten span.end(), this one will silently drop data.

Pitfall 4: assuming events without a span are correlated

A bare info!("processing") with no enclosing span has no trace context. In a request handler, make sure the handler itself is instrumented (or sits under tower-http::TraceLayer) so events inherit a span and a trace_id.

Best Practices

Instrument at boundaries. Put #[instrument] on request handlers, service-layer functions, and any function that does I/O. Skip trivial helpers; spans have a (small) cost and clutter traces.
skip noisy and sensitive arguments. Use #[instrument(skip(pool, password), fields(user_id = %user.id))] to keep traces clean and safe. Never trace secrets or PII.
Prefer #[instrument]/.instrument() over .enter() in async code. Reserve .enter() for purely synchronous scopes.
Use EnvFilter from the environment. Wire RUST_LOG so operators can raise verbosity without a redeploy.
Set a Resource with service.name. Without it, your service shows up as unknown_service in the backend.
Use a batch exporter in production, a simple/synchronous one only in tests. Batch reduces export overhead dramatically.
Emit JSON in production so a log pipeline can index fields; keep the pretty formatter for local dev.
Propagate W3C traceparent (the default) so traces join across services, including non-Rust ones.

The JSON formatter is one line and gives a structured, machine-parseable stream:

1
use tracing::{info, instrument};
2
use tracing_subscriber::EnvFilter;
3

4
#[instrument(fields(request_id = %uuid_stub()))]
5
fn handle_request(path: &str) {
6
    info!("received request");
7
    let user = load_user(7);
8
    info!(user = %user, "request complete");
9
}
10

11
#[instrument]
12
fn load_user(id: u64) -> String {
13
    info!("querying users table");
14
    format!("user-{id}")
15
}
16

17
fn uuid_stub() -> &'static str {
18
    "req-abc123"
19
}
20

21
fn main() {
22
    tracing_subscriber::fmt()
23
        .json()
24
        .with_env_filter(EnvFilter::new("info"))
25
        .with_current_span(true)
26
        .with_span_list(true)
27
        .init();
28

29
    handle_request("/users/7");
30
}

Real output (one JSON object per line; spans shows the full ancestry):

1
{"timestamp":"2026-06-02T06:37:54.222585Z","level":"INFO","fields":{"message":"received request"},"target":"probe","span":{"path":"/users/7","request_id":"req-abc123","name":"handle_request"},"spans":[{"path":"/users/7","request_id":"req-abc123","name":"handle_request"}]}
2
{"timestamp":"2026-06-02T06:37:54.224232Z","level":"INFO","fields":{"message":"querying users table"},"target":"probe","span":{"id":7,"name":"load_user"},"spans":[{"path":"/users/7","request_id":"req-abc123","name":"handle_request"},{"id":7,"name":"load_user"}]}
3
{"timestamp":"2026-06-02T06:37:54.224512Z","level":"INFO","fields":{"message":"request complete","user":"user-7"},"target":"probe","span":{"path":"/users/7","request_id":"req-abc123","name":"handle_request"},"spans":[{"path":"/users/7","request_id":"req-abc123","name":"handle_request"}]}

Real-World Example

A production axum service that (a) exports spans to an OpenTelemetry Collector over OTLP/gRPC, (b) extracts the upstream W3C trace context from incoming request headers so this service’s spans become children of the caller’s trace, and (c) uses tower-http’s TraceLayer to span every request.

The current OpenTelemetry Rust API is builder-based: SpanExporter::builder().with_tonic().build() creates the OTLP exporter, and SdkTracerProvider::builder()...build() wires it up. The current stable toolchain is Rust 1.96.0 on the 2024 edition; cargo new selects it automatically.

1
[dependencies]
2
axum = "0.8.9"
3
tokio = { version = "1.52.3", features = ["full"] }
4
anyhow = "1.0.102"
5
tower-http = { version = "0.6.11", features = ["trace"] }
6
tracing = "0.1.44"
7
tracing-subscriber = { version = "0.3.23", features = ["env-filter", "json"] }
8
tracing-opentelemetry = "0.33.0"
9
opentelemetry = { version = "0.32.0", features = ["trace"] }
10
opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
11
opentelemetry-otlp = { version = "0.32.0", features = ["grpc-tonic"] }

1
use axum::extract::Path;
2
use axum::http::HeaderMap;
3
use axum::routing::get;
4
use axum::Router;
5
use opentelemetry::global;
6
use opentelemetry::propagation::Extractor;
7
use opentelemetry::trace::TracerProvider as _;
8
use opentelemetry::KeyValue;
9
use opentelemetry_otlp::SpanExporter;
10
use opentelemetry_sdk::propagation::TraceContextPropagator;
11
use opentelemetry_sdk::trace::SdkTracerProvider;
12
use opentelemetry_sdk::Resource;
13
use tower_http::trace::TraceLayer;
14
use tracing::{info, info_span, instrument, Instrument};
15
use tracing_opentelemetry::OpenTelemetrySpanExt;
16
use tracing_subscriber::layer::SubscriberExt;
17
use tracing_subscriber::util::SubscriberInitExt;
18
use tracing_subscriber::EnvFilter;
19

20
// Read W3C trace headers (traceparent/tracestate) off the incoming request.
21
struct HeaderExtractor<'a>(&'a HeaderMap);
22
impl Extractor for HeaderExtractor<'_> {
23
    fn get(&self, key: &str) -> Option<&str> {
24
        self.0.get(key).and_then(|v| v.to_str().ok())
25
    }
26
    fn keys(&self) -> Vec<&str> {
27
        self.0.keys().map(|k| k.as_str()).collect()
28
    }
29
}
30

31
#[instrument(skip(pool))]
32
async fn load_order(pool: &str, order_id: u64) -> String {
33
    info!("querying database");
34
    format!("order #{order_id} from {pool}")
35
}
36

37
async fn get_order(headers: HeaderMap, Path(order_id): Path<u64>) -> String {
38
    // Continue the distributed trace started by the upstream caller.
39
    let parent_cx =
40
        global::get_text_map_propagator(|prop| prop.extract(&HeaderExtractor(&headers)));
41

42
    let span = info_span!("get_order", %order_id);
43
    // Make the extracted remote context this span's parent.
44
    let _ = span.set_parent(parent_cx);
45

46
    async move {
47
        info!("handling order request");
48
        load_order("pg-pool", order_id).await
49
    }
50
    .instrument(span)
51
    .await
52
}
53

54
fn init_telemetry() -> anyhow::Result<SdkTracerProvider> {
55
    // Use the W3C TraceContext propagator (the de-facto standard, matches Node).
56
    global::set_text_map_propagator(TraceContextPropagator::new());
57

58
    // OTLP/gRPC exporter — default endpoint http://localhost:4317.
59
    let exporter = SpanExporter::builder().with_tonic().build()?;
60

61
    let resource = Resource::builder()
62
        .with_service_name("orders-api")
63
        .with_attribute(KeyValue::new("deployment.environment", "production"))
64
        .build();
65

66
    let provider = SdkTracerProvider::builder()
67
        .with_batch_exporter(exporter)
68
        .with_resource(resource)
69
        .build();
70

71
    let tracer = provider.tracer("orders-api");
72

73
    tracing_subscriber::registry()
74
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
75
        .with(tracing_subscriber::fmt::layer())
76
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
77
        .init();
78

79
    Ok(provider)
80
}
81

82
#[tokio::main]
83
async fn main() -> anyhow::Result<()> {
84
    let provider = init_telemetry()?;
85

86
    let app = Router::new()
87
        .route("/orders/{id}", get(get_order))
88
        .layer(TraceLayer::new_for_http());
89

90
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
91
    info!("listening on {}", listener.local_addr()?);
92
    axum::serve(listener, app).await?;
93

94
    // Flush any buffered spans before exit so the last batch is not lost.
95
    provider.shutdown()?;
96
    Ok(())
97
}

Note: This uses the current axum 0.8 path syntax {id} (not the old :id) and axum::serve(listener, app) (not the removed Server::bind().serve()). The OTLP exporter targets a running OpenTelemetry Collector or a Jaeger instance with OTLP enabled on port 4317; without one, send() simply fails to export and logs a warning rather than crashing your service.

The matching outbound side — injecting context into a downstream call so the trace continues — uses the symmetric inject_context and an Injector. Here is a self-contained, runnable demo of both halves using a plain HashMap as the carrier, which makes the actual traceparent header visible:

1
use std::collections::HashMap;
2

3
use opentelemetry::global;
4
use opentelemetry::propagation::{Extractor, Injector};
5
use opentelemetry::trace::TracerProvider as _;
6
use opentelemetry_sdk::propagation::TraceContextPropagator;
7
use opentelemetry_sdk::trace::SdkTracerProvider;
8
use tracing::{info, info_span, instrument, Span};
9
use tracing_opentelemetry::OpenTelemetrySpanExt;
10
use tracing_subscriber::layer::SubscriberExt;
11
use tracing_subscriber::util::SubscriberInitExt;
12
use tracing_subscriber::EnvFilter;
13

14
// Writes trace headers into an outgoing request.
15
struct HeaderInjector<'a>(&'a mut HashMap<String, String>);
16
impl Injector for HeaderInjector<'_> {
17
    fn set(&mut self, key: &str, value: String) {
18
        self.0.insert(key.to_string(), value);
19
    }
20
}
21

22
// Reads trace headers from an incoming request.
23
struct HeaderExtractor<'a>(&'a HashMap<String, String>);
24
impl Extractor for HeaderExtractor<'_> {
25
    fn get(&self, key: &str) -> Option<&str> {
26
        self.0.get(key).map(|s| s.as_str())
27
    }
28
    fn keys(&self) -> Vec<&str> {
29
        self.0.keys().map(|s| s.as_str()).collect()
30
    }
31
}
32

33
// Service A: starts a span, injects its context into outgoing headers.
34
#[instrument]
35
fn service_a_handler() -> HashMap<String, String> {
36
    info!("service A handling request");
37
    let mut headers = HashMap::new();
38
    let cx = Span::current().context();
39
    global::get_text_map_propagator(|prop| {
40
        prop.inject_context(&cx, &mut HeaderInjector(&mut headers));
41
    });
42
    headers
43
}
44

45
// Service B: extracts the upstream context and adopts it as the parent.
46
fn service_b_handler(headers: &HashMap<String, String>) {
47
    let parent_cx =
48
        global::get_text_map_propagator(|prop| prop.extract(&HeaderExtractor(headers)));
49
    let span = info_span!("service_b_handler");
50
    let _ = span.set_parent(parent_cx);
51
    let _guard = span.enter(); // synchronous code — .enter() is fine here
52
    info!("service B continued the upstream trace");
53
}
54

55
fn main() {
56
    global::set_text_map_propagator(TraceContextPropagator::new());
57

58
    let provider = SdkTracerProvider::builder().build();
59
    let tracer = provider.tracer("propagation-demo");
60

61
    tracing_subscriber::registry()
62
        .with(EnvFilter::new("info"))
63
        .with(tracing_subscriber::fmt::layer().without_time())
64
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
65
        .init();
66

67
    let headers = service_a_handler();
68
    println!("--- headers passed between services ---");
69
    for (k, v) in &headers {
70
        println!("{k}: {v}");
71
    }
72
    println!("---------------------------------------");
73
    service_b_handler(&headers);
74
}

Real output (NO_COLOR=1; the traceparent value is non-deterministic):

1
 INFO service_a_handler: probe: service A handling request
2
--- headers passed between services ---
3
traceparent: 00-7c6ef9ae1f7cac5d3cb9108d400378ee-d2e7e5752bc0dca4-01
4
tracestate:
5
---------------------------------------
6
 INFO service_b_handler: probe: service B continued the upstream trace

The traceparent header is the W3C standard format version-trace_id-span_id-flags. Because service B extracts that trace_id and calls set_parent, its span lands under the same trace in your backend — the request becomes one connected tree across both processes. This is identical to what OpenTelemetry’s JS propagation.inject/extract produce, which is why a Rust service and a Node service interoperate out of the box.

Exercises

Exercise 1: Record a field you only know later

Difficulty: Beginner

Objective: Learn to declare a span field up front and fill it in once the value is computed — the pattern for recording a result (row count, status code) on the span that produced it.

Instructions: Write a function run_query(sql: &str) -> usize instrumented so its span declares an empty rows field. Inside, after “running” the query (just return a constant), record the row count onto the current span with Span::current().record("rows", ...). Emit an info! before and after. Initialize a plain fmt subscriber.

Solution

1
use tracing::{info, instrument, Span};
2
use tracing_subscriber::EnvFilter;
3

4
// Declare an empty field, then fill it in once the value is known.
5
#[instrument(fields(rows = tracing::field::Empty))]
6
fn run_query(sql: &str) -> usize {
7
    info!("executing query");
8
    let rows = 7; // pretend we actually ran it
9
    Span::current().record("rows", rows);
10
    rows
11
}
12

13
fn main() {
14
    tracing_subscriber::fmt()
15
        .with_env_filter(EnvFilter::new("info"))
16
        .with_target(false)
17
        .without_time()
18
        .init();
19

20
    let n = run_query("SELECT * FROM users");
21
    info!(returned = n, "done");
22
}

Real output:

1
 INFO run_query{sql="SELECT * FROM users"}: executing query
2
 INFO done returned=7

The rows field is attached to the span (visible when the span itself is reported by a backend or a span-aware formatter), while returned=7 is a field on the final event. The key technique is tracing::field::Empty to reserve a slot plus Span::current().record(...) to populate it.

Exercise 2: Inject trace context into an outgoing `reqwest` call

Difficulty: Intermediate

Objective: Implement the outbound half of context propagation so a downstream HTTP service joins the same trace.

Instructions: Implement an Injector for a reqwest::header::HeaderMap. Write async fn call_downstream(client: &reqwest::Client, url: &str) that grabs the current span’s OpenTelemetry context, injects it into a fresh HeaderMap via the active text-map propagator, and sends a GET with those headers. (reqwest with default-features = false, features = ["json"].)

Solution

1
use opentelemetry::global;
2
use opentelemetry::propagation::Injector;
3
use tracing::Span;
4
use tracing_opentelemetry::OpenTelemetrySpanExt;
5

6
// Carrier that injects W3C trace headers into a reqwest header map.
7
struct ReqwestInjector<'a>(&'a mut reqwest::header::HeaderMap);
8
impl Injector for ReqwestInjector<'_> {
9
    fn set(&mut self, key: &str, value: String) {
10
        if let (Ok(name), Ok(val)) = (
11
            reqwest::header::HeaderName::from_bytes(key.as_bytes()),
12
            reqwest::header::HeaderValue::from_str(&value),
13
        ) {
14
            self.0.insert(name, val);
15
        }
16
    }
17
}
18

19
async fn call_downstream(client: &reqwest::Client, url: &str) -> reqwest::Result<()> {
20
    let mut headers = reqwest::header::HeaderMap::new();
21
    let cx = Span::current().context();
22
    global::get_text_map_propagator(|prop| {
23
        prop.inject_context(&cx, &mut ReqwestInjector(&mut headers));
24
    });
25
    let _resp = client.get(url).headers(headers).send().await?;
26
    Ok(())
27
}
28

29
fn main() {
30
    // Compile-only demo: shows the injection wiring.
31
    let _ = call_downstream;
32
}

1
# Cargo.toml additions
2
[dependencies]
3
reqwest = { version = "0.13.4", default-features = false, features = ["json"] }
4
opentelemetry = { version = "0.32.0", features = ["trace"] }
5
opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
6
tracing = "0.1.44"
7
tracing-opentelemetry = "0.33.0"

The inject_context call writes traceparent (and tracestate) into headers; the downstream service’s Extractor (Exercise/Real-World pattern) reads them back and calls set_parent, joining both processes into one trace.

Exercise 3: Per-module log levels with `EnvFilter`

Difficulty: Intermediate

Objective: Use RUST_LOG-style directives to silence a noisy dependency while keeping your own module verbose.

Instructions: Build a subscriber whose filter is read from the environment, falling back to a directive that sets your crate to debug but a (simulated) noisy module to warn. Emit a debug! from your code and a warn! “from” the noisy module (use #[instrument]/events inside a mod noisy), and confirm only the warning from the noisy module appears while your debug line shows.

Solution

1
use tracing::{debug, warn};
2
use tracing_subscriber::EnvFilter;
3

4
mod noisy {
5
    use tracing::{debug, warn};
6
    pub fn chatter() {
7
        debug!("noisy debug — should be filtered out");
8
        warn!("noisy warning — should appear");
9
    }
10
}
11

12
fn main() {
13
    // Operators can override via RUST_LOG; otherwise use this default.
14
    let filter = EnvFilter::try_from_default_env()
15
        .unwrap_or_else(|_| EnvFilter::new("exercise=debug,exercise::noisy=warn"));
16

17
    tracing_subscriber::fmt()
18
        .with_env_filter(filter)
19
        .with_target(true)
20
        .without_time()
21
        .init();
22

23
    debug!("my own debug — should appear");
24
    noisy::chatter();
25
}

Run it with the crate named exercise (cargo new --name exercise ...). With no RUST_LOG set, real output is:

1
DEBUG exercise: my own debug — should appear
2
 WARN exercise::noisy: noisy warning — should appear

The exercise::noisy=warn directive raises the threshold for just that module, so its debug! is dropped while your crate-level debug! survives — the same idea as DEBUG=app:*,-app:db in a Node app, but resolved per module path. Setting RUST_LOG=trace at runtime overrides the fallback entirely with no recompile.

Distributed Tracing

Quick Overview

TypeScript/JavaScript Example

Rust Equivalent

Detailed Explanation

Key Differences

Common Pitfalls

Pitfall 1: #[instrument] on a function with a non-Debug argument

Pitfall 2: holding a span guard across .await

Pitfall 3: forgetting to flush the exporter on shutdown

Pitfall 4: assuming events without a span are correlated

Best Practices

Real-World Example

Further Reading

Exercises

Exercise 1: Record a field you only know later

Exercise 2: Inject trace context into an outgoing reqwest call

Exercise 3: Per-module log levels with EnvFilter

Pitfall 1: `#[instrument]` on a function with a non-`Debug` argument

Pitfall 2: holding a span guard across `.await`

Exercise 2: Inject trace context into an outgoing `reqwest` call

Exercise 3: Per-module log levels with `EnvFilter`