OpenTelemetry traces in New Relic

OpenTelemetry provides a rich tracing ecosystem, with an API for recording trace telemetry, an SDK for exporting span data, and context propagation for tracing across application boundaries.

This page describes how New Relic handles OpenTelemetry spans it receives via the New Relic OTLP endpoint. To send OpenTelemetry traces to New Relic, you'll need to configure your trace source to export data to the New Relic OpenTelemetry Collector via the OTLP endpoint. See the following pages:

For endpoint configuration requirements, see New Relic OTLP endpoint.
For instructions to configure services with OpenTelemetry, see OpenTelemetry APM monitoring.

OTLP span mapping

New Relic maps OTLP spans to the Span data type. The table below describes how fields from the trace proto message definitions are interpreted:

OTLP proto field	New Relic `Span` field
`ResourceSpans.Resource.attributes`	Each key/value is an attribute on the `Span` [1]
`ScopeSpans.InstrumentationScope.name`	`otel.library.name`
`ScopeSpans.InstrumentationScope.version`	`otel.library.version`
`ScopeSpans.InstrumentationScope.attributes`	Each key/value is an attribute on the `Span` [1]
`Span.trace_id`	`trace.id`
`Span.span_id`	`id`
`Span.trace_state`	`w3c.tracestate`
`Span.parent_span_id`	`parent.id`
`Span.name`	`name`
`Span.kind`	`span.kind`
`Span.start_time_unix_nano`	`timestamp`
`Span.end_time_unix_nano`	`duration.ms` (computed with `Span.start_time_unix_nano`)
`Span.attributes`	Each key/value is an attribute on the `Span` [1]
`Span.dropped_attribute_count`	`otel.dropped_attributes_count`
`Span.events`	Each event is recorded as a `SpanEvent` with `span.id` / `trace.id` referring to source span, count stored as `nr.spanEventCount`
`Span.events[*].time_unix_nano`	Stored as `timestamp` on `SpanEvent`
`Span.events[*].name`	Stored as `name` on `SpanEvent`
`Span.events[*].attributes`	Each key/value is stored as an attribute on `SpanEvent`
`Span.events[*].dropped_attributes_count`	Stored as `ote.dropped_Attributes_count` on `SpanEvent`
`Span.dropped_events_count`	`otel.dropped_events_count`
`Span.status.message`	`otel.status_description`
`Span.status.code`	`otel.status_code`

Table footnotes

[1] In case of conflict in resource attributes, scope attributes, span attributes, and top level span fields, the order of precedent (highest to lowest) is the top level Span.* fields > Span.attributes > ScopeSpans.InstrumentationScope.attributes > ResourceSpans.Resource.attributes.

See OTLP attribute types for details on New Relic OTLP endpoint supported attribute types and OTLP attribute limits for details on validation performed on attributes.

Span links

New Relic supports OpenTelemetry span links, which allow you to create causal relationships between spans that don't have a direct parent-child connection. Span links are essential for understanding distributed traces that get split across asynchronous boundaries like message queues, event streams, and batch processing systems.

When to use span links

Use span links in the following scenarios:

Message queue producers and consumers: Link a consuming span back to the producing span when processing messages from queues like AWS SQS, RabbitMQ, or Kafka.
Fan-in patterns: Link multiple producer traces to a single consumer trace that aggregates their outputs.
Batch processing: Link spans that process batched messages back to their individual originating traces.
Long-running workflows: Connect spans across workflow steps that exceed normal trace duration limits.

Implementing span links

To implement span links in your OpenTelemetry instrumentation, you need to:

Extract trace context from the incoming message or event
Create a span link when starting a new span in the consumer
Ensure trace context is propagated through your messaging infrastructure

The following examples show how to implement span links in different languages:

from opentelemetry import trace
from opentelemetry.trace import Link, SpanContext, TraceFlags
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer(__name__)
propagator = TraceContextTextMapPropagator()

# Producer: Publishing a message with trace context
def publish_message(queue, message_body):
    with tracer.start_as_current_span("publish_message") as span:
        # Prepare message with trace context headers
        carrier = {}
        propagator.inject(carrier)

        # Add carrier headers to your message metadata
        message = {
            'body': message_body,
            'headers': carrier
        }

        queue.publish(message)
        span.set_attribute("messaging.destination", queue.name)
        span.set_attribute("messaging.system", "custom_queue")

# Consumer: Processing a message with span link
def process_message(message):
    # Extract trace context from message headers
    carrier = message.get('headers', {})
    ctx = propagator.extract(carrier)

    # Get the span context from the extracted context
    span_context = trace.get_current_span(ctx).get_span_context()

    # Create a new span with a link to the producer span
    links = []
    if span_context.is_valid:
        links = [Link(span_context)]

    with tracer.start_as_current_span(
        "process_message",
        links=links
    ) as span:
        # Process the message
        result = handle_message(message['body'])

        span.set_attribute("messaging.system", "custom_queue")
        span.set_attribute("messaging.operation", "process")

        return result

For AWS SQS specifically:

import boto3
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

sqs = boto3.client('sqs')
propagator = TraceContextTextMapPropagator()

# Publishing to SQS
def send_sqs_message(queue_url, message_body):
    with tracer.start_as_current_span("sqs_publish") as span:
        carrier = {}
        propagator.inject(carrier)

        # SQS message attributes for trace context
        message_attributes = {
            'traceparent': {
                'StringValue': carrier.get('traceparent', ''),
                'DataType': 'String'
            }
        }

        if 'tracestate' in carrier:
            message_attributes['tracestate'] = {
                'StringValue': carrier['tracestate'],
                'DataType': 'String'
            }

        sqs.send_message(
            QueueUrl=queue_url,
            MessageBody=message_body,
            MessageAttributes=message_attributes
        )

# Consuming from SQS
def process_sqs_message(message):
    # Extract trace context from SQS message attributes
    carrier = {}
    if 'MessageAttributes' in message:
        attrs = message['MessageAttributes']
        if 'traceparent' in attrs:
            carrier['traceparent'] = attrs['traceparent']['StringValue']
        if 'tracestate' in attrs:
            carrier['tracestate'] = attrs['tracestate']['StringValue']

    ctx = propagator.extract(carrier)
    span_context = trace.get_current_span(ctx).get_span_context()

    links = [Link(span_context)] if span_context.is_valid else []

    with tracer.start_as_current_span(
        "sqs_process",
        links=links
    ) as span:
        # Process message
        body = message['Body']
        return handle_message(body)

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanBuilder;
import io.opentelemetry.api.trace.SpanContext;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Context;
import io.opentelemetry.context.propagation.TextMapGetter;
import io.opentelemetry.context.propagation.TextMapPropagator;
import io.opentelemetry.context.propagation.TextMapSetter;

public class MessageProcessor {
    private final Tracer tracer;
    private final TextMapPropagator propagator;

    // Producer: Publishing a message with trace context
    public void publishMessage(Queue queue, String messageBody) {
        Span span = tracer.spanBuilder("publish_message")
            .startSpan();

        try (var scope = span.makeCurrent()) {
            // Inject trace context into message headers
            Map<String, String> headers = new HashMap<>();
            propagator.inject(Context.current(), headers,
                (carrier, key, value) -> carrier.put(key, value));

            Message message = new Message(messageBody, headers);
            queue.publish(message);

            span.setAttribute("messaging.destination", queue.getName());
            span.setAttribute("messaging.system", "custom_queue");
        } finally {
            span.end();
        }
    }

    // Consumer: Processing a message with span link
    public void processMessage(Message message) {
        // Extract trace context from message headers
        Context extractedContext = propagator.extract(
            Context.current(),
            message.getHeaders(),
            (carrier, key) -> carrier.get(key)
        );

        // Get the span context from extracted context
        SpanContext producerSpanContext = Span.fromContext(extractedContext)
            .getSpanContext();

        // Create span with link to producer
        SpanBuilder spanBuilder = tracer.spanBuilder("process_message");

        if (producerSpanContext.isValid()) {
            spanBuilder.addLink(producerSpanContext);
        }

        Span span = spanBuilder.startSpan();

        try (var scope = span.makeCurrent()) {
            handleMessage(message.getBody());

            span.setAttribute("messaging.system", "custom_queue");
            span.setAttribute("messaging.operation", "process");
        } finally {
            span.end();
        }
    }

    // AWS SQS example
    public void processSQSMessage(
        software.amazon.awssdk.services.sqs.model.Message sqsMessage
    ) {
        Map<String, String> carrier = new HashMap<>();

        // Extract trace context from SQS message attributes
        sqsMessage.messageAttributes().forEach((key, value) -> {
            if (key.equals("traceparent") || key.equals("tracestate")) {
                carrier.put(key, value.stringValue());
            }
        });

        Context extractedContext = propagator.extract(
            Context.current(),
            carrier,
            (c, k) -> c.get(k)
        );

        SpanContext producerSpanContext = Span.fromContext(extractedContext)
            .getSpanContext();

        SpanBuilder spanBuilder = tracer.spanBuilder("sqs_process");

        if (producerSpanContext.isValid()) {
            spanBuilder.addLink(producerSpanContext);
        }

        Span span = spanBuilder.startSpan();

        try (var scope = span.makeCurrent()) {
            handleMessage(sqsMessage.body());
            span.setAttribute("messaging.system", "AmazonSQS");
        } finally {
            span.end();
        }
    }
}

const { trace, context, SpanKind } = require('@opentelemetry/api');
const { W3CTraceContextPropagator } = require('@opentelemetry/core');

const tracer = trace.getTracer('message-processor');
const propagator = new W3CTraceContextPropagator();

// Producer: Publishing a message with trace context
async function publishMessage(queue, messageBody) {
  const span = tracer.startSpan('publish_message', {
    kind: SpanKind.PRODUCER
  });

  return context.with(trace.setSpan(context.active(), span), async () => {
    try {
      // Inject trace context into message headers
      const carrier = {};
      propagator.inject(
        context.active(),
        carrier,
        {
          set: (carrier, key, value) => {
            carrier[key] = value;
          }
        }
      );

      const message = {
        body: messageBody,
        headers: carrier
      };

      await queue.publish(message);

      span.setAttribute('messaging.destination', queue.name);
      span.setAttribute('messaging.system', 'custom_queue');
    } finally {
      span.end();
    }
  });
}

// Consumer: Processing a message with span link
async function processMessage(message) {
  // Extract trace context from message headers
  const extractedContext = propagator.extract(
    context.active(),
    message.headers || {},
    {
      get: (carrier, key) => carrier[key]
    }
  );

  // Get the span context from extracted context
  const producerSpan = trace.getSpan(extractedContext);
  const producerSpanContext = producerSpan?.spanContext();

  // Create span with link
  const links = [];
  if (producerSpanContext && trace.isSpanContextValid(producerSpanContext)) {
    links.push({
      context: producerSpanContext
    });
  }

  const span = tracer.startSpan(
    'process_message',
    {
      kind: SpanKind.CONSUMER,
      links: links
    }
  );

  return context.with(trace.setSpan(context.active(), span), async () => {
    try {
      await handleMessage(message.body);

      span.setAttribute('messaging.system', 'custom_queue');
      span.setAttribute('messaging.operation', 'process');
    } finally {
      span.end();
    }
  });
}

// AWS SQS example using AWS SDK v3
const { SQSClient, SendMessageCommand, ReceiveMessageCommand } = require('@aws-sdk/client-sqs');

async function sendSQSMessage(queueUrl, messageBody) {
  const span = tracer.startSpan('sqs_publish');

  return context.with(trace.setSpan(context.active(), span), async () => {
    try {
      const carrier = {};
      propagator.inject(context.active(), carrier, {
        set: (c, k, v) => { c[k] = v; }
      });

      const messageAttributes = {
        traceparent: {
          StringValue: carrier.traceparent || '',
          DataType: 'String'
        }
      };

      if (carrier.tracestate) {
        messageAttributes.tracestate = {
          StringValue: carrier.tracestate,
          DataType: 'String'
        };
      }

      const client = new SQSClient({});
      await client.send(new SendMessageCommand({
        QueueUrl: queueUrl,
        MessageBody: messageBody,
        MessageAttributes: messageAttributes
      }));
    } finally {
      span.end();
    }
  });
}

async function processSQSMessage(message) {
  const carrier = {};

  if (message.MessageAttributes) {
    if (message.MessageAttributes.traceparent) {
      carrier.traceparent = message.MessageAttributes.traceparent.StringValue;
    }
    if (message.MessageAttributes.tracestate) {
      carrier.tracestate = message.MessageAttributes.tracestate.StringValue;
    }
  }

  const extractedContext = propagator.extract(context.active(), carrier, {
    get: (c, k) => c[k]
  });

  const producerSpanContext = trace.getSpan(extractedContext)?.spanContext();
  const links = producerSpanContext && trace.isSpanContextValid(producerSpanContext)
    ? [{ context: producerSpanContext }]
    : [];

  const span = tracer.startSpan('sqs_process', { links });

  return context.with(trace.setSpan(context.active(), span), async () => {
    try {
      await handleMessage(message.Body);
      span.setAttribute('messaging.system', 'AmazonSQS');
    } finally {
      span.end();
    }
  });
}

package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/trace"
)

var (
    tracer     = otel.Tracer("message-processor")
    propagator = propagation.TraceContext{}
)

// Producer: Publishing a message with trace context
func publishMessage(ctx context.Context, queue Queue, messageBody string) error {
    ctx, span := tracer.Start(ctx, "publish_message")
    defer span.End()

    // Inject trace context into message headers
    carrier := propagation.MapCarrier{}
    propagator.Inject(ctx, carrier)

    message := Message{
        Body:    messageBody,
        Headers: map[string]string(carrier),
    }

    err := queue.Publish(message)

    span.SetAttributes(
        attribute.String("messaging.destination", queue.Name()),
        attribute.String("messaging.system", "custom_queue"),
    )

    return err
}

// Consumer: Processing a message with span link
func processMessage(ctx context.Context, message Message) error {
    // Extract trace context from message headers
    carrier := propagation.MapCarrier(message.Headers)
    extractedCtx := propagator.Extract(ctx, carrier)

    // Get the span context from extracted context
    producerSpanContext := trace.SpanContextFromContext(extractedCtx)

    // Create span with link
    var links []trace.Link
    if producerSpanContext.IsValid() {
        links = []trace.Link{
            {
                SpanContext: producerSpanContext,
            },
        }
    }

    ctx, span := tracer.Start(
        ctx,
        "process_message",
        trace.WithLinks(links...),
    )
    defer span.End()

    err := handleMessage(message.Body)

    span.SetAttributes(
        attribute.String("messaging.system", "custom_queue"),
        attribute.String("messaging.operation", "process"),
    )

    return err
}

// AWS SQS example
func processSQSMessage(ctx context.Context, sqsMessage *sqs.Message) error {
    // Extract trace context from SQS message attributes
    carrier := propagation.MapCarrier{}

    if sqsMessage.MessageAttributes != nil {
        if tp, ok := sqsMessage.MessageAttributes["traceparent"]; ok {
            carrier["traceparent"] = *tp.StringValue
        }
        if ts, ok := sqsMessage.MessageAttributes["tracestate"]; ok {
            carrier["tracestate"] = *ts.StringValue
        }
    }

    extractedCtx := propagator.Extract(ctx, carrier)
    producerSpanContext := trace.SpanContextFromContext(extractedCtx)

    var links []trace.Link
    if producerSpanContext.IsValid() {
        links = []trace.Link{
            {SpanContext: producerSpanContext},
        }
    }

    ctx, span := tracer.Start(
        ctx,
        "sqs_process",
        trace.WithLinks(links...),
    )
    defer span.End()

    err := handleMessage(*sqsMessage.Body)

    span.SetAttributes(
        attribute.String("messaging.system", "AmazonSQS"),
    )

    return err
}

using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Context.Propagation;

public class MessageProcessor
{
    private static readonly ActivitySource ActivitySource = new("MessageProcessor");
    private static readonly TextMapPropagator Propagator = Propagators.DefaultTextMapPropagator;

    // Producer: Publishing a message with trace context
    public async Task PublishMessage(IQueue queue, string messageBody)
    {
        using var activity = ActivitySource.StartActivity("publish_message", ActivityKind.Producer);

        // Inject trace context into message headers
        var carrier = new Dictionary<string, string>();
        Propagator.Inject(
            new PropagationContext(activity.Context, Baggage.Current),
            carrier,
            (c, key, value) => c[key] = value
        );

        var message = new Message
        {
            Body = messageBody,
            Headers = carrier
        };

        await queue.PublishAsync(message);

        activity?.SetTag("messaging.destination", queue.Name);
        activity?.SetTag("messaging.system", "custom_queue");
    }

    // Consumer: Processing a message with span link
    public async Task ProcessMessage(Message message)
    {
        // Extract trace context from message headers
        var propagationContext = Propagator.Extract(
            default,
            message.Headers ?? new Dictionary<string, string>(),
            (carrier, key) => carrier.TryGetValue(key, out var value) ? new[] { value } : Array.Empty<string>()
        );

        var producerContext = propagationContext.ActivityContext;

        // Create span with link
        var links = new List<ActivityLink>();
        if (producerContext != default)
        {
            links.Add(new ActivityLink(producerContext));
        }

        using var activity = ActivitySource.StartActivity(
            "process_message",
            ActivityKind.Consumer,
            parentContext: default,
            links: links
        );

        await HandleMessage(message.Body);

        activity?.SetTag("messaging.system", "custom_queue");
        activity?.SetTag("messaging.operation", "process");
    }

    // AWS SQS example
    public async Task ProcessSQSMessage(Amazon.SQS.Model.Message sqsMessage)
    {
        // Extract trace context from SQS message attributes
        var carrier = new Dictionary<string, string>();

        if (sqsMessage.MessageAttributes != null)
        {
            if (sqsMessage.MessageAttributes.TryGetValue("traceparent", out var tp))
            {
                carrier["traceparent"] = tp.StringValue;
            }
            if (sqsMessage.MessageAttributes.TryGetValue("tracestate", out var ts))
            {
                carrier["tracestate"] = ts.StringValue;
            }
        }

        var propagationContext = Propagator.Extract(
            default,
            carrier,
            (c, key) => c.TryGetValue(key, out var value) ? new[] { value } : Array.Empty<string>()
        );

        var links = new List<ActivityLink>();
        if (propagationContext.ActivityContext != default)
        {
            links.Add(new ActivityLink(propagationContext.ActivityContext));
        }

        using var activity = ActivitySource.StartActivity(
            "sqs_process",
            ActivityKind.Consumer,
            parentContext: default,
            links: links
        );

        await HandleMessage(sqsMessage.Body);

        activity?.SetTag("messaging.system", "AmazonSQS");
    }
}

Best practices for span links

When implementing span links, follow these best practices:

Always propagate trace context: Ensure W3C trace context (traceparent and tracestate headers) are included in message headers or metadata.
Validate span context: Always check if the extracted span context is valid before creating a span link. Invalid contexts should not create links.
Use appropriate span kinds: Set PRODUCER kind for message publishing spans and CONSUMER kind for message processing spans.
Add messaging attributes: Include semantic conventions for messaging systems (like messaging.system, messaging.destination, messaging.operation) to provide context.
Consider sampling: Both linked traces must be sampled to appear in New Relic. Adjust sampling strategies for critical workflows that use span links.
Handle batch processing carefully: When processing batched messages, create individual span links for each message to maintain traceability.

Viewing span links in New Relic

Once you've implemented span links in your instrumentation, you can view and navigate them in the New Relic UI:

Navigate to the trace details page for a trace
Look for the span links badge in the filter bar showing the number of spans with links
Select a span with links to see the Span links tab in the span details pane
Click on linked traces to navigate between related traces

For detailed information on using span links in the UI, see Understanding span links.

OTLP span mapping

Table footnotes

Span links

When to use span links

Implementing span links

Python

Java

Node.js

Go

.NET

Best practices for span links

Viewing span links in New Relic

OpenTelemetry traces in New Relic

OTLP span mapping .css-21sua1{background:none;border:none;width:0;padding:0;}

Table footnotes

Span links

When to use span links

Implementing span links

Java

Node.js

Go

.NET

Best practices for span links

Viewing span links in New Relic

OTLP span mapping