Lamda handler Context does not contain Segment

[underscorenygren]

Bug

When an APIGateway enabled lambda function with tracing information passes it's context to xray.TraceID() it does not return the trace id from the header. This leads to downstream issues for tracing since GetSegment is always nil.

Expected Result

xray.TraceID should return the same trace ID that's present in the header.

Reproduce

tried on v1.0.0rc-12

The following handler illustrates the problem, the output should return the same trace ids.

package xray

import (
	"context"
	"encoding/json"
	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-xray-sdk-go/xray"
)

// Handler creates a new Peer and/or adds a new connectionId to it
func Handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {

	headerTraceID := req.Headers["X-Amzn-Trace-Id"]
	xrayTraceID := xray.TraceID(ctx)

	out := map[string]string{
		"header":        headerTraceID,
		"lambdaContext": xrayTraceID,
	}

	code := 200
	var body string
	if _body, err := json.Marshal(out); err == nil {
		body = string(_body)
	} else {
		body = err.Error()
		code = 400
	}

	return events.APIGatewayProxyResponse{
		Body:       body,
		StatusCode: code,
	}, nil
}

[awssandra]

Hi underscorenygren,

The issue here is, there is no segment to get.

In Lambda, the segment is created (and emitted) on behalf of the Lambda runtime start-up code.
The SDK hooks into this (via the header data on the context) and streams subsegments independently to have the X-Ray front-end for re-assembly. The SDK doesn't actually have the segment or it's data. Merely, the trace ID, segment ID and sampling decision.

The SDK uses the header as derived from the Lambda context to create a facade segment (and an underlying basic segment):

aws-xray-sdk-go/xray/segment.go

Lines 45 to 52 in 3565e35

	func BeginFacadeSegment(ctx context.Context, name string, h *header.Header) (context.Context, *Segment) {
	seg := basicSegment(name, h)
	cfg := GetRecorder(ctx)
	seg.assignConfiguration(cfg)
	return context.WithValue(ctx, ContextKey, seg), seg
	}

aws-xray-sdk-go/xray/segment.go

Line 102 in 3565e35

seg.TraceID = h.TraceID

This facade segment acts as a placeholder, such that the SDK can stream subsegments without additional special code to handle this Lambda case. But, this does not exist on the ctx object until a subsegment is created. The facade is created as needed when a new subsegment is created:

aws-xray-sdk-go/xray/segment.go

Lines 153 to 161 in 73cfeb4

	func BeginSubsegment(ctx context.Context, name string) (context.Context, *Segment) {
	if len(name) > 200 {
	name = name[:200]
	}
	parent := &Segment{}
	// first time to create facade segment
	if getTraceHeaderFromContext(ctx) != nil && GetSegment(ctx) == nil {
	_, parent = newFacadeSegment(ctx)

Thus, any GetSegment will always return nil when outside of a created subsegment.
Simply create a subsegment, and the SDK will do the hook-ups automatically.
There is a small write-up at the top here explaining the behavior: https://docs.aws.amazon.com/lambda/latest/dg/go-tracing.html#go-tracing-configuring-sdk

In the Node SDK we emit a warning saying this is unsupported:
https://github.com/aws/aws-xray-sdk-node/blob/master/packages/core/lib/env/aws_lambda.js#L56

But, you should be able to see the facade segment after the construction of the subsegment:
ctx, subSeg := xray.BeginSubsegment(ctx, "subsegment-name”)
subSeg.Close(nil)
xrayTraceID := xray.TraceID(ctx)

Hopefully this helps, let us know your thoughts. Is there a better way we can address this situation?
Thanks!
Sandra

[underscorenygren]

Hi Sandra,

I found the BeginFacadeSegment by trawling through the source, but the reference you refer to is a bit obscure, and the lambda example in the readme does not cover this case. Reading it with limited knowledge of xray it seemed to imply the segments are already in there.

Another example below lambda showcasing the use of straight xray without the Client would be super helpful. In this case reading up on instrumenting sql and lamda separately did not add up to the facaded segment. Happy to provide a readme PR with a pruned down version of this example code.

[awssandra]

Yes, admittedly, it's not super clear you do not need to worry about the segment on the examples, or the readme for that matter. We have had the question a few times on the Node issues. We'd be happy to review anything you'd like to submit.

Thanks!
Sandra

[luluzhao]

@underscorenygren Updated our Go SDK README to better explain the Lambda use case. Feel free to open issues if you have questions in future.

[amir-faliam]

The comment here seems to suggest we do the following:

ctx, subSeg := xray.BeginSubsegment(ctx, "subsegment-name”)
defer subSeg.Close(nil)
xrayTraceID := xray.TraceID(ctx)

However, this implementation inside of my lambda does not associate a traceID to the subsegment and all my logs are no longer associated with the trace this execution is tied to.

The alternative is we create a segment inside of our lambda and that way all of our logs get associated with the trace. But this causes Xray to treat the lambda execution as separate from the root trace because it creates a brand new trace.

What is the correct way to introduce tracing to my lambda and have all my logs be tied to it and be able to extract a traceID to put inside of my log messages? I can't find any documentation on this