Amazon X-Ray interop

[codefromthecrypt]

Similar to how last year, we had users requesting for Google Stackdriver compatibility, there are users explicitly requesting Amazon X-Ray, as well general support and interest from @abhiksingh (X-Ray product lead), and indirect comments which I can't currently find about tension between lambda and zipkin architecture*.

There's no doubt that Zipkin and Amazon interop have been important in the past. Many of our core team rely on AWS and/or make custom components for AWS. This issue will explore how we could fit-in, and how we could allow 3rd party tracers designed for B3 to have the smallest impact to support X-Ray.

Similar to StackDriver, there's two major concerns: propagation and out-of-band data.

Unlike StackDriver, propagation is wider than the X-Ray service. For example, in AWS the same propagation format is used even when X-Ray isn't: ELB uses the same format eventhough ELB doesn't write to X-Ray. There are also interesting concerns such as that Api Gateway restarts traces at its edge, also in X-Ray format. These types of concerns weren't present when we integrated with StackDriver, although they are building in the new trace-context format, targeted initially inside Google to StackDriver and gRPC services.

Also unlike StackDriver, the trace ID requires a 32-bit timestamp. This has some impact, as if there's an invalid timestamp, the service will drop any related data. For this reason, pragmatic ID generation strategy is something we must consider. For example, creating "interop" IDs by default where the first 32-bits are a timestamp and the following are 96-bits of random. This discussion occurred at the end of #1262.

Reporting is very much like what we did in StackDriver. The X-Ray format is span structured, with an api to post data to. While it has more structure than Zipkin data, mapping rules like exist in StackDriver are more than possible. Zipkin-compatible (or otherwise) tracers could send data to X-Ray's daemon (automatically present in lambda), the X-Ray POST api, or to a zipkin destination that does one of the two.

The above is fairly reliable from initial explorations, but could change with experience. This issue will track the exploration and any related issues on Zipkin's side.

• If you are running a lambda (serverless) architecture, running a zipkin server, even a stupid proxy to X-Ray one, could feel heavyweight. In this case, writing directly to X-Ray, or translating zipkin to AWS via a kinesis lambda could make sense.

[codefromthecrypt]

cc @llinder @devinsba @openzipkin/core

[yurishkuro]

So these are the interop models for the out-of-band data?

• [zipkin SDK] -> [X-Ray backend] directly
• [zipkin SDK] -> [zipkin collector] -> [X-Ray backend]

[codefromthecrypt]

So these are the interop models for the out-of-band data? - [zipkin SDK] -> [X-Ray backend] directly - [zipkin SDK] -> [zipkin collector] -> [X-Ray backend] TL;DR; correct :)

nit: the component relevant to out-of-band in zipkin lingo is "reporter". We don't define the term SDK. http://zipkin.io/pages/architecture.html So, something already recording data in a zipkin format sends directly to X-Ray or via a "collector". A collector can be any component on a transport (streaming or otherwise, zipkin-server or otherwise) that takes standard zipkin encoding and pushes it to X-Ray. I'm intentionally being abstract as people have different ways they accept data. The first (direct) technique allows folks for example who are running lambdas and want to pay by the trace a way to proceed without any infrastructure. They just align permissions and go. For example, one can write a lambda and run it inside or outside AWS (reporting to zipkin when outside)

[codefromthecrypt]

random note: http://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html The POST format is a list of escaped json, as the doc implies.

Ex.

{"TraceSegmentDocuments": [
"{\"id\": \"0b89f1dec76af795\", ..."
]}

[codefromthecrypt]

Added an example implementation of trace ID with a time component. Unsurprisingly, it is slower than a fully random ID. However, the scale is still sub microsecond (on my laptop™), and only affects the root span: openzipkin/brave#509

Next step is to add a converter which proves the concept.

[codefromthecrypt]

experimental work starting in Brave here openzipkin/brave#510

[codefromthecrypt]

Thanks to @jcarres-mdsol for making new trace ID provisioning instructions a bit simpler:

|---- 32 bits for epoc seconds --- | ----- 96 bits for random number --- |
it can potentially be implemented by:
High 64:
|---- 32 bits for epoc seconds --- | ----- 32 bits for random number --- |
Low 64:
| ----- 64 bits for random number --- |

Optional cheap sanity check the high 32 bits are epoch seconds
58000000 = 1476395008 = 2016-10-13 < prior to X-Ray and zipkin supporting 128-bit trace IDs
60000000 = 1610612736 = 2021-01-14 < of course you can even more future proof

[devinsba]

Closing as I believe we have addressed this from zipkin-aws

[msmsimondean]

@devinsba the issue is about providing interoperability with AWS X-Ray. Looking at zipkin-aws, that doesn't seem to provide that interopability; from what I can tell, it interops with other AWS services (SQS, Kinesis and Elasticsearch Service) but not AWS X-Ray. I can see some code in zipkin-aws that mentions X-Ray (e.g. reporter-xray-udp and storage-xray-udp). Does zipkin-aws provide some X-Ray integration that isn't documented in the codebase's README? Thanks in advance!

[devinsba]

We have support for the XRay propagation format and sending traces/spans to XRay, is there some kind of integration that you are looking for that isn't either of these? Also any feature requests for more integration should be handled in the zipkin-aws repo

[msmsimondean]

@devinsba that's great. Is there any documentation available for setting it up or is that still to come? I'm just thinking of the good documentation at https://cloud.google.com/trace/docs/zipkin and https://github.com/openzipkin/zipkin-gcp for the equivalent Google Stackdriver Trace integration. Thanks

[codefromthecrypt]

what's in the repo are in the README files at the moment. Our penalty for lack of docs is answering 1-1 :) Mind doing q/a on gitter as we don't use issues for Q/A https://gitter.im/openzipkin/zipkin

…

On Fri, May 24, 2019 at 3:22 PM msmsimondean ***@***.***> wrote: @devinsba <https://github.com/devinsba> that's great. Is there any documentation available for setting it up or is that still to come? Thanks — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1754>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAPVV6E5S5HPRXMJXTXDG3PW7TY3ANCNFSM4D5IRF4A> .