help request: How to use x-request-id as trace_id with apisix opentelemetry over istio service mesh

[circlelychen]

Description

Hi guys,

We enable apisix opentelemetry plugin with trace_id_source = x-request-id over istio service mesh. I got following error in apisix log

bad argument #1 to '?' (invalid value)
stack traceback:
coroutine 0:
        [C]: in function 'gsub'
        ...deps/share/lua/5.1/opentelemetry/trace/exporter/encoder.lua:21: in function 'hex2bytes'

The root cause is x-request-id header must be match with regex [0-9a-f]{32} since trace_Id only support that pattern. However, x-request-id is generated by istio service mesh with "-"(hyphen) between hex-style character, e.g. 84961386-6d84-929d-98bd-c5aee93b5c88. These "-"(hyphen) contribute the error hex2bytes function in opentelemetry-lua.

In our use cases, removing istio service mesh is forbidden and we regard this "-"hyphen embedded x-request-id as global unique trace_id cross multiple Kubernetes for single request life cycle.

We leverage apisix as micro-service API gateway. Is there any best practice or guidelines for us to integrate apisix opentelemetry over istio service mesh?

Please advises.
Thanks

Environment

• APISIX version (run apisix version): 3.4
• Operating system (run uname -a): redhat
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): none. standalone mode
• APISIX Dashboard version, if relevant: none
• Plugin runner version, for issues related to plugin runners: none
• LuaRocks version, for installation issues (run luarocks --version):

[shreemaan-abhishek]

I think we should add hyphen to the regex. WDYT @Revolyssup

[Revolyssup]

@circlelychen The opentelemetry specification says that this should be hex string and not all uuids are valid hex. So apisix uses the opentelemetry library which just follows the spec. You can use plugin to remove the - from the header but that might not be good since you are using hyphenated id inside the system.

So one way can be to create a copy of the x-request-id header with removed hyphens using the plugin and use the non-hyphen header for opentelemetry plugin leaving the original header intact. Long story short, it's not possible to use hyphens with opentelemetry plugin

[circlelychen]

@Revolyssup Thanks for your advice. We should make a bridge between opentelemetry and istio service mesh.

[github-actions]

Due to lack of the reporter's response this issue has been labeled with "no response". It will be close in 3 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.

[shreemaan-abhishek]

I investigated this and realised that the underlying otel lua library expectes trace_id to be hexadecimals. The trace-id needs to be decoded to bytes in order to be sent via transit over OLTP. The hexadecimal to bytes decoding process fails due to the way how the underlying library is written:

https://github.com/yangxikun/opentelemetry-lua/blob/6e44fe8352a032f8a24fffef3c0a43a05ee49e33/lib/opentelemetry/trace/exporter/encoder.lua#L14C1-L28C4

[shreemaan-abhishek]

according to the specification, the trace_id should be a valid hex and hyphens cannot be a part of it. So I will close this issue.