-
Notifications
You must be signed in to change notification settings - Fork 693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(bedrock): support metrics for bedrock #1957
Conversation
83ca96c
to
df14a45
Compare
df14a45
to
4f2e54c
Compare
️✅ There are no secrets present in this pull request anymore.If these secrets were true positive and are still valid, we highly recommend you to revoke them. 🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request. |
be263f5
to
09ddf7f
Compare
09ddf7f
to
e17835a
Compare
Revert "Update" This reverts commit 0214db8. Update Update metrics tests and cassettes file
0214db8
to
682d2d2
Compare
I already know the root cause why I always keep getting the error |
040b0c8
to
6f5a11b
Compare
6f5a11b
to
6138f4a
Compare
@@ -29,6 +29,8 @@ class Meters: | |||
LLM_WATSONX_COMPLETIONS_RESPONSES = "llm.watsonx.completions.responses" | |||
LLM_WATSONX_COMPLETIONS_TOKENS = "llm.watsonx.completions.tokens" | |||
|
|||
LLM_BEDROCK_COMPLETIONS_EXCEPTIONS = "llm.bedrock.completions.exceptions" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this required?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's an open issue. I think it's better to use a common name, for example llm.completions.exceptions
for all ai system to count exceptions. If yes, I would like to use another PR to fix all related code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jinsongo if we're doing this here I'd try to use the new genAI conventions - https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-metrics
I think they're already released so you can even use them directly from the original otel semconv package
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(in this case - it means exceptions are counted in the duration metric)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nirga I cannot find an appropriate metric name for the exceptions counter of Bedrock. As you know, version 0.4.1 of opentelemetry-semantic-conventions-ai does not include LLM_BEDROCK_COMPLETIONS_EXCEPTIONS
although it's already included in this current PR commits, which causes the build to fail when I use it in the Bedrock instrumentation code. Actually, that's the same approach for OpenAI, Authropic and Watsonx.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jinsongo this should be in the standard opentelemetry-semantic-conventions
package. And you should count exceptions on the duration metric as specified in the spec.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jinsongo If this parameter is not required, I think we can ignore this for now, and we can fix this in future as a consolidation for semantic convention.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gyliu513 I have removed LLM_BEDROCK_COMPLETIONS_EXCEPTIONS
from semconv_ai
, and added a TODO
comment about fixing in future as a consolidation for semantic convention.
...ages/opentelemetry-instrumentation-bedrock/opentelemetry/instrumentation/bedrock/__init__.py
Outdated
Show resolved
Hide resolved
...ages/opentelemetry-instrumentation-bedrock/opentelemetry/instrumentation/bedrock/__init__.py
Outdated
Show resolved
Hide resolved
...ages/opentelemetry-instrumentation-bedrock/opentelemetry/instrumentation/bedrock/__init__.py
Show resolved
Hide resolved
5b7d747
to
e2605e5
Compare
19ed21a
to
9e1110e
Compare
e5b6acc
to
1580800
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice @jinsongo! a couple of styling comments and we're ready to go
...ages/opentelemetry-instrumentation-bedrock/opentelemetry/instrumentation/bedrock/__init__.py
Outdated
Show resolved
Hide resolved
...ages/opentelemetry-instrumentation-bedrock/opentelemetry/instrumentation/bedrock/__init__.py
Outdated
Show resolved
Hide resolved
/lgtm @nirga can you help review? Thanks |
@nirga Do you have more comments? Thanks. |
feat(instrumentation): ...
orfix(instrumentation): ...
.