Cached / multimodal token are not passed through to Langfuse

[hassiebp]

What happened?

Context

Langfuse has shipped V3 cost tracking that allows tracking usage details and cost details by arbitrary usage keys beyond justinput, output, and total. If OpenAI returns prompt_tokens_details.cached_tokens then Langfuse can infer costs when the cached_tokens are provided in usage_details.input_cached_tokens for the Generation. See docs here

Issue
LiteLLM is currently using the soon-to-be-deprecated usage key and not the new usage_details and cost_details map. See here.

Desired behavior
Langfuse accepts the OpenAI schema server side and via its SDK interface for usage_details and cost_details. Langfuse flattens the prompt_tokens_details and completion_tokens_details provided by OpenAI by prefixing the keys with input_ and output_ respectively. Please pass the OpenAI usage object as generation.usage_details and for cost_details respectively.

Here's the Pydantic schema enforced by Langfuse for the OpenAI Usage schema

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.60.4

Twitter / LinkedIn details

https://www.linkedin.com/in/hassieb/