feat: display token usage

[fengsh27]

Abstract

This submission is to display the token usage. If it reaches server daily token limit (ERROR_BIOSERVER_EXCEEDS_TOKEN_LIMIT), we will restrict further usage and encourage users to provide their OpenAI API key for access.

This PR also fixed #48

[slobentanzer]

Hi @fengsh27, many thanks for the PR!

The chat is functional for me, but I don't see the actual tokens in the sidebar (they stay at 0). I built the biochatter-server PR locally as biochatter-server:0.5.5 and am using it in the docker compose of this PR. Do we always want to display token usage, also in the case of a user-supplied key?

Where is the sqlite database built in those use cases? Is it the server or the next that manages the DB?

Minor detail: I think it makes no sense to display the OncoKB radio button in the generic version of the Next app; this should be hidden if not specifically requested / implemented. If we show something there, it should be a generic "tool calling" option, but that we would need to implement to be able to select any of the supported tools in the API agent of biochatter, which seems like a bit of work. I would just hide the OncoKB button for now in the generic biochatter-next.

[fengsh27]

Hi @fengsh27, many thanks for the PR!

The chat is functional for me, but I don't see the actual tokens in the sidebar (they stay at 0). I built the biochatter-server PR locally as biochatter-server:0.5.5 and am using it in the docker compose of this PR. Do we always want to display token usage, also in the case of a user-supplied key?

Where is the sqlite database built in those use cases? Is it the server or the next that manages the DB?

Minor detail: I think it makes no sense to display the OncoKB radio button in the generic version of the Next app; this should be hidden if not specifically requested / implemented. If we show something there, it should be a generic "tool calling" option, but that we would need to implement to be able to select any of the supported tools in the API agent of biochatter, which seems like a bit of work. I would just hide the OncoKB button for now in the generic biochatter-next.

Yes, it is supposed to display token usage for user-provided key, let me check what is wrong.

The sqlite database is managed by biochatter-server.

For OncoKB in generic app, it makes sense to disable it, I'll submit a fix for it, it won't take much efforts.

[fengsh27]

@slobentanzer , I've submitted a fix in biochatter-server, which fixed an error in get_embedding_function. The error was introduced in upgrading langchain_community.openai to langchain_openai, in which the server ENV OPENAI_API_KEY=azure will affect OpenAIEmbeddings.
Could you check if this fix works on your machine?
Additionally, please make sure to set ENV DATA_DIR in .bioserver.env that specifies where to save sqlite database

[slobentanzer]

If the location is necessary for the functionality, wouldn't it make sense to set a reasonable default that works also in the context of running the server in docker compose, and only expect the user to set the variable if they want to override the default?

[fengsh27]

Yes, the DATA_DIR is not necessary. Currently, it worked well without setting it as it already has default value "./data", see the following code in token_usage_database.py:

db_path = os.environ.get(DATA_DIR, "./data")

[fengsh27]

@slobentanzer, Could you please check if the fix in PR in biochatter-server resolves the issue that client-provided key does not work?

[slobentanzer]

@fengsh27 apologies for the delay, was working on another PR. I have now tested with the new server fix, it shows the tokens correctly.

[slobentanzer]

Server is merged and bumped to 0.5.5; I pushed the image to dockerhub.

[fengsh27]

Thanks.

It seems I forgot to disable OncoKB for generic app. I'll fix it with a separate PR.

[fengsh27]

@slobentanzer I've made a submission to disable OncoKB feature for generic app

[slobentanzer]

@fengsh27 thank you, looks good! Good to merge from my end.