Calls to the get_online_features API are slower in v0.42.0 than in v0.40.1

[jparkzz]

Expected Behavior

The call to the get_online_features API must be faster or equal to v0.42.0 than v0.40.1.

Current Behavior

v0.42.0 is slower than v0.40.1

Steps to reproduce

Specifications

• Version: v0.42.0, v0.41.*
• Platform: Kubernetes + Redis (Online) + Bigquery (Offline) + PostgreSQL(SQLRegistry)
• Subsystem:

Possible Solution

Need to change the default behavior to sync (based fastapi - anyio threadpool), or change the part where blocking (or delay) occurs in async to async.

I was running 0.40.1, and upgraded to 0.41.2 / 0.42.0, but I actually found that it got slower.

Looking at the structure of async def get_online_features
(1) Get the information of the requested feature_view.
(2) Check the permission (when using RBAC).
(3) Get the value from the online store using asynchronous.
(4) Deserialize the protocol buffer message from the online store into a dict.

It flows in this order, but only (3) is actually protected by the threadpool or eventloop.
However, (1), (2), and (4) also take a lot of time for protobuf serialization-deserialization, and as a result, (1), (2), and (4) lose time when performed with a single thread (async) compared to multi-threading.

[jparkzz]

This is a graph of the latency changes (v0.42.0 -> v0.40.1) in my system.