Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic gRPC client support #259

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Dynamic gRPC client support #259

wants to merge 23 commits into from

Conversation

nv-hwoo
Copy link
Contributor

@nv-hwoo nv-hwoo commented Jan 29, 2025

This is an initial effort to support streaming NIMs using gRPC protocol. The PR introduces a new gRPC client and client backend that enables PA to send requests to gRPC server dynamically without requiring PA to pre-compile the protobuf to generate codes for gRPC client stub.

The current version of the client only supports bidirectional streaming RPC and will support other RPC types in the future.

Brief Example

  1. User writes a short script (with language of their choice) that creates protobuf messages to send and writes it to a file stream:
# input_stream.py
import  sys

def generate_msgs():
    # generate protobuf messages
    yield ...
    yield ...
    ...

for msg in generate_msgs():
    serialized = msg.SerializeToString()
    # encode msg size
    sys.stdout.buffer.write(len(serialized).to_bytes(4, byteorder=sys.byteorder))
    # encode actual msg
    sys.stdout.buffer.write(serialized)
    sys.stdout.buffer.flush()
  1. Create JSON file to specify the command to run the script (e.g. inputs.json):
{
  "data": [
    {
      "ipc_stream": "python3 input_stream.py"
    }
  ]
}
  1. Run Perf Analyzer:
perf_analyzer -u localhost:8004 --service-kind dynamic_grpc --streaming --input-data inputs.json --proto <path/to/proto> --rpc <full grpc method name>

PA will execute the user provided command, read/store the messages generated by user script, and send them over to server (in the same order as the user sent them).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant