Add Prefix Caching Benchmark to Serving Benchmark

[ywang96]

This PR:

• Added a way to benchmark automatic prefix caching using sonnet.txt dataset from llmperf-legacy, with additional arguments to control input, output and prefix lengths.
• Fixed TPOT/ITL(time per output token/inter token latency) calculation - now TTFT is excluded from end-to-end latency before latency is averaged over number of generated tokens - 1 (first token).
• Misc fixes due to changes in other backends.
• Error logging to address IMPORTANT Bug: Model return empty response (output len = 0), when recieved multiple concurrent request. #3209

Not sure if we should separate the prompt generation functions to a different file so it can be reused by benchmark_throughput.py and benchmark_prefix_caching.py as well. For now I still kept them in the same file.

[ywang96]

cc @robertgshaw2-neuralmagic as we discussed offline on benchmarking automatic prefix caching. PTAL and let me know if you have any question!

[ywang96]

@tattrongvu - I added some additional functionalities to save error, generated text and actual output len to the result json file so it's easier to debug. I've already tested it myself with num_prompts=200 and request-rate='inf' on a mixtral hosted on 2xA100-80G with vllm 0.3.3, so if you could give this PR a try, that would be great!