Release KV cache memory occupation when not needed

[fzyzcjy]

Motivation

This PR depends on #2586, thus there are much more code in the diff panel than its actual content.

Please see #2542

Modifications

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[merrymercy]

The design makes sense. The only downside is that it is not compatible with cuda graph. Does you have any ideas to make it compatible with cuda graph? Can we try to hack into the pytorch memory allocator and reuse buffer or fix the cuda pointers?

[fzyzcjy]

@merrymercy Thank you! I posted in #2542 (comment)

[fzyzcjy]

I will temporarily close this since #2542 (comment) seems more promising :)