Cherry-pick #7664 into v1.66.x

[purnesh42H]

Original PR: #7664

RELEASE NOTES:

• xds/server: Fix xDS Server memory leak

[codecov]

Codecov Report

Attention: Patch coverage is 93.75000% with 1 line in your changes missing coverage. Please review.

Project coverage is 81.85%. Comparing base (bcbf356) to head (dc0c185).
Report is 1 commits behind head on v1.66.x.

Files with missing lines	Patch %	Lines
xds/internal/server/listener_wrapper.go	93.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           v1.66.x    #7684      +/-   ##
===========================================
- Coverage    81.95%   81.85%   -0.10%     
===========================================
  Files          361      361              
  Lines        27858    27862       +4     
===========================================
- Hits         22830    22807      -23     
- Misses        3841     3855      +14     
- Partials      1187     1200      +13     

Files with missing lines	Coverage Δ
xds/internal/server/conn_wrapper.go	77.04% <100.00%> (-4.62%)	⬇️
xds/internal/server/listener_wrapper.go	79.01% <93.33%> (+0.39%)	⬆️

... and 21 files with indirect coverage changes

[arjan-bal]

Though it's not documented, the title for cherry-pick PRs is usually of the form: "Cherry-pick #7664 into v1.66.x"