[KYUUBI #5952] Disconnect connections without running operations after engine maxlife time graceful period

[turboFei]

🔍 Description

Issue References 🔗

We found that, some kyuubi connections(maybe managed by jdbc connection pool likes hikari) always keep alive, and the engine can not be terminated after exceeds the max life time.

So, In this pr, I introduce a graceful period after spark engine max life time, after the graceful period, the connections without running operations will be disconnected forcibly.

Close #5952

Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Behavior Without This Pull Request ⚰️

Behavior With This Pull Request 🎉

Related Unit Tests

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (87ed400) 61.20% compared to head (bd18a38) 61.21%.
Report is 1 commits behind head on master.

❗ Current head bd18a38 differs from pull request most recent head 0352dd1. Consider uploading reports for the commit 0352dd1 to get more accurate results

Files	Patch %	Lines
...rg/apache/kyuubi/engine/spark/SparkSQLEngine.scala	69.23%	1 Missing and 3 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #5950      +/-   ##
============================================
+ Coverage     61.20%   61.21%   +0.01%     
  Complexity       23       23              
============================================
  Files           622      622              
  Lines         36897    36899       +2     
  Branches       5016     5017       +1     
============================================
+ Hits          22582    22589       +7     
+ Misses        11875    11868       -7     
- Partials       2440     2442       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cxzl25]

#2250 has been released for a while, I recommend using a new issue id to track the issue.

[turboFei]

Not sure why EtcdConnectionLevelSparkEngineSuite always fail, cc @hddong is there any special for ETCD?

[wForget]

We found that, some kyuubi connections(maybe managed by jdbc connection pool likes hikari) always keep alive, and the engine can not be terminated after exceeds the max life time.

Shall we reject new operations after the engine has stopped gracefully? That way the connection will be closed after session idle timeout.

[turboFei]

Shall we reject new operations after the engine has stopped gracefully? That way the connection will be closed after session idle timeout.

It is difficult to implement in code.

Some rpc does not create operation but will update the session last access time with withAcquireRelease.

[pan3793]

@turboFei the jetcd upgrade involves gRPC upgrading, I have done it on master, please revert your latest commit and rebase master to see if helps

[turboFei]

raised #6020