[Core] Instance Preemption Causing Application Level Broken Pipe Issue

[MengjinYan]

What happened + What you expected to happen

In a recent investigation, we found that when a task writes message to the local object store at the same time the local object store is being shut down, the Broken pipe error thrown from the write message operation will fail the current task as an application level error. This caused the task to fail without retry.

The expected behavior should be the Broken pipe error should be thrown as a system error so that the task can further retry as configured.

Versions / Dependencies

N/A

Reproduction script

N/A

Issue Severity

None