Normalize the gRPC error handling

[JmPotato]

As #5161 and tikv/tikv#12345 show, the gRPC error handling of PD is arbitrary. We mix using the gRPC error code and response header to return the error, which causes the TiKV may reconnect the PD incorrectly and unnecessarily. This is the tracking issue for normalizing the gRPC error handling of PD.

TODOs

• The error returned by PD gRPC service is not expected #5161
• Normalize the definition of the two types of error on the PD side.
  • gRPC Connection Error: an error that happens on the gRPC connection level, e.g, not the leader or unmatched cluster ID, which should cause the TiKV to re-establish the connection.
  • gRPC Interface Error: an error that happens inside the gRPC interface logic, e.g, region/store not found or other PD internal errors, which should cause the TiKV to retry rather than reconnect.
• Redefine the error proto https://github.com/pingcap/kvproto/blob/08b02befd813c6f5e6629fc24ab249da98e87bb3/proto/pdpb.proto#L142
• Normalize the gRPC error handling on PD.
• Normalize the gRPC error handling on TiKV.

[JmPotato]

cc @HuSharp @Connor1996

[Connor1996]

Maybe not bother to add new error types.
For grpc connection error, it should return grpc error. For interface error, it should return unknown error. These error are properly handled in tikv/tikv#12827

[JmPotato]

Maybe not bother to add new error types. For grpc connection error, it should return grpc error. For interface error, it should return unknown error. These error are properly handled in tikv/tikv#12827

Yes, the current task is to fix the PD's gRPC error handling according to this rule. Other normalizing work is more long-term or expected to be done sometime in the future.