[gcs] Enable grpc based broadcasting by default

[fishbone]

Why are these changes needed?

This is part of redis removal project. This PR is going to enable grpc based broadcasting by default.

Related issue number

#19438

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[fishbone]

This needs to be done before merging this PR:

• merge this PR [gcs] Fix the regression of enabling grpc based broadcasting in actor scheduling #19664
• run nightly test to make sure things are working well

[fishbone]

Test looks good https://buildkite.com/ray-project/periodic-ci/builds/1317

[fishbone]

previously I thought test failure it's related to some unit test, but it looks like system error. merge to master and re-test it.

[fishbone]

@rkooo567 we should keep an eye on the nightly tests after this one merged.

[rkooo567]

I think for this particular feature, the "daily" test might give more signals than nightly. (since daily tests have higher stress tests).

Can you also try running them?

[fishbone]

Sure, I'll run daily test before merging.

[fishbone]

Sadly some daily pg tests failed. Thanks, @rkooo567 for the catching.

[fishbone]

pg_autoscaling_regression_test
train_small: perf regression
multi_deployment_1k_noop_replica: timeout

RuntimeError: Process timed out while running a command.; Command logs:(ServeController pid=522) 2021-10-26 17:20:17,253	WARNING backend_state.py:1090 -- Deployment 'Echo_98' has 4 replicas that have taken more than 30s to start up. This may be caused by waiting for the cluster to auto-scale, waiting for a runtime environment to install, or a slow constructor. Resources required for each replica: {'CPU': 1}, resources available: {}. component=serve deployment=Echo_98

Wired thing here is that once retry they all passed

[fishbone]

@rkooo567 verified
train_small: perf regression
multi_deployment_1k_noop_replica: timeout

are flaky even without this.

I'll add pg-related tests to staging mode and @rkooo567 will get some complicated pg tests (Thanks!!!)

I'll come back to this one later once things are stable.

[fishbone]

#19779

[fishbone]

nightly tests look good. merging to trunk and merge after ci

[fishbone]

blocking by this pr #19996

[fishbone]

merge it since nightly looks good.