Deadlock fix in acquireShards

[Groxx]

Fixing the deadlock demonstrated in #5824.

I decided to move the channel-writing entirely before consuming so it's a bit more accidental-change-resistant: some kinds of simple incorrect changes will lead to an immediate deadlock every time, rather than a random chance of one.

And if someone does want to move it after and go back to a smaller buffer, more code will have to be changed, so hopefully people will pay more attention to the concurrency risks involved.

More generally, the atomic shutdown stuff is highly prone to causing this kind of error because there's no way to wait on it safely, and I would really love for us to get rid of it.

[codecov]

Codecov Report

Merging #5825 (148dd5a) into master (831ebf5) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Additional details and impacted files

Files	Coverage Δ
service/history/shard/controller.go	68.91% <100.00%> (+0.63%)	⬆️

... and 7 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 831ebf5...148dd5a. Read the comment docs.

[coveralls]

Pull Request Test Coverage Report for Build 018e8c1a-a3b1-4b88-bddb-02d2f50a8e85

Details

• 8 of 8 (100.0%) changed or added relevant lines in 1 file are covered.
• 38 unchanged lines in 8 files lost coverage.
• Overall coverage increased (+0.04%) to 65.296%

---

Files with Coverage Reduction	New Missed Lines	%
common/task/weighted_round_robin_task_scheduler.go	1	89.05%
common/task/fifo_task_scheduler.go	2	83.51%
common/persistence/sql/sqlplugin/mysql/task.go	2	73.68%
common/persistence/sql/sqlplugin/mysql/db.go	2	79.49%
common/persistence/historyManager.go	2	66.67%
service/history/task/fetcher.go	3	86.6%
service/history/task/transfer_standby_task_executor.go	6	85.77%
service/history/task/task_util.go	20	70.57%

Totals
Change from base Build 018e86dc-121c-4962-9b30-23df2fb36173:	0.04%
Covered Lines:	95483
Relevant Lines:	146231

---

💛 - Coveralls