fix(cluster update): separate cache update context from API call

[karol-kokoszka]

Fixes #4249

This PR "unplugs" background config-cache update from the API call context.

Before change (context cancelled):

➜  scylla-manager git:(kk/4249-update-config-cache-in-separation-from-api-call) ✗ ./sctool.dev cluster update --cluster 6bb4f1d2-8ab7-4ed8-894b-767d52b1664e --username cassandra --password cassandra

04:49:06.975    INFO    http    PUT /api/v1/cluster/6bb4f1d2-8ab7-4ed8-894b-767d52b1664e        {"from": "192.168.100.1:41228", "status": 200, "bytes": 223, "duration": "61ms", "_trace_id": "DzcJULtaRPu77hSBdQM8zw"}
04:49:06.975    INFO    cluster Creating new Scylla HTTP client {"cluster_id": "6bb4f1d2-8ab7-4ed8-894b-767d52b1664e", "_trace_id": "DzcJULtaRPu77hSBdQM8zw"}
04:49:06.976    ERROR   cluster Couldn't discover hosts using stored coordinator host, proceeding with other known ones {"coordinator-host": "192.168.200.11", "error": "context canceled", "_trace_id": "DzcJULtaRPu77hSBdQM8zw"}
github.com/scylladb/go-log.Logger.log
        github.com/scylladb/[email protected]/logger.go:101
github.com/scylladb/go-log.Logger.Error
        github.com/scylladb/[email protected]/logger.go:84
github.com/scylladb/scylla-manager/v3/pkg/service/cluster.(*Service).discoverClusterHosts
        github.com/scylladb/scylla-manager/v3/pkg/service/cluster/service.go:186
github.com/scylladb/scylla-manager/v3/pkg/service/cluster.(*Service).discoverAndSetClusterHosts
        github.com/scylladb/scylla-manager/v3/pkg/service/cluster/service.go:163
github.com/scylladb/scylla-manager/v3/pkg/service/cluster.(*Service).CreateClientNoCache
        github.com/scylladb/scylla-manager/v3/pkg/service/cluster/service.go:144
github.com/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).updateSingle
        github.com/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:178
github.com/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).ForceUpdateCluster
        github.com/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:147
04:49:06.977    ERROR   cluster There is no single valid known host for the cluster. Please update it with 'sctool cluster update -h <host>'    {"cluster": "6bb4f1d2-8ab7-4ed8-894b-767d52b1664e", "contact point": "192.168.200.11", "discovered hosts": ["192.168.200.11", "192.168.200.12", "192.168.200.13", "192.168.200.21", "192.168.200.22", "192.168.200.23"], "_trace_id": "DzcJULtaRPu77hSBdQM8zw"}
github.com/scylladb/go-log.Logger.log
        github.com/scylladb/[email protected]/logger.go:101
github.com/scylladb/go-log.Logger.Error
        github.com/scylladb/[email protected]/logger.go:84
github.com/scylladb/scylla-manager/v3/pkg/service/cluster.(*Service).discoverAndSetClusterHosts
        github.com/scylladb/scylla-manager/v3/pkg/service/cluster/service.go:166
github.com/scylladb/scylla-manager/v3/pkg/service/cluster.(*Service).CreateClientNoCache
        github.com/scylladb/scylla-manager/v3/pkg/service/cluster/service.go:144
github.com/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).updateSingle
        github.com/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:178
github.com/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).ForceUpdateCluster
        github.com/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:147
04:49:06.977    ERROR   Cluster config update   Couldn't create scylla client   {"cluster": "6bb4f1d2-8ab7-4ed8-894b-767d52b1664e", "cluster": "6bb4f1d2-8ab7-4ed8-894b-767d52b1664e", "error": "discover and set cluster hosts: unable to connect to any of cluster's known hosts", "_trace_id": "DzcJULtaRPu77hSBdQM8zw", "errorStack": "github.com/scylladb/scylla-manager/v3/pkg/service/cluster.(*Service).CreateClientNoCache\n\tgithub.jparrowsec.cn/scylladb/scylla-manager/v3/pkg/service/cluster/service.go:145\ngithub.jparrowsec.cn/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).updateSingle\n\tgithub.jparrowsec.cn/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:178\ngithub.jparrowsec.cn/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).ForceUpdateCluster\n\tgithub.jparrowsec.cn/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:147\nruntime.goexit\n\truntime/asm_amd64.s:1700\n"}
github.com/scylladb/go-log.Logger.log
        github.com/scylladb/[email protected]/logger.go:101
github.com/scylladb/go-log.Logger.Error
        github.com/scylladb/[email protected]/logger.go:84
github.com/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).updateSingle
        github.com/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:180
github.com/scylladb/scylla-manager/v3/pkg/service/configcache.(*Service).ForceUpdateCluster
        github.com/scylladb/scylla-manager/v3/pkg/service/configcache/service.go:147

After change (no error, cache is updated):

04:51:41.638    INFO    http    PUT /api/v1/cluster/6bb4f1d2-8ab7-4ed8-894b-767d52b1664e        {"from": "192.168.100.1:44138", "status": 200, "bytes": 223, "duration": "62ms", "_trace_id": "gndfNd7jQT-Mh5KEUQxkqA"}
04:51:41.638    INFO    cluster Creating new Scylla HTTP client {"cluster_id": "6bb4f1d2-8ab7-4ed8-894b-767d52b1664e"}
04:51:41.647    INFO    cluster.client  Datacenters by latency (dec)    {"dcs": ["dc2", "dc1"]}

---

Please make sure that:

• Code is split to commits that address a single change
• Commit messages are informative
• Commit titles have module prefix
• Commit titles have issue nr. suffix