Add node and zonal antiaffinity to store gateway jsonnet

[Bruno-DaSilva]

What this PR does

Currently the Store Gateway jsonnet does not specify any antiAffinity for the zonal or regular running modes. This means that multiple storegateways can be on the same node in regular mode, or the same zone in zonal replication mode. This is obviously not ideal in the event of a failure.

To match the antiAffinity logic of the Ingester service, this PR adds a storegateway_allow_multiple_replicas_on_same_node option to jsonnet. It has the following behaviour:

STOREGATEWAY, storegateway_allow_multiple_replicas_on_same_node=false:
regular: nodeAntiAffinity=true, zoneAntiAffinity=false
zonal: nodeAntiAffinity=false, zoneAntiAffinity=true

STOREGATEWAY, storegateway_allow_multiple_replicas_on_same_node=true:
regular: nodeAntiAffinity=false, zoneAntiAffinity=false
zonal: nodeAntiAffinity=false, zoneAntiAffinity=false

Notice with flag=true, existing behaviour is the same as prior to this change (no antiAffinity). With flag=false, it matches the Ingester's antiAffinity behaviour.

Which issue(s) this PR fixes or relates to

Fixes #1729

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[CLAassistant]

All committers have signed the CLA.

[56quarters]

A few small changes, otherwise this looks good. Thank you!

[Bruno-DaSilva]

Do you guys prefer the changes squashed together or will you do that at merge?

[56quarters]

Do you guys prefer the changes squashed together or will you do that at merge?

That'll happen at merge, it's fine to add multiple commits.