Kestra adds an extra slash as a root folder in S3 bucket which makes it inaccessible by s3 GUI tools

[aku]

Issue description

Kestra puts its files into <s3 bucket>//<namespace> folder (notice double slashes). Most GUI tools can't see these files.

I've tried Cyberduck, DataGrip, cloudlena/s3manager - none of these tools was able to see Kestra's files
Have a look at this issue for example aws/aws-toolkit-visual-studio#248

Consider removing extra slash for root folders

[brian-mulier-p]

Hello ! That's a choice that was made at the beginning of Kestra and removing the leading slash will break everything so I'm not sure we can do much 🤔 @tchiotludo wdyt ?

[anna-geller]

Hi @aku - we discussed this in a team and changing that would require a breaking change we cannot do in a minor release upgrade. We want to offer stability so we will add the issue to the Planned changes for v1.0.0 -- hope you can understand!

[anna-geller]

Great news, we talked more, and it seems we will be able to do it sooner without a breaking change. We'll add a configuration:

kestra:
  storage:
    keep-root-slashes: true

Once you set this configuration to false, the leading slashes should be gone 🎉

Note to the engineer who will pick this up: if set to false, we will need to check for "/" at startup to check if there is something - if there is, we do the migration

[aku]

@anna-geller sounds good to me! it would be nice to have this feature without breaking changes. thanks!

[anna-geller]

Next steps:

• ensure CS team warns every customer to change that boolean if they want to keep the root slashes
• implement migration path (configuration parameter or running some migration CLI)

[anna-geller]

Next steps:

• don't move any files
• allow configuring this at instance, tenant and namespace level i.e. everywhere where you can configure internal storage backend
• without moving, we can do it in 0.21

[brian-mulier-p]

To do things in a clean way we need to have a singleton that will provide the paths to all kind of resources as a single entry point because we need a way to inject that configuration and ensure any storage path request goes through that component. Currently that's not the case and the fact that the paths are provided as static fields make this feature really hard to do without a proper refactoring (because currently we probably have to go at the implementation level and change every storage to perform that).

Totally WIP branch going on the implementation-side changes (but I really think that's a wrong direction and the handling should be performed before-hand):
https://github.com/kestra-io/kestra/tree/wip/leading-slash-opt-for-storages
https://github.com/kestra-io/storage-gcs/tree/wip/leading-slash-opt-for-storages