"Special" character issues when using the S3 storage backend

[bschmalhofer]

When running the test suite with MinIO syncing. #1777, there are some failures that seem to be related to the characters used in the file name. An example is the test _scripts/test/Console/Command/Admin/Package/Upgrade.t _ . There we get the message:

Could not write the key Kernel/Config/Files/ZZZZUnitTestAdmin::Package::Upgradetest3947006996400002.pm to S3 at /opt/otobo/Kernel/System/UnitTest/Helper.pm line 583.
That something fishy is going on can be show in the MinIO Gui. Let's upload a dummy file called ZZZZUnitTestAdmin::Package::Upgradetest3947006996400002.pm and then retrieve it from MinIO. storing it in ../tmp:

bes:~/devel/OTOBO/otobo-docker (issue-#99-minio)$ cat ZZZZUnitTestAdmin::Package::Upgradetest3947006996400002.pm 
xxxx
bes:~/devel/OTOBO/otobo-docker (issue-#99-minio)$ cat ../tmp/ZZZZUnitTestAdmin\ Package\ Upgradetest3947006996400002.pm 
xxxx
bes:~/devel/OTOBO/otobo-docker (issue-#99-minio)$ 

The content xxxx is fine. But in the file name :: seems to be replaced ** ** (single space).

Needs to be investigated.

[bschmalhofer]

The AWS documentation regarding object keys is at https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html.

[bschmalhofer]

Not really understanding the AWS documentation. It's obvious that characters like ';' and '&' must be URL-encoded when used in URL. But why shouldn't they be used in object keys? As a first approach, let's be empiric. Wenn requesting PUT on 'http://minio:9000/otobo-bucket-20220524a/OTOBO/Kernel/Config/Files/ZZZZUnitTestAdmin::Package::Upgradetest1181006654200002.pm' then MinIO returns a 403. This needs to be worked around.

But first investigate how that works when there is no S3 syncing. Apparently the '::' do not end up in the file written to Kernel/Config/Files .

[bschmalhofer]

A lot of things are wrong in scripts/test/Console/Command/Admin/Package/Upgrade.t .

• the custom code was never executed
• the package upgrade is never done, see scripts/test/Console/Command/Admin/Package/Upgrade.t does not test package upgrade #1796

[bschmalhofer]

Found that there is not really a need to support fancy characters in the object keys. Adapted the test scripts/test/Storage/S3.t accordingly. But in that test script we should check whether we can store file names that were sanitized by FilenameCleanup. This is important for the article storage.

Here is the clean up from ArticleStorageFS.pm:

    # Perform FilenameCleanUp here already to check for
    #   conflicting existing attachment files correctly
    my $MainObject   = $Kernel::OM->Get('Kernel::System::Main');
    my $OrigFilename = $MainObject->FilenameCleanUp(
        Filename  => $Param{Filename},
        Type      => 'Local',
        NoReplace => 0,
    );

[bschmalhofer]

Two findings:

• ArticleStorageS3.pm should pass NoReplace => 0 to FilenameCleanUp() , just like ArticleStorageFS.pm
• In the S3 object key '+' is problematic. Replace it too with an underscore.
• same with the character '#'

[bschmalhofer]

The code has been adapted. Currently there are no known problems with unsupported characters. Closing this issue.