Encoding problem for ajax return in the "apache" variant of 10.1

[svenoe]

To see the effect visit /otobo/index.pl?Action=AgentStatistics;Subaction=Add and with a German profile click e.g. on "Dynamische Matrix".

This is currently fetched in line 63 of var/httpd/htdocs/js/Core.Agent.Statistics.js, the probably important side is Kernel/Modules/AgentStatistics.pm Subaction=GeneralSpecificationsWidgetAJAX

[bschmalhofer]

First surprise, the Umlaute look as expected to me. Tested with running Firefox 94.0 and Chromium Version 96.0.4664.45 under Linux. So this effect might be browser dependent. But let's see what possible could go wrong.

The page http://localhost/otobo/index.pl?Action=AgentStatistics;Subaction=Add looks initially fine. The content-type is not surprisingly ext/html; charset=utf-8. Clicking on Dynamische Matrix loads the HTML snippet http://localhost/otobo/index.pl?Action=AgentStatistics;Subaction=GeneralSpecificationsWidgetAJAX via the JavaScript code:

            Core.AJAX.FunctionCall(URL, Data, function(Response) {
                $('#GeneralSpecifications .Content').removeClass('Center').html(Response);
                Core.UI.InputFields.Activate($('#GeneralSpecifications .Content'));
                $('#SaveWidget').show();
            }, 'html');

The snippet has content type text/html, but the content is UTF-8 encoded, subsequently the output is messed up.

Lets doublecheck. Pasting "können" into vim and use 'g8' on the funny chars gives us the hex encodings "c373" and "c2b6" for the funny chars. Looking up the code points we confirm that the funny chars are the bytes of UTF-8 encoded 'ö' encoded as UTF-8.

bes:/devel/OTOBO/CodePolicy (master)$ uni -x c383
à - U+000C3 - LATIN CAPITAL LETTER A WITH TILDE
bes:/devel/OTOBO/CodePolicy (master)$ uni -x c2b6
¶ - U+000B6 - PILCROW SIGN
bes:~/devel/OTOBO/CodePolicy (master)$ uni -x c3b6
ö - U+000F6 - LATIN SMALL LETTER O WITH DIAERESIS

The guess is that we need to declare the content of the snippet as UTF-8.

[bschmalhofer]

Declaration as UTF-8 can done with the parameter Charset UTF-8 in the method Kernel::Output::HTML::Layout::Attachment().

Or for AJAX html snippets:

$HTML = $LayoutObject->Attachment(
    Type        => 'inline',        # optional, default: attachment, possible: inline|attachment
    Filename    => 'FileName.html', # optional
    ContentType => 'text/html',
    Charset     => 'utf-8',         # optional
    Content     => $Content,
    NoCache     => 1,               # optional
);

Let's try whether adding the Charset parameter fixes the problem.

[bschmalhofer]

Testing this in the virtual OTOBO installation where the error was noticed. Opening the snippet, without the fix from 32119fa, shows a sane 'ö'. Doublechecking the page info, "Extras" -> "Seiteninformationen" confirms that the virtual OTOBO delivers the page content as ISO-8859-1, or windows-1252. This means that for some reason there is defined difference between the behaviour in the local installation and in the virtual OTOBO. But in a way that divergent behaviour is fine. Perl make no promises regarding the internal representation of strings. Therefore Perl is allowed to downgrade the HTML snippet if it possible. It is the responsibility of the OTOBO developer to make sure that the delivered bytes and the Content-Type declaration are in sync. This guarantee is achieved by passing Charset => 'utf-8' to the method Attachment().

Doublechecking this by manually adapting Kernel::Modules::AgentStatistics.pm gave the expected result. The snippet still looks fine, while the page info reports UTF-8 as charset. Inserting the UTF-8 content in the UTF-8 encoded containing page now works fine too.

[bschmalhofer]

PR is merged. Created a new virtual OTOBO for the rel-10_1 branch. The page otobo/index.pl?Action=AgentStatistics;Subaction=Add looks fine. Closing this issue.

[svenoe]

There are probably 200 instances where Attachment() is called and likely returned somewhere. Do you know at which place the encoding was "fixed" in 10.0? Because either we have to check all places individually, or we should try to change this in a common spot... I will reopen this for the moment, if I misunderstood something, please just close it again.

[bschmalhofer]

TODO:

• Check the calls to ->Attachment() for other problematic cases

[bschmalhofer]

Checked the calls to Attachment(). Most of them looked sensible, but for some I added the Charset='utf-8' declarations.

[bschmalhofer]

TODO: check whether one of these test failures is related to the changes:

> Test Summary Report
> -------------------
> /opt/otobo/scripts/test/Compile.t                                                                         (Wstat: 0 Tests: 1767 Failed: 0)
>   TODO passed:   999, 1024
> /opt/otobo/scripts/test/Selenium/Agent/Admin/AdminDynamicField.t                                          (Wstat: 256 Tests: 929 Failed: 1)
>   Failed test:  929
>   Non-zero exit status: 1
> /opt/otobo/scripts/test/Selenium/Agent/Admin/AdminSystemMaintenance.t                                     (Wstat: 256 Tests: 240 Failed: 1)
>   Failed test:  120
>   Non-zero exit status: 1
> /opt/otobo/scripts/test/Selenium/Agent/DynamicFieldShowLink.t                                             (Wstat: 256 Tests: 21 Failed: 1)
>   Failed test:  10
>   Non-zero exit status: 1
> /opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
>   TODO passed:   21, 24, 28
> /opt/otobo/scripts/test/WebUserAgent.t                                                                    (Wstat: 1024 Tests: 121 Failed: 4)
>   Failed tests:  76-77, 89, 101
>   Non-zero exit status: 4
> Files=956, Tests=81746, 6681 wallclock secs (31.70 usr  2.82 sys + 1747.02 cusr 207.89 csys = 1989.43 CPU)
> Result: FAIL
> ran tests for product OTOBO 10.1.x on host ef1959b690ec .
> 

[bschmalhofer]

There has been an issue with an missing or misspelled Charset declaration for JSON responses. Checking RFC8259, https://datatracker.ietf.org/doc/html/rfc8259, I see that the charset declaration is optional:

Note: No "charset" parameter is defined for this registration.
Adding one really has no effect on compliant recipients.

Therefore Kernel::System::Web::Response::Finalize() should encode all JSON responses, that is responses with Content-Type: application/json.

[bschmalhofer]

No further problems were encountered. Closing this issue.