Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test and fix the wrappers in bin/cgi-bin #1054

Closed
bschmalhofer opened this issue Jun 3, 2021 · 19 comments
Closed

Test and fix the wrappers in bin/cgi-bin #1054

bschmalhofer opened this issue Jun 3, 2021 · 19 comments
Labels
selenium Related to fixing or improving testing with Selenium
Milestone

Comments

@bschmalhofer
Copy link
Contributor

bschmalhofer commented Jun 3, 2021

Once this works with PerlResponseHandler Plack::Handler::Apache2 we should check whether there are any problems with the wrapper script in bin/cgi-bin . See also #320

Originally posted by @bschmalhofer in #1032 (comment)

@bschmalhofer
Copy link
Contributor Author

bschmalhofer commented Jun 9, 2021

TODO:

  • investigate whether scripts/apache2-perl-startup.pl should be kept or removed
  • run the test suite

bschmalhofer added a commit that referenced this issue Jun 14, 2021
This also avoids some warnings in the error log.
bschmalhofer added a commit that referenced this issue Jun 14, 2021
This is similar to the old apache2-httpd.include.conf
bschmalhofer added a commit that referenced this issue Jun 14, 2021
Currently not used, so remove it for decreasing the level of confusion.
@bschmalhofer
Copy link
Contributor Author

scripts/apache2-perl-startup.pl is gone now.

Manual testing showed a strange effect.

$ uni �
� - U+0FFFD - REPLACEMENT CHARACTER

In the Apache error log there are also messages like:

[Mon Jun 14 13:46:25 2021] -e: Wide character in print at /usr/local/share/perl/5.30.0/Plack/Handler/CGI.pm line 143.

I suspect that some modules meddle with the mode of STDOUT. Needs to be investigated.

@bschmalhofer
Copy link
Contributor Author

bschmalhofer commented Jun 23, 2021

Investigated why scripts/test/DynamicField/Driver/WebService/PossibleValuesGet.t failes when using the wrapper scripts. There a webservice, using OTOBO itself as endpoint, is used to retrieve info about a test ticket. This fails because the testscript can't parse the response from the OTOBO web server:

SOAP Transport: Could not deserialize received XML data

The communication is nicely tracked in the table gi_debugger_entry_content and can be nicely inspected with the Debugger available via the Webservice Admin page. The only stumbling block is that the test script mustn't delete webservice under test.

Let's compare the response from Apache using Plack::Handler::Apache2 and from Apache using the wrapper scripts:

Plack::Handler::Apache2:

<?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><TicketGetResponse xmlns="http://www.otobo.org/TicketConnector/"><Ticket>.../Ticket></TicketGetResponse></soap:Body></soap:Envelope>

Wrapper scripts in cgi-bin:

Status: 200 OK
Connection: close
Content-Length: 1521
Content-Type: text/xml; charset=UTF-8
 
<?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><TicketGetResponse xmlns="http://www.otobo.org/TicketConnector/"><Ticket>....</Ticket></TicketGetResponse></soap:Body></soap:Envelope>

The first version is working fine, as the HTTP headers are not part of the SOAP content. The latter version is broken, as HTTP headers end up in the SOAP content. Needs to be fixed. Propably specifying non-parsed-headers somewhere.

bschmalhofer added a commit that referenced this issue Jun 24, 2021
Looks like Plack::Handler::Apache2 already handles the headers
bschmalhofer added a commit that referenced this issue Jun 24, 2021
in order to get a more readable report
@bschmalhofer
Copy link
Contributor Author

Fixed initial problem concerning non-parsed headers. This leaves 100 test failure, of which some are related to SOAP too.

Message: DebugLog error: Summary: Error in SOAP call: 500 Status line too long (limit is 8192) at Kernel/GenericInterface/Transport/HTTP/SOAP.pm line 775. Data : Error in SOAP call: 500 Status line too long (limit is 8192) at Kernel/GenericInterface/Transport/HTTP/SOAP.pm line 775..

Traceback (100729):

@bschmalhofer
Copy link
Contributor Author

There are still SOAP related failures when the wrapper scripts in cgi-bin are used. An example for such an failing script is scripts/test/GenericInterface/Operation/Ticket/TicketGet.t . The error message is:

SOAP::Transport::HTTP::Client::send_receive: 500 Status line too long (limit is 8192)

This error message is emitted from NET::HTTP::Methods::my_readline() when trying to read in the HTTP header lines of the SOAP response.

Let's first take a look whether the webservice is generating the expected result. For that, the test script need to be adapted:

  • Remove TestMode => 1from the webservice config in the TicketGet.t
  • Remove the WebserviceDelete() calls in the test script
  • Limit the test script to the cases "SessionCreate" and "Test Ticket 4 With All Articles and Attachments". The latter case is the first test that actually fails.
  • Copy the changed script into the top level otobo dir, for convenience

Now the Debugger in the web service Admin page can be used for inspecting the web service under test. The response generated on the server side looks fine. Hint: the XML response is longer than 8191 chars, HTTP headers are not displayed. But parsing the response on the client side fails with the known error.

So, let's dig in deeper. Tracing can be enabled when SOAP::Lite is imported into Kernel::GenericInterface::Transport::HTTP::SOAP: use SOAP::Lite +trace => 'all';
Now the output from prove -v -I . -I Kernel/cpan-lib/ TicketGet.t 2>&1 | tee prove.out contains debugging information, including the HTTP headers that were sent and received. For comparison the output can be generated for the two relevant cases:

  1. scripts/apache2-httpd.include.conf linked into /etc/apache2/sites-enabled (working)
  2. scripts/apache2-httpd.include.conf linked into /etc/apache2/sites-enabled (with failures)

The relevant difference are in the response from the web service. Here is the response for SessionCreate:

re. 1.

Connection: close
Date: Sat, 26 Jun 2021 11:23:17 GMT
Server: Apache/2.4.41 (Ubuntu)
Vary: Accept-Encoding
Content-Length: 513
Content-Type: text/xml; charset=UTF-8
Client-Date: Sat, 26 Jun 2021 11:23:18 GMT
Client-Peer: 192.168.178.29:80
Client-Response-Num: 1

<?xml version="1.0" encoding="UTF-8"?><soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><SessionCreateResponse xmlns="http://otobo.org/SoapTestInterface/"><SessionID>fMsWLN1LmBh34AFVq3TQ5f3SH9Tn9V3O</SessionID></SessionCreateResponse></soap:Body></soap:Envelope>

re. 2.

Client-Date: Sat, 26 Jun 2021 11:19:16 GMT
Client-Peer: 192.168.178.29:80
Client-Response-Num: 1

S8TdJZSAWup9xoocFzcbtoiGhIi5JKQq

The Client-* header fields are generated internally in the client library. So it looks like no HTTP headers are emitted at all in the case 2 . SOAP::Lite then assumes that the XML-Response contains headers and sometimes ran into 'Line to long'. So this looks like it is related to #1100.

@bschmalhofer
Copy link
Contributor Author

Another class of failures from the cgi-bin wrapper scripts:

not ok 14 - Breadcrumb text 'Add Auto Response' is found on screen (is 'Automatische Antwort hinzuf�gen' should be 'Automatische Antwort hinzufügen')

Looks like funny double or triple encoding. Needs to be investigated.

@bschmalhofer
Copy link
Contributor Author

There are actually two cases where the scripts in bin/cgi-bin are used.

  • CGI scripts are running under Modperl::Registry
  • CGI script are running as ancient CGI, where a new process is started for every request

Both cases should be supported. The Modperl::Registry is the more important case. The latter case should be handled in a different issue. There might be a problem with the detection of mod_perl. It could be that Apache2::Module can successfully be required, but mod_perl is effectively not active.

@bschmalhofer
Copy link
Contributor Author

Looks better now, but still some warnings and failures. First the warnings:

[Sat Jul 10 21:18:13 2021] otobo.psgi: Subroutine Dispatch redefined at /opt/otobo/bin/cgi-bin/../psgi-bin/otobo.psgi line 95.
[Sat Jul 10 21:18:13 2021] otobo.psgi: Subroutine DispatchMultipleTicketMethods redefined at /opt/otobo/bin/cgi-bin/../psgi-bin/otobo.psgi line 173.
[Sat Jul 10 21:20:15 2021] otobo.psgi: Subroutine new redefined at /opt/otobo/bin/cgi-bin/../psgi-bin/otobo.psgi line 87.

These warnings come from the OTOBO::RPC package in otobo.psgi.

The failures:

/opt/otobo/scripts/test/Selenium/Agent/AgentTicketBulkACL.t (Wstat: 1280 Tests: 212 Failed: 5)
/opt/otobo/scripts/test/Selenium/Customer/CustomerPasswordRecovery.t (Wstat: 512 Tests: 5 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketOverview.t (Wstat: 512 Tests: 26 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/FooterLinks.t (Wstat: 512 Tests: 3 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/Login.t (Wstat: 512 Tests: 3 Failed: 2)

@bschmalhofer
Copy link
Contributor Author

Fixed the missing headers in CustomerLogin(). Run the test suite and got three failing scripts:

/opt/otobo/scripts/test/Selenium/Agent/AgentTicketMoveACL.t (Wstat: 768 Tests: 28 Failed: 3)
/opt/otobo/scripts/test/Selenium/Agent/AgentTicketProcess.t (Wstat: 512 Tests: 30 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketOverview.t (Wstat: 256 Tests: 18 Failed: 1)

@bschmalhofer bschmalhofer added the selenium Related to fixing or improving testing with Selenium label Jul 11, 2021
@bschmalhofer
Copy link
Contributor Author

Another test run:

/opt/otobo/scripts/test/Selenium/Agent/AgentTicketActionCommonACL.t (Wstat: 2560 Tests: 120 Failed: 10)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketOverview.t (Wstat: 256 Tests: 18 Failed: 1)

@bschmalhofer
Copy link
Contributor Author

Yet another test run:

/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketOverview.t (Wstat: 256 Tests: 18 Failed: 1)

Looks like CustomerTicketOverview.t is the only consistenly failing test.

@bschmalhofer
Copy link
Contributor Author

Reran the test suite with a sleep in CustomerTicketOverview.t . This is all a bit unpredictable.

/opt/otobo/scripts/test/Selenium/Agent/Admin/AdminSession.t (Wstat: 512 Tests: 22 Failed: 2)
/opt/otobo/scripts/test/Selenium/Agent/AgentDashboardCommon.t (Wstat: 256 Tests: 30 Failed: 1)
/opt/otobo/scripts/test/Selenium/Agent/AgentPasswordRecovery.t (Wstat: 512 Tests: 6 Failed: 2)
/opt/otobo/scripts/test/Selenium/Agent/Login.t (Wstat: 512 Tests: 5 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/CustomerPasswordRecovery.t (Wstat: 512 Tests: 5 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/FooterLinks.t (Wstat: 512 Tests: 3 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/Login.t (Wstat: 512 Tests: 3 Failed: 2)
/opt/otobo/scripts/test/Selenium/SysConfig/AgentSetting.t (Wstat: 512 Tests: 12 Failed: 2)

@bschmalhofer
Copy link
Contributor Author

Yet another run:

/opt/otobo/scripts/test/Selenium/Agent/AgentTicketActionCommonACL.t (Wstat: 2560 Tests: 120 Failed: 10)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketOverview.t (Wstat: 512 Tests: 26 Failed: 2)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketProcess.t (Wstat: 256 Tests: 30 Failed: 1)
/opt/otobo/scripts/test/Selenium/Output/TicketZoom/ProcessInformation.t (Wstat: 768 Tests: 26 Failed: 3)

@bschmalhofer
Copy link
Contributor Author

Still a mixed bag:

/opt/otobo/scripts/test/Selenium/Agent/AgentTicketActionCommonACL.t (Wstat: 2560 Tests: 120 Failed: 10)
/opt/otobo/scripts/test/Selenium/Agent/AgentTicketSplitInlineImage.t (Wstat: 256 Tests: 28 Failed: 1)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketOverview.t (Wstat: 256 Tests: 18 Failed: 1)
/opt/otobo/scripts/test/Selenium/Customer/CustomerTicketProcess.t (Wstat: 256 Tests: 30 Failed: 1)
/opt/otobo/scripts/test/Selenium/Customer/MultiAttachmentUpload/TicketZoom.t (Wstat: 256 Tests: 59 Failed: 1)

@bschmalhofer bschmalhofer modified the milestones: OTOBO 10.1, OTOBO 10.1.1 Jan 31, 2022
@bschmalhofer
Copy link
Contributor Author

Still failures before the OTOBO 10.1 Beta 2 release:

Test Summary Report
-------------------
/opt/otobo/scripts/test/Compile.t                                                                         (Wstat: 0 Tests: 1801 Failed: 0)
  TODO passed:   1010, 1035
/opt/otobo/scripts/test/GenericInterface/Operation/Ticket/TicketGet.t                                     (Wstat: 3584 Tests: 62 Failed: 14)
  Failed tests:  30, 34-42, 44, 46-48
  Non-zero exit status: 14
/opt/otobo/scripts/test/GenericInterface/Operation/Ticket/TicketUpdate.t                                  (Wstat: 3072 Tests: 136 Failed: 12)
  Failed tests:  93-94, 102-103, 111-115, 123-125
  Non-zero exit status: 12
/opt/otobo/scripts/test/GenericInterface/Operation/Ticket/TicketUpdateIncludeTicketData.t                 (Wstat: 3584 Tests: 83 Failed: 14)
  Failed tests:  51, 53-54, 56-59, 68, 70-71, 73-76
  Non-zero exit status: 14
/opt/otobo/scripts/test/GenericInterface/Transport/HTTP/REST.t                                            (Wstat: 1280 Tests: 26 Failed: 5)
  Failed tests:  16-20
  Non-zero exit status: 5
/opt/otobo/scripts/test/GenericInterface/Transport/HTTP/SOAP.t                                            (Wstat: 256 Tests: 58 Failed: 1)
  Failed test:  7
  Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t   (Wstat: 768 Tests: 87 Failed: 3)
  Failed tests:  81, 84, 87
  Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Customer/Ajax/ErrorHandling.t                                            (Wstat: 512 Tests: 19 Failed: 2)
  Failed tests:  16, 19
  TODO passed:   5-12
  Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/TestingMethods.t                                                         (Wstat: 0 Tests: 36 Failed: 0)
  TODO passed:   21, 24, 28
/opt/otobo/scripts/test/WebUserAgent.t                                                                    (Wstat: 1024 Tests: 121 Failed: 4)
  Failed tests:  76-77, 89, 101
  Non-zero exit status: 4
Files=960, Tests=81726, 6695 wallclock secs (25.65 usr  2.88 sys + 1390.11 cusr 171.00 csys = 1589.64 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host bernhard-HP-250-G6-Notebook-PC .

@bschmalhofer
Copy link
Contributor Author

At least some of these failures were due to triple encoding of non-ASCII characters. Lets take a case in ./scripts/test/GenericInterface/Operation/Ticket/TicketGet.t. The title of ticket 4 contains the German umlaut
'ä' encoded in UTF-8 as C3A4. This two bytes are double encoded twice along the line and we end up with eight bytes. The culprit seems to be utf8::encode($Line) in bin/cgi-bin/nph-genericinterface.pl. Investigating why this script differs from the other scripts brought me back to #1104. I had totally forgotten about that.

Removing the questionable line made TicketGet.t succeed. Let's see how this works out for the complete test suite.

@bschmalhofer
Copy link
Contributor Author

bschmalhofer commented Feb 1, 2022

Removed the utf8::encode() that apparently caused double encoding. Unit tests look better now.

  • scripts/test/Selenium/Agent/Admin/AdminSort.t seems to be sporadic.
  • scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t is a known problem that also occurs when running under Docker.
  • scripts/test/Selenium/Customer/Ajax/ErrorHandling.t is still a problem.

Test Summary Report

/opt/otobo/scripts/test/Compile.t (Wstat: 0 Tests: 1801 Failed: 0)
TODO passed: 1010, 1035
/opt/otobo/scripts/test/Selenium/Agent/Admin/AdminSort.t (Wstat: 256 Tests: 16 Failed: 1)
Failed test: 16
Non-zero exit status: 1
/opt/otobo/scripts/test/Selenium/Agent/Admin/ProcessManagement/AdminProcessManagementTransitionAction.t (Wstat: 768 Tests: 87 Failed: 3)
Failed tests: 81, 84, 87
Non-zero exit status: 3
/opt/otobo/scripts/test/Selenium/Customer/Ajax/ErrorHandling.t (Wstat: 512 Tests: 19 Failed: 2)
Failed tests: 16, 19
TODO passed: 5-12
Non-zero exit status: 2
/opt/otobo/scripts/test/Selenium/TestingMethods.t (Wstat: 0 Tests: 36 Failed: 0)
TODO passed: 21, 24, 28
/opt/otobo/scripts/test/WebUserAgent.t (Wstat: 1024 Tests: 121 Failed: 4)
Failed tests: 76-77, 89, 101
Non-zero exit status: 4
Files=960, Tests=81619, 8153 wallclock secs (25.87 usr 3.10 sys + 1435.03 cusr 174.70 csys = 1638.70 CPU)
Result: FAIL
ran tests for product OTOBO 10.1.x on host bernhard-HP-250-G6-Notebook-PC .

@bschmalhofer
Copy link
Contributor Author

bschmalhofer commented Feb 2, 2022

The failures for scripts/test/Selenium/Customer/Ajax/ErrorHandling.t are related to the still open issue #909. So let's ignore them for this issue, especially as other tests succeed that fail when running with Gazelle.
The failures for AdminProcessManagementTransitionAction.t also occur when running under Gazelle, so they are most likely not related to the cgi-bin Scripts.
Closing this issue now. Open a new issue when new failures crop up.

@bschmalhofer
Copy link
Contributor Author

Supporting the scripts in bin/cgi-bin is a kind of hassle and the hope is that nobody will use the. I propose to remove them in OTOBO 11.0.1 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
selenium Related to fixing or improving testing with Selenium
Projects
None yet
Development

No branches or pull requests

1 participant