[v6.20.0] Console Reporter broken for PHP flavour #2425

llaville · 2023-03-06T07:22:02Z

Describe the bug
The summary table is not completed after an analysis.

To Reproduce

Download latest image of PHP flavour with docker pull oxsecurity/megalinter-php:v6 (digest: sha256:b38beb2a78234c3719a473b318b9e67651f56423c9a0faf979a1a2c8c2b035d3)
Run following command sudo rm -rf megalinter-reports/ && time docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:rw -v $(pwd):/tmp/lint:rw oxsecurity/megalinter-php:v6 on a local git clone of my repository https://github.com/llaville/php-compatinfo-db (master branch)
See error

Expected behavior
As previous version a summary table complete.

Screenshots

Additional context
Local platform : Ubuntu 22.04 LTS on WSL2

The text was updated successfully, but these errors were encountered:

llaville · 2023-03-06T07:32:14Z

Here are results I got with previous version

----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 2023-02-05T11:58:18Z
 - Image Revision: f8d535e
 - Image Version: v6
----------------------------------------------------------------------------------------------------

Docker image has disgest sha256:2744b2f1d9e919e13d1190dfaaa95bee93d22d6828bd0e7449396ad7c555a1cc

BTW v6.20.0 has header version identification changed. Now I got

----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 
 - Image Revision: 0d699a348a2c104040efd78f2d2970e6ae8e5769
 - Image Version: 0d699a348a2c104040efd78f2d2970e6ae8e5769
----------------------------------------------------------------------------------------------------

Compare to previous version !

llaville · 2023-03-06T07:34:19Z

And summary table crash gave me other results each time I run it. Sometimes it break results after, sometimes I got this error

+----SUMMARY------+----------------------+---------------+-------+-------+--------+--------------+
| Descriptor      | Linter               | Mode          | Files | Fixed | Errors | Elapsed time |
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/multiprocessing_logging-0.3.4-py3.11.egg/multiprocessing_logging.py", line 77, in _receive
    record = self.queue.get(timeout=0.2)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/queues.py", line 117, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 215, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 420, in _recv_bytes
    return self._recv(size)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
    chunk = read(handle, remaining)
            ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object cannot be interpreted as an integer
Exception in thread mp-handler-0:
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/site-packages/multiprocessing_logging-0.3.4-py3.11.egg/multiprocessing_logging.py", line 77, in _receive
    record = self.queue.get(timeout=0.2)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/queues.py", line 117, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 215, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 420, in _recv_bytes
    return self._recv(size)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/multiprocessing/connection.py", line 378, in _recv
    chunk = read(handle, remaining)
            ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object cannot be interpreted as an integer

real    0m31.776s
user    0m0.096s
sys     0m0.132s

Kurt-von-Laven · 2023-03-06T07:37:02Z

This may be related to #2348. Which OS are you on, and are you using rootless or rootful Docker?

llaville · 2023-03-06T07:41:16Z

@Kurt-von-Laven No it seems to come from parallelization ! I've just tried to disable it with PARALLEL: false in my megalinter config file, and I got results

Kurt-von-Laven · 2023-03-06T07:52:50Z

That's great to hear. Actually the answer to my previous question may still be relevant to #2348.

llaville · 2023-03-06T07:54:32Z

I confim, each time I run with PARALLEL directive enabled, the console reporter crash.
Now I'll disable it by default to avoid issue with GitHub Workflow, and enable it locally :

sudo rm -rf megalinter-reports/ && time docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:rw -v $(pwd):/tmp/lint:rw -e PARALLEL=true oxsecurity/megalinter-php:v6

Reminder for me : llaville/php-compatinfo-db@90dfb29

Kurt-von-Laven · 2023-03-06T08:21:09Z

Makes sense. I meant this question though:

Which OS are you on, and are you using rootless or rootful Docker?

llaville · 2023-03-06T08:34:47Z

Rootfull Docker (default) on Ubuntu 20.04.2 LTS WSL2

Kurt-von-Laven · 2023-03-06T08:52:22Z

Thank you! That helps confirm our theory that the v6.20.0 images can only be pulled using rootful Docker.

nvuillam · 2023-03-06T18:22:05Z

Grmblmblm I hope it's not related to multiprocessing-logging

llaville · 2023-03-08T08:14:05Z

FYI: I've just tested version 6.20.1, and this issue still exists !

But the creation date in header was fixed (thanks)

----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 2023-03-07T19:35:10Z
 - Image Revision: 7411316caa2897b3839af1834e5fd8ebd5b7b81e
 - Image Version: 7411316caa2897b3839af1834e5fd8ebd5b7b81e
----------------------------------------------------------------------------------------------------

But, bad new, I've noticed that links to https://megalinter.io/**/descriptors/** were broken (404)

This is a chunk of report produced, but all links are affected !

llaville · 2023-03-08T08:23:45Z

Just to compare with previous release that did not have this issue

----------------------------------------------------------------------------------------------------
------------------------------------ MegaLinter, by OX Security ------------------------------------
----------------------------------------------------------------------------------------------------
 - Image Creation Date: 2023-02-05T11:58:18Z
 - Image Revision: f8d535e
 - Image Version: v6
----------------------------------------------------------------------------------------------------

And all link to megalinter.io descriptors were not broken (usage of latest compare to hash version used in 6.20.1)

llaville · 2023-03-08T08:28:04Z

One possible reason, is that the latest release did not point to v6.20.1 (unavailable on https://megalinter.io)

nvuillam · 2023-03-09T19:02:45Z

@llaville the latest release is always behind "latest", and becomes available in the list once there is a new release :)

It's because of the messy doc version management with mike

llaville · 2023-03-10T08:59:08Z

After a long period to retrieve full flavor docker image

Digest: sha256:fb88bc1c8af10fd31443789ba7390008bf257e1e32eb9da540495eb89a0cb9f8
Status: Downloaded newer image for oxsecurity/megalinter:v6
docker.io/oxsecurity/megalinter:v6

real    55m27.963s
user    0m1.304s
sys     0m1.784s

This issue affect all MegaLinter v6.20.1 versions (and not only the PHP flavor)

I can confirm @nvuillam your intuition (3 days ago !!!)

nvuillam · 2023-03-10T15:52:06Z

@llaville on what do you run MegaLinter ?
Parallel stuff seems to work verywhere , else we would have been flooded by issues... do you have a specific hardcore local config ? :(

nvuillam · 2023-03-10T15:52:53Z

If it is multiprocessing-logging, the maintainer took 4 months to accept the PR that fixed the bug, I was close to make a hard-fork :'(

llaville · 2023-03-11T08:54:52Z

multiprocessing-logging seems not supporting Python 3.11 (https://github.com/jruere/multiprocessing-logging/blob/master/setup.py), while MegaLinter 6.20.x did it https://github.com/oxsecurity/megalinter/blob/main/Dockerfile#L47

nvuillam · 2023-03-11T09:31:55Z

@llaville if it was the case everywhere we wouldn't have any MegaLinter 6.20.x working :/

I posted an issue on the repo, maybe they can help :/

Meanwhile, you can remain with the PARALLEL: false workaround, but I understand that's is not good for perfs, especially if you run on an engine with more than 2 cores :/

Kurt-von-Laven · 2023-03-12T08:49:28Z

I just had a non-deterministic encounter with the same stack trace as @llaville while running the v6.20.1 JavaScript flavor on ScribeMD/docker-cache. The first time it showed up was also the first time I got ESLint running (although very badly since we're using Yarn, so it can't find any tsconfig.json installed via Yarn). That makes me worry that landing #2455 will exacerbate this bug for our JavaScript users who are using mega-linter-runner.

github-actions · 2023-04-12T00:53:25Z

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

llaville added the bug Something isn't working label Mar 6, 2023

nvuillam mentioned this issue Mar 6, 2023

Some issues with v6.20.0: Workarounds ! #2431

Closed

nvuillam mentioned this issue Mar 11, 2023

Support python 3.11 ? jruere/multiprocessing-logging#61

Closed

llaville mentioned this issue Mar 27, 2023

Upgrade PHPLint to v9 #2483

Closed

nvuillam mentioned this issue Mar 27, 2023

Fix MegaLinter version displayed in logs #2485

Merged

github-actions bot added the O: stale 🤖 This issue or pull request is stale, it will be closed if there is no activity label Apr 12, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v6.20.0] Console Reporter broken for PHP flavour #2425

[v6.20.0] Console Reporter broken for PHP flavour #2425

llaville commented Mar 6, 2023

llaville commented Mar 6, 2023

llaville commented Mar 6, 2023

Kurt-von-Laven commented Mar 6, 2023

llaville commented Mar 6, 2023

Kurt-von-Laven commented Mar 6, 2023

llaville commented Mar 6, 2023 •

edited

Loading

Kurt-von-Laven commented Mar 6, 2023

llaville commented Mar 6, 2023

Kurt-von-Laven commented Mar 6, 2023

nvuillam commented Mar 6, 2023

llaville commented Mar 8, 2023

llaville commented Mar 8, 2023

llaville commented Mar 8, 2023

nvuillam commented Mar 9, 2023

llaville commented Mar 10, 2023

nvuillam commented Mar 10, 2023

nvuillam commented Mar 10, 2023

llaville commented Mar 11, 2023

nvuillam commented Mar 11, 2023

Kurt-von-Laven commented Mar 12, 2023

github-actions bot commented Apr 12, 2023

[v6.20.0] Console Reporter broken for PHP flavour #2425

[v6.20.0] Console Reporter broken for PHP flavour #2425

Comments

llaville commented Mar 6, 2023

llaville commented Mar 6, 2023

llaville commented Mar 6, 2023

Kurt-von-Laven commented Mar 6, 2023

llaville commented Mar 6, 2023

Kurt-von-Laven commented Mar 6, 2023

llaville commented Mar 6, 2023 • edited Loading

Kurt-von-Laven commented Mar 6, 2023

llaville commented Mar 6, 2023

Kurt-von-Laven commented Mar 6, 2023

nvuillam commented Mar 6, 2023

llaville commented Mar 8, 2023

llaville commented Mar 8, 2023

llaville commented Mar 8, 2023

nvuillam commented Mar 9, 2023

llaville commented Mar 10, 2023

nvuillam commented Mar 10, 2023

nvuillam commented Mar 10, 2023

llaville commented Mar 11, 2023

nvuillam commented Mar 11, 2023

Kurt-von-Laven commented Mar 12, 2023

github-actions bot commented Apr 12, 2023

llaville commented Mar 6, 2023 •

edited

Loading