Use of warnings.catch_warnings resets the global warnings registry

[dairiki]

There is a long-standing bug in Python python/cpython#73858 whereby any use of warnings.catch_warnings causes all memory of which warnings have already been displayed to be lost. So warnings whose action is configured as "default" (meaning show once for each module/line-number) get displayed multiple (possibly many, many) times.

Note that use of catch_warnings resets memory of what warnings have been displayed globally — use of catch_warnings by werkzeug can cause redisplay of unrelated warnings from other modules.

PR #2608 (19d55c1) introduced a lot of new uses of catch_warnings.

I'm not sure what the clean way is to work-around the issue, but it would be nice not to break the "default" warning action so drastically.

Example: too-verbose.py:

from werkzeug.urls import url_parse

for _ in range(10):
    url_parse("x")

Running

python too-verbose.py

will elicit 10 url_parse deprecation warnings. (The default warnings filter specifies an action of "default" for DeprecationWarnings, which means that they should be displayed just once per location.)

Environment:

• Python version: I've tested under 3.9 and 3.11. I think it affects all 3.x pythons.
• Werkzeug version: 2.3.3

[davidism]

The answer is to stop using the code that's showing the warnings, or report and ignore them if they're in other libraries. "Warnings are even more visible" isn't really a bad thing.

[davidism]

The reason it's used is to prevent functions that call other functions that take the same deprecated arguments from showing two warnings one of which is not addressable by users.

[dairiki]

The reason it's used is to prevent functions that call other functions that take the same deprecated arguments from showing two warnings one of which is not addressable by users.

That I realized, which is why I don't have a real clean way to work-around the issue. One way would be to use a thread-local flag or some such to disable the generation of the non-user-addressable warnings — admittedly, a messy solution.

[davidism]

I don't plan to add more complexity to the warnings, they're meant to be low overhead. I can remove catch_warnings and let the extra warnings pop up, if we think it won't be too confusing.

[dairiki]

I can remove catch_warnings and let the extra warnings pop up, if we think it won't be too confusing.

Standby. Let me try (removing the catch_warnings, that is) with my use case and see how confusing the resulting extra warnings are. I'll report back later today.

[dairiki]

I can not speak for others, but, having now tried both options, I definitely prefer the version without the catch_warnings. (I will submit the trivial PR that deletes the catch_warnings in a moment.)

---

My use case (the one that caused me to open this issue, at least) is Lektor — see lektor/lektor#1142. Lektor forces its warnings filter to "default" for all warnings¹. Running Lektor 3.3.9 with werkzeug 2.3.3 produces output like:

$ lektor --project example server
 * Project path: /.../lektor/example/Example.lektorproject
 * Output path: /.../.cache/lektor/builds/a7d6335bb90a5b7f2e89a92ec5a45133
Started source info update
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
Finished source info update in 0.05 sec
Started build
U de/index.html
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
U index.html
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
[...]

continuing on with multiple identical warnings printed for each page rendered.

With werkzeug's catch_warnings removed, the output looks like:

$ lektor --project example serverver
 * Project path: /.../lektor/example/Example.lektorproject
 * Output path: /.../.cache/lektor/builds/a7d6335bb90a5b7f2e89a92ec5a45133
Started source info update
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit
Finished source info update in 0.06 sec
Started build
U de/index.html
/home/dairiki/git/web/lektor-packages/lektor/lektor/markdown.py:21: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 2.4. Use 'urllib.parse.urlsplit' instead.
  url = url_parse(link)
/home/dairiki/git/web/lektor-packages/lektor/werkzeug/src/werkzeug/urls.py:545: DeprecationWarning: 'werkzeug.urls.URL' is deprecated and will be removed in Werkzeug 2.4. Use the 'urllib.parse' library instead.
  return result_type(scheme, netloc, url, query, fragment)
U index.html
U fr/index.html
U de/about/index.html
U de/projects/index.html
U de/404.html
[...]

---

In this case, at least, I don't think the numerous repeated identical warnings do anyone any good.

With catch_warnings removed, the extra warning about werkzeug.urls.URL being deprecated — while looking a little unclassy, being reported as due to a call from within werkzeug itself — is, in fact, incrementally useful information. (Lektor does make use of URL elsewhere. It's good to know that that's going away, too.)

Footnotes

1. I don't completely understand whether or not there's a good reason that Lektor forces the warnings filter like this, so I've been afraid to touch it. Admittedly, a fix in this case would be for Lektor to leave the warnings filter at the Python default (thus ignoring DeprecationWarnings). ↩

[davidism]

Thanks for investigating and making a PR, I've marked this for 2.3.4.