feat(gazelle): Include types/stubs packages

[ewianda]

This PR adds logic that checks if a package has a corresponding types or stubs package and automatically adds that to the BUILD file. This is useful for typeckers e.g pyright , mypy

[aignas]

Thank you @ewianda for the contribution. @dougthor42, would you have time to look at this?

[dougthor42]

I'll probably be able to test this out sometime this week.

Until then, here are some review comments. Also, please add a flag to turn this on or off. It should be off by default for now, and then in a couple releases we can consider making it on by default.

[dougthor42]

Please add documentation to gazelle/README.md

---

Running with the default include_stub_packages = False didn't cause any problems with our large project running gazelle, so 👍 there.

Running with include_stub_packages = True didn't add any *_types or _stubs deps to targets 👎.

Is there something else I need to do? We already have some type/stub packages as part of our requirements.in (and thus requirements.txt and gazelle_python.yaml). For example, we have sqlmypy: sqlalchemy_stubs and annotated_types: annotated_types in gazelle_python.yaml

Test methodology

1. bazel run //:buildozer 'remove deps' '//src/pyle_xc/...:*'
2. bazel run //:gazelle src/pyle_xc
3. git diff and see what has changed.

[ewianda]

Please add documentation to gazelle/README.md

Running with the default include_stub_packages = False didn't cause any problems with our large project running gazelle, so 👍 there.

Running with include_stub_packages = True didn't add any *_types or _stubs deps to targets 👎.

Is there something else I need to do? We already have some type/stub packages as part of our requirements.in (and thus requirements.txt and gazelle_python.yaml). For example, we have sqlmypy: sqlalchemy_stubs and annotated_types: annotated_types in gazelle_python.yaml

Test methodology

1. bazel run //:buildozer 'remove deps' '//src/pyle_xc/...:*'
2. bazel run //:gazelle src/pyle_xc
3. git diff and see what has changed.

Thanks, @dougthor42, for testing. I think you missed bazel run //:gazelle_python_manifest.update to add the types to the manifest.

Did you fix 631ab19
locally before running the test?

Also, the logic is that sqlalchemy has to be a direct dependency before sqlalchemy-stubs will be added to the BUILD file. annotated-types doesn't fall under this category as well.

[dougthor42]

Did you fix 631ab19 locally before running the test?

Ah, no I did not. I applied that change and ran gazelle_python_manifest.update. Here's the full diff of gazelle_python.yaml:

diff --git a/gazelle_python.yaml b/gazelle_python.yaml
index 5752a8274a..66e8ab5c4f 100644
--- a/gazelle_python.yaml
+++ b/gazelle_python.yaml
@@ -395,6 +395,7 @@ manifest:
     pymatching: PyMatching
     pymodbus: pymodbus
     pyparsing: pyparsing
+    pyqt5_stubs: pyqt5_stubs
     pytest: pytest
     pytest_asyncio: pytest_asyncio
     pytest_benchmark: pytest_benchmark
@@ -484,9 +485,8 @@ manifest:
     sphinxcontrib.svgbob: sphinxcontrib_svgbob
     sqlalchemy: SQLAlchemy
     sqlalchemy_bigquery: sqlalchemy_bigquery
-    sqlmypy: sqlalchemy_stubs
+    sqlalchemy_stubs: sqlalchemy_stubs
     sqlparse: sqlparse
-    sqltyping: sqlalchemy_stubs
     stack_data: stack_data
     stim: stim
     stimcirq: stimcirq

So it's is correctly pulling in new stubs (like pyqt5_stubs) 👍 but also changing names a bit 👎. This 2nd part is an issue if user code ends up trying to import the original name (eg sqlmypy) directly: gazelle will fail saying "sqlmypy" is an invalid dependency.

Luckily our code doesn't import sqlmypy or sqltyping directly, so I didn't run into that error and was able to move on and run gazelle. Here's a partial diff:

diff --git a/src/labrad/servers/GUIs/ADR/BUILD.bazel b/src/labrad/servers/GUIs/ADR/BUILD.bazel
index 0667587998..b8e716cc30 100644
--- a/src/labrad/servers/GUIs/ADR/BUILD.bazel
+++ b/src/labrad/servers/GUIs/ADR/BUILD.bazel
@@ -11,5 +11,6 @@ py_library(
         "//src/pyle/datavault:util",
         "@pypi//duet",
         "@pypi//pyqt5",
+        "@pypi//pyqt5_stubs",
     ],
 )
diff --git a/src/pyle/cloud/bigquery/scripts/BUILD.bazel b/src/pyle/cloud/bigquery/scripts/BUILD.bazel
index 0d10c6b62d..f86c846852 100644
--- a/src/pyle/cloud/bigquery/scripts/BUILD.bazel
+++ b/src/pyle/cloud/bigquery/scripts/BUILD.bazel
@@ -86,6 +86,7 @@ pyle_py_binary(
         "@pypi//google_cloud_bigquery",
         "@pypi//pandas",
         "@pypi//sqlalchemy",
+        "@pypi//sqlalchemy_stubs",
     ],
 )

So overall it looks like it's WAI 👍 🎉 🎊

[ewianda]

This 2nd part is an issue if user code ends up trying to import the original name (eg sqlmypy) directly: gazelle will fail saying "sqlmypy" is an invalid dependency.

These are stub files, I will be surprised if folks are importing this directly since they are meant for type-checking tools, how ever we can let dig_wheel proceed to add package names.

[ewianda]

ping @dougthor42 is there something else you will like to change for this to go in

[aignas]

I have merged main here since it has various CI fixes.

[dougthor42]

Thanks for the ping. There are still two open items:

1. Please add documentation to gazelle/README.md. A comment and example here

   rules_python/gazelle/README.md

   Lines 109 to 122 in ca98773

   	gazelle_python_manifest(
   	name = "gazelle_python_manifest",
   	modules_mapping = ":modules_map",
   	# This is what we called our `pip_parse` rule, where third-party
   	# python libraries are loaded in BUILD files.
   	pip_repository_name = "pip",
   	# This should point to wherever we declare our python dependencies
   	# (the same as what we passed to the modules_mapping rule in WORKSPACE)
   	# This argument is optional. If provided, the `.test` target is very
   	# fast because it just has to check an integrity field. If not provided,
   	# the integrity field is not added to the manifest which can help avoid
   	# merge conflicts in large repos.
   	requirements = "//:requirements_lock.txt",
   	)

   is probably sufficient.
2. Please add a short blurb to CHANGELOG.md describing the new feature.

Otherwise this LGTM.

[dougthor42]

Other new features get labeled as "experimental" for a couple releases (eg: experimental_requirement_cycles). Do we want to do the same here as experimental_include_stub_packages?

Given that it's opt-in and that no other gazelle features are experimental, I'm inclined to leave it as-is.

[ewianda]

I vote to leave it as is

[aignas]

Thank you @dougthor42 for the review and thank you @ewianda for the contribution. Stamping and merging.

[aignas]

nit: FYI, this could return a list of repos that are created internally via the bazel skylib's extension metadata. Then bazel mod tidy would ensure that the use_repo statement in the MODULE.bazel is always up to date.