Optimize Gazelle's filesystem traversal

[dzbarsky]

What type of PR is this?

Uncomment one line below and remove others.

Performance

What package or component does this PR mostly affect?

All

What does this PR do? Why is it needed?

This PR improves Gazelle's filesystem walk in 2 ways:

1. By using a parallel walk, resolving a long-standing TODO
2. By adding an optional interface that plugins can implement to receive the directory contents, thus saving them from issuing duplicative syscalls to gather the same info

Which issues(s) does this PR fix?

Fixes #1819

Other notes for review

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[dzbarsky]

@linzhp please take a look?

[uhthomas]

Why not simply use filepath.WalkDir instead? Doesn't need a new dependency and achieves the same goal of being more efficient by not calling os.Lstat for each file or directory.

[DavidZbarsky-at]

Why not simply use filepath.WalkDir instead? Doesn't need a new dependency and achieves the same goal of being more efficient by not calling os.Lstat for each file or directory.

Because it doesn't parallelize the walk. This is faster

See also #1737

[uhthomas]

Because it doesn't parallelize the walk. This is faster

I understand that, though I imagine a lot of the time savings will be from not calling os.Lstat. Do you have some figures to demonstrate the difference parallelisation makes here? Such, if WalkDir provides a significant improvement over Walk, but the difference between WalkDir and fastwalk is not as significant, then the maintenance burden of an external package may not be worth it.

[DavidZbarsky-at]

Because it doesn't parallelize the walk. This is faster

I understand that, though I imagine a lot of the time savings will be from not calling os.Lstat. Do you have some figures to demonstrate the difference parallelisation makes here? Such, if WalkDir provides a significant improvement over Walk, but the difference between WalkDir and fastwalk is not as significant, then the maintenance burden of an external package may not be worth it.

I don't have exact numbers offhand anymore; we wrote these patches months ago. FWIW WalkDir on its own is not trivial to use due to the complex iteration; that's part of the purpose of the Trie. Also in our testing the parallelization definitely made a significant different; between that and the Trie we reduced the runtime from 10s to 4.5s.

We do run gazelle as part of pre-commit hook so we are fairly perf-sensitive on this. I recognize that perhaps performance is not a primary motivation for the project; feel free to close out the PR if you aren't interested.

(BTW are you sure the os.Lstat is still an issue? I believe os.Readdir is already avoiding it since Go 1.16, perhaps you were thinking of ioutil.Readdir? I did measure some speedups when I contributed https://github.com/bazelbuild/bazel-gazelle/pull/1704/files)

[linzhp]

I agreed with @uhthomas that we should avoid another external library before comparing it with filepath.WalkDir.