improve the gradle build correctness and performance

[postamar]

Topic

building with gradle is too slow

Revelant information

This issue tracks improving the gradle build so that it gets less in the way of developer productivity.

Definition of done.

For each project, we want the following build targets (aka tasks) to exist and work in a certain way

• generate runs code generation tasks and the like, requiring minimal dependencies.
• format depends on generate and applies formatting rules. Presently we have this only at the root of the repo, we should have this task for each project. The challenge in making that work is that python, yaml & json formatting relies on python and node, which we don't want to have to pull for every project.
• compileJava and friends (compileTestJava etc) should work repo-wide (some connectors throw compilation errors, either fix them or remove them from the build)
• assemble depends on generate and should produce a docker image (if applicable) and be fast (< 1 minute, assuming all dependencies have been built already)
• check should run all unit tests and should, by definition, also be fast (< 1 minute). This should probably include formatting checks.
• build is assemble + check (this has always been the case)
• integrationTest should run all tests which are slow and which depend on docker images produced by this repo.
• performanceTest should work (I think it's broken, at least for source-postgres)

Impact

Having done this we'll have reached a fairly minimal baseline of quality, at which point we can look into other performance improvements, like dockerizing the build (as opposed to using gradle's --build-cache perhaps), stop downloading from public pypi servers (do we really do that?), perhaps even versioning dependencies. The subsequent goal might be "fast, correct CI", who knows.

What this isn't

This isn't about messing with airbyte-ci. In fact, this effort is orthogonal and complementary to those currently undertaken by the @airbytehq/connector-operations team.

One thing to keep in mind, though, is that airbyteDocker will delegate to airbyte-ci at least for connectors, and since airbyte-ci in turns invokes gradle, so there's the potential for circularity. For this reason, airbyte-ci invokes gradle with -x airbyteDocker.

Related issues and PRs

#28817
#30187
#30060
#30011