Python main dependencies are not installed with gradle build

[sherifnada]

Reported by @eugene-kulak on Slack. Summarizing the issue here.

Expected Behavior

I expect running ./gradlew :airbyte-integrations:connectors:source-<connector>:build to install all dependencies specified in [main]. However, this is not happening because ./gradlew :airbyte-integrations:connectors:source-<connector>:airbytePythonApply does not install deps in the [main] package.

We can either change the plugin task to install [main]. But the more correct option is probably to include any deps currently included in [main] in install_requires because those are not optional and , but required for the core module to function correctly. We should also remove usage of [main] from the modules and any templates. The side effect is that they will be pulled into the test code.

Acceptance Criteria

1. Move any dependencies in [main] to install_requires in all python packages
2. Update templates to indicate this change.

[cgardens]

I think the issue is that we don't actually ever run install on non-test deps described in setup.py. we just run install on the requirements.txt it seems like 🤷‍♀️ . I don't understand why we this is this way though.

https://github.com/airbytehq/airbyte/blob/master/buildSrc/src/main/groovy/airbyte-python.gradle#L47

[cgardens]

I think we may have a lot of hidden dependencies in our python stuff. I deleted my project and started from scratch and it's been a long road to get to a point where I can build the python stuff.

> Task :airbyte-integrations:bases:base-normalization:unitTest FAILED
[python] .venv/bin/python -m pytest unit_tests
         ============================= test session starts ==============================
         platform darwin -- Python 3.7.8, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
         rootdir: /Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization
         collected 0 items / 1 error

         ==================================== ERRORS ====================================
         _____________ ERROR collecting unit_tests/test_transform_config.py _____________
         ImportError while importing test module '/Users/charles/code/airbyte/airbyte-integrations/bases/base-normalization/unit_tests/test_transform_config.py'.
         Hint: make sure your test modules/packages have valid Python names.
         Traceback:
         .venv/lib/python3.7/importlib/__init__.py:127: in import_module
             return _bootstrap._gcd_import(name[level:], package, level)
         unit_tests/test_transform_config.py:28: in <module>
             from normalization import TransformConfig
         normalization/__init__.py:25: in <module>
             from .transform_catalog.transform import TransformCatalog
         normalization/transform_catalog/__init__.py:25: in <module>
             from .transform import TransformCatalog
         normalization/transform_catalog/transform.py:32: in <module>
             import yaml
         E   ModuleNotFoundError: No module named 'yaml'
         =========================== short test summary info ============================
         ERROR unit_tests/test_transform_config.py
         !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
         =============================== 1 error in 0.57s ===============================

We need to approach this this issue making sure that we do not expect any pip dependencies to be installed that we don't explicitly tell the user about.