Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker build fails for python source connectors in development #6443

Closed
marc-marketparts opened this issue Sep 24, 2021 · 6 comments · Fixed by #6457
Closed

Docker build fails for python source connectors in development #6443

marc-marketparts opened this issue Sep 24, 2021 · 6 comments · Fixed by #6457
Assignees
@antixar
Labels
CDK Connector Development Kit type/bug Something isn't working

Comments

@marc-marketparts
Copy link

Enviroment

  • Airbyte version: v0.29.3-alpha
  • OS Version / Instance: Ubuntu 20.04.2 LTS on Windows 10 WSL 2
  • Deployment: n/a
  • Source Connector and version: python connector in development (Python HTTP API Source, Python source)
  • Destination Connector and version: n/a
  • Severity: Medium
  • Step where error happened: While building the docker image for development testing

Current Behavior

Building a local docker image of a new python source connector fails.
(docker build . -t airbyte/source-test:dev)

Expected Behavior

Docker build should succeed.

Logs

LOG 
$ docker build . -t airbyte/source-test:dev
[+] Building 32.0s (9/15)
 => [internal] load build definition from Dockerfile                                                                                                          0.0s
 => => transferring dockerfile: 1.07kB                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                             0.0s
 => => transferring context: 112B                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/python:3.7.11-alpine3.14                                                                                   1.2s
 => [base 1/1] FROM docker.io/library/python:3.7.11-alpine3.14@sha256:fcf44c5aaf990b16a37f5ad9f27d98c5da8b9bf280e48fe3383820882926ddee                        0.0s
 => [internal] load build context                                                                                                                             0.0s
 => => transferring context: 18.55kB                                                                                                                          0.0s
 => CACHED [builder 1/4] WORKDIR /airbyte/integration_code                                                                                                    0.0s
 => [builder 2/4] RUN apk --no-cache upgrade     && pip install --upgrade pip     && apk --no-cache add tzdata                                                3.1s
 => [builder 3/4] COPY setup.py ./                                                                                                                            0.0s
 => ERROR [builder 4/4] RUN pip install --prefix=/install .                                                                                                  27.6s
------
 > [builder 4/4] RUN pip install --prefix=/install .:
#9 0.705 Processing /airbyte/integration_code
#9 0.706   DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
#9 0.706    pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
#9 1.194 Collecting airbyte-cdk
#9 1.271   Downloading airbyte_cdk-0.1.23-py3-none-any.whl (81 kB)
#9 1.423 Collecting PyYAML~=5.4
#9 1.439   Downloading PyYAML-5.4.1.tar.gz (175 kB)
#9 1.857   Installing build dependencies: started
#9 5.979   Installing build dependencies: finished with status 'done'
#9 5.981   Getting requirements to build wheel: started
#9 7.997   Getting requirements to build wheel: finished with status 'done'
#9 8.000     Preparing wheel metadata: started
#9 8.321     Preparing wheel metadata: finished with status 'done'
#9 8.428 Collecting requests
#9 8.444   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
#9 8.663 Collecting pydantic~=1.6
#9 8.681   Downloading pydantic-1.8.2-py3-none-any.whl (126 kB)
#9 8.738 Collecting backoff
#9 8.754   Downloading backoff-1.11.1-py2.py3-none-any.whl (13 kB)
#9 8.812 Collecting vcrpy
#9 8.827   Downloading vcrpy-4.1.1-py2.py3-none-any.whl (40 kB)
#9 8.858 Collecting jsonref~=0.2
#9 8.872   Downloading jsonref-0.2-py3-none-any.whl (9.3 kB)
#9 8.921 Collecting jsonschema~=3.2.0
#9 8.936   Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
#9 8.991 Collecting Deprecated~=1.2
#9 9.004   Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)
#9 9.145 Collecting pendulum
#9 9.160   Downloading pendulum-2.1.2.tar.gz (81 kB)
#9 9.376   Installing build dependencies: started
#9 11.91   Installing build dependencies: finished with status 'done'
#9 11.91   Getting requirements to build wheel: started
#9 12.00   Getting requirements to build wheel: finished with status 'done'
#9 12.00     Preparing wheel metadata: started
#9 12.32     Preparing wheel metadata: finished with status 'done'
#9 12.43 Collecting wrapt<2,>=1.10
#9 12.44   Downloading wrapt-1.12.1.tar.gz (27 kB)
#9 12.80 Collecting pyrsistent>=0.14.0
#9 12.82   Downloading pyrsistent-0.18.0.tar.gz (104 kB)
#9 13.02   Installing build dependencies: started
#9 15.72   Installing build dependencies: finished with status 'done'
#9 15.72   Getting requirements to build wheel: started
#9 15.95   Getting requirements to build wheel: finished with status 'done'
#9 15.96     Preparing wheel metadata: started
#9 16.20     Preparing wheel metadata: finished with status 'done'
#9 16.33 Collecting importlib-metadata
#9 16.33   Using cached importlib_metadata-4.8.1-py3-none-any.whl (17 kB)
#9 16.33 Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from jsonschema~=3.2.0->airbyte-cdk->source-test==0.0.0) (57.4.0)
#9 16.37 Collecting six>=1.11.0
#9 16.38   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
#9 16.43 Collecting attrs>=17.4.0
#9 16.44   Downloading attrs-21.2.0-py2.py3-none-any.whl (53 kB)
#9 16.53 Collecting typing-extensions>=3.7.4.3
#9 16.54   Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
#9 16.65 Collecting zipp>=0.5
#9 16.65   Using cached zipp-3.5.0-py3-none-any.whl (5.7 kB)
#9 16.76 Collecting pytzdata>=2020.1
#9 16.77   Downloading pytzdata-2020.1-py2.py3-none-any.whl (489 kB)
#9 16.86 Collecting python-dateutil<3.0,>=2.6
#9 16.88   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
#9 17.04 Collecting urllib3<1.27,>=1.21.1
#9 17.06   Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
#9 17.12 Collecting certifi>=2017.4.17
#9 17.13   Downloading certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
#9 17.20 Collecting charset-normalizer~=2.0.0
#9 17.23   Downloading charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
#9 17.28 Collecting idna<4,>=2.5
#9 17.30   Downloading idna-3.2-py3-none-any.whl (59 kB)
#9 17.63 Collecting yarl
#9 17.65   Downloading yarl-1.6.3.tar.gz (176 kB)
#9 17.88   Installing build dependencies: started
#9 20.49   Installing build dependencies: finished with status 'done'
#9 20.49   Getting requirements to build wheel: started
#9 20.74   Getting requirements to build wheel: finished with status 'done'
#9 20.75     Preparing wheel metadata: started
#9 21.00     Preparing wheel metadata: finished with status 'done'
#9 21.54 Collecting multidict>=4.0
#9 21.55   Downloading multidict-5.1.0.tar.gz (53 kB)
#9 21.76   Installing build dependencies: started
#9 24.41   Installing build dependencies: finished with status 'done'
#9 24.42   Getting requirements to build wheel: started
#9 24.67   Getting requirements to build wheel: finished with status 'done'
#9 24.67     Preparing wheel metadata: started
#9 24.91     Preparing wheel metadata: finished with status 'done'
#9 25.00 Building wheels for collected packages: source-test, pyrsistent, PyYAML, wrapt, pendulum, yarl, multidict
#9 25.00   Building wheel for source-test (setup.py): started
#9 25.32   Building wheel for source-test (setup.py): finished with status 'done'
#9 25.32   Created wheel for source-test: filename=source_test-0.0.0-py3-none-any.whl size=1150 sha256=879f6a486a6a1eccefaecf23640ac8aa0233675bc279506530bf53a0b2da498a
#9 25.32   Stored in directory: /tmp/pip-ephem-wheel-cache-ygj5qe9q/wheels/ae/94/dc/1c707c3e6652ba466c448a71a5a8bc364aeb95358e9f9979e6
#9 25.32   Building wheel for pyrsistent (PEP 517): started
#9 25.64   Building wheel for pyrsistent (PEP 517): finished with status 'done'
#9 25.64   Created wheel for pyrsistent: filename=pyrsistent-0.18.0-cp37-cp37m-linux_x86_64.whl size=56359 sha256=9d96cb33785bae05d8c589f0e655359090c7bbccad8a3b030afe35a296058b3e
#9 25.64   Stored in directory: /root/.cache/pip/wheels/54/35/6e/9827c774ec40ae001ae2a2ef063268add65762c4014951e97c
#9 25.65   Building wheel for PyYAML (PEP 517): started
#9 26.02   Building wheel for PyYAML (PEP 517): finished with status 'done'
#9 26.02   Created wheel for PyYAML: filename=PyYAML-5.4.1-cp37-cp37m-linux_x86_64.whl size=45657 sha256=af0d6ec876ff3e52a9beb5fbae23a76d31a5f4f673244b37b4a65fdf4daeabfa
#9 26.02   Stored in directory: /root/.cache/pip/wheels/f4/51/cc/858604f7bb9cab887106ff266a541546ad783b4fa20875051d
#9 26.02   Building wheel for wrapt (setup.py): started
#9 26.33   Building wheel for wrapt (setup.py): finished with status 'done'
#9 26.33   Created wheel for wrapt: filename=wrapt-1.12.1-py3-none-any.whl size=19572 sha256=e04d623495dee01893cbe8387d3fc7ab19fba2bba541e83316d1755ad25cd14e
#9 26.33   Stored in directory: /root/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6
#9 26.34   Building wheel for pendulum (PEP 517): started
#9 26.81   Building wheel for pendulum (PEP 517): finished with status 'done'
#9 26.81   Created wheel for pendulum: filename=pendulum-2.1.2-cp37-cp37m-linux_x86_64.whl size=109761 sha256=5bc7fbf5a8a5e8e5f16e8b49b9dfc4beb9317abab0695b828cb15091f4caae92
#9 26.81   Stored in directory: /root/.cache/pip/wheels/c4/67/26/134c5eca211ca5d86d51881b4f1e857268f1e793db490bd3d5
#9 26.81   Building wheel for yarl (PEP 517): started
#9 27.07   Building wheel for yarl (PEP 517): finished with status 'error'
#9 27.07   ERROR: Command errored out with exit status 1:
#9 27.07    command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpfs2_t19f
#9 27.07        cwd: /tmp/pip-install-fpyfpb4s/yarl_906f24384c7e46b3be586a22cbf68aa2
#9 27.07   Complete output (39 lines):
#9 27.07   **********************
#9 27.07   * Accellerated build *
#9 27.07   **********************
#9 27.07   running bdist_wheel
#9 27.07   running build
#9 27.07   running build_py
#9 27.07   creating build
#9 27.07   creating build/lib.linux-x86_64-3.7
#9 27.07   creating build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/_url.py -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/_quoting.py -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/_quoting_py.py -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/__init__.py -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   running egg_info
#9 27.07   writing yarl.egg-info/PKG-INFO
#9 27.07   writing dependency_links to yarl.egg-info/dependency_links.txt
#9 27.07   writing requirements to yarl.egg-info/requires.txt
#9 27.07   writing top-level names to yarl.egg-info/top_level.txt
#9 27.07   reading manifest file 'yarl.egg-info/SOURCES.txt'
#9 27.07   reading manifest template 'MANIFEST.in'
#9 27.07   warning: no previously-included files matching '*.pyc' found anywhere in distribution
#9 27.07   warning: no previously-included files matching '*.cache' found anywhere in distribution
#9 27.07   warning: no previously-included files found matching 'yarl/*.html'
#9 27.07   warning: no previously-included files found matching 'yarl/*.so'
#9 27.07   warning: no previously-included files found matching 'yarl/*.pyd'
#9 27.07   no previously-included directories found matching 'docs/_build'
#9 27.07   adding license file 'LICENSE'
#9 27.07   writing manifest file 'yarl.egg-info/SOURCES.txt'
#9 27.07   copying yarl/__init__.pyi -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/_quoting_c.c -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/_quoting_c.pyi -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/_quoting_c.pyx -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   copying yarl/py.typed -> build/lib.linux-x86_64-3.7/yarl
#9 27.07   running build_ext
#9 27.07   building 'yarl._quoting_c' extension
#9 27.07   creating build/temp.linux-x86_64-3.7
#9 27.07   creating build/temp.linux-x86_64-3.7/yarl
#9 27.07   gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.7m -c yarl/_quoting_c.c -o build/temp.linux-x86_64-3.7/yarl/_quoting_c.o
#9 27.07   error: command 'gcc' failed with exit status 1
#9 27.07   ----------------------------------------
#9 27.07   ERROR: Failed building wheel for yarl
#9 27.07   Building wheel for multidict (PEP 517): started
#9 27.33   Building wheel for multidict (PEP 517): finished with status 'error'
#9 27.33   ERROR: Command errored out with exit status 1:
#9 27.33    command: /usr/local/bin/python /usr/local/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpj85eolgz
#9 27.33        cwd: /tmp/pip-install-fpyfpb4s/multidict_0c85704c13dc4ab7a818dfad7c2fe538
#9 27.33   Complete output (44 lines):
#9 27.33   *********************
#9 27.33   * Accelerated build *
#9 27.33   *********************
#9 27.33   running bdist_wheel
#9 27.33   running build
#9 27.33   running build_py
#9 27.33   creating build
#9 27.33   creating build/lib.linux-x86_64-3.7
#9 27.33   creating build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/_multidict_py.py -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/_abc.py -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/_multidict_base.py -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/_compat.py -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/__init__.py -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   running egg_info
#9 27.33   writing multidict.egg-info/PKG-INFO
#9 27.33   writing dependency_links to multidict.egg-info/dependency_links.txt
#9 27.33   writing top-level names to multidict.egg-info/top_level.txt
#9 27.33   reading manifest file 'multidict.egg-info/SOURCES.txt'
#9 27.33   reading manifest template 'MANIFEST.in'
#9 27.33   warning: no previously-included files matching '*.pyc' found anywhere in distribution
#9 27.33   warning: no previously-included files found matching 'multidict/_multidict.html'
#9 27.33   warning: no previously-included files found matching 'multidict/*.so'
#9 27.33   warning: no previously-included files found matching 'multidict/*.pyd'
#9 27.33   warning: no previously-included files found matching 'multidict/*.pyd'
#9 27.33   no previously-included directories found matching 'docs/_build'
#9 27.33   adding license file 'LICENSE'
#9 27.33   writing manifest file 'multidict.egg-info/SOURCES.txt'
#9 27.33   copying multidict/__init__.pyi -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/_multidict.c -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   copying multidict/py.typed -> build/lib.linux-x86_64-3.7/multidict
#9 27.33   creating build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   copying multidict/_multilib/defs.h -> build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   copying multidict/_multilib/dict.h -> build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   copying multidict/_multilib/istr.h -> build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   copying multidict/_multilib/iter.h -> build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   copying multidict/_multilib/pair_list.h -> build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   copying multidict/_multilib/views.h -> build/lib.linux-x86_64-3.7/multidict/_multilib
#9 27.33   running build_ext
#9 27.33   building 'multidict._multidict' extension
#9 27.33   creating build/temp.linux-x86_64-3.7
#9 27.33   creating build/temp.linux-x86_64-3.7/multidict
#9 27.33   gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.7m -c multidict/_multidict.c -o build/temp.linux-x86_64-3.7/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
#9 27.33   error: command 'gcc' failed with exit status 1
#9 27.33   ----------------------------------------
#9 27.33   ERROR: Failed building wheel for multidict
#9 27.33 Successfully built source-test pyrsistent PyYAML wrapt pendulum
#9 27.33 Failed to build yarl multidict
#9 27.33 ERROR: Could not build wheels for yarl, multidict which use PEP 517 and cannot be installed directly
------
executor failed running [/bin/sh -c pip install --prefix=/install .]: exit code: 1

Steps to Reproduce

  1. generate a new Python HTTP API Source named "test"
  2. go to the source directory
  3. execute docker build . -t airbyte/source-test:dev

To be noticed:
A possible workaround to this issue is downgrading manually the airbyte cdk version in setup.py
The build succeeds with airbyte-cdk==0.1.22

Are you willing to submit a PR?

no
@marc-marketparts marc-marketparts added the type/bug Something isn't working label Sep 24, 2021
@marcosmarxm
Copy link
Member

thanks @marc-marketparts ! we're going to investigate the best way to solve this problem. Thanks for give some directions using a fixed version of CDK. @bazarnov fyi!

@marcosmarxm marcosmarxm added the CDK Connector Development Kit label Sep 24, 2021
@avida avida self-assigned this Sep 27, 2021
@avida avida added this to the Connectors, October 1st 2021 milestone Sep 27, 2021
@avida
Copy link
Contributor

avida commented Sep 27, 2021 •
edited
Loading

@marc-marketparts According to logs you are using python:3.7.11-alpine3.14 base image to build your connector. Unlike python:3.7-slim this is a very minimalist and has no gcc package required to build multidict and yarl packages which go with 0.23 cdk version. As a solution you can either switch to python:3.7-slim image or if you would like to go with python:3.7.11-alpine3.14 you can add gcc package to your dockerfile:

RUN apk update && apk add gcc libc-dev

@avida avida closed this as completed Sep 27, 2021
@sherifnada sherifnada assigned antixar and unassigned avida Sep 27, 2021
@sherifnada sherifnada removed this from the Connectors, October 1st 2021 milestone Sep 27, 2021
@sherifnada
Copy link
Contributor

@antixar could this be an issue in our usage of alpine images?

Related: https://airbytehq.slack.com/archives/C01A4CAP81L/p1630584170026400?thread_ts=1630561754.022600&cid=C01A4CAP81L

@sherifnada sherifnada reopened this Sep 27, 2021
@antixar antixar linked a pull request Sep 27, 2021 that will close this issue
🐛 Python Alpine tempales: add build-base packages #6457
Merged
@antixar
Copy link
Contributor

antixar commented Sep 27, 2021

Hi All,
I've added a hotfix of all Alpine templates.
I propose to add the set of building tools "build-base". It includes all necessary packages for c/c++ compilations

@marc-marketparts
Copy link
Author

Adding build-base to the Dockerfile works well.

IMHO, to handle security and airbyte cdk updates more transparently, the Dockerfile generated by the python generator should reference an Airbyte docker image (as the java source seems to do) instead of the Alpine/Python image.

@antixar
Copy link
Contributor

antixar commented Sep 28, 2021

Adding build-base to the Dockerfile works well.

IMHO, to handle security and airbyte cdk updates more transparently, the Dockerfile generated by the python generator should reference an Airbyte docker image (as the java source seems to do) instead of the Alpine/Python image.

Agree. I proposed same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@antixar antixar

Labels
CDK Connector Development Kit type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🐛 Python Alpine tempales: add build-base packages airbytehq/airbyte
5 participants
@avida @marcosmarxm @sherifnada @antixar @marc-marketparts