-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use MariaDB client binaries in arm64 image for support MySQL backend #29519
Conversation
Interesting. Might works since ARM image is experimental anyway :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice one. I hope it will pass :)
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [16 lines of output]
running bdist_wheel
running build
running build_ext
building 'pyodbc' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPYODBC_VERSION=4.0.32 -I/usr/local/include/python3.7m -c src/buffer.cpp -o build/temp.linux-x86_64-3.7/src/buffer.o -Wno-write-strings
In file included from /usr/include/sql.h:19,
from src/pyodbc.h:56,
from src/buffer.cpp:12:
/usr/include/sqltypes.h:56:10: fatal error: unixodbc.h: No such file or directory
56 | #include "unixodbc.h"
| ^~~~~~~~~~~~
compilation terminated.
error: command 'gcc' failed with exit status 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyodbc I guess changes for this PR have led to rebuild image almost from scratch (without caches) and get Same as described here: |
|
In the morning I think how to fix this sometimes "just wait" it is a better tactic 🤣 |
f875a0f
to
0d0f6f2
Compare
Oh absolutely. But you never know when. |
New test for collection of flaky tests: if job_run_status in DbtCloudJobRunStatus.TERMINAL_STATUSES.value:
assert mock_get_job_run.call_count == 1
else:
# When the job run status is not in a terminal status or "Success", the operator will
# continue to call ``get_job_run()`` until a ``timeout`` number of seconds has passed
# (3 seconds for this test). Therefore, there should be 4 calls of this function: one
# initially and 3 for each check done at a 1 second interval.
> assert mock_get_job_run.call_count == 4
E AssertionError: assert 2 == 4
E + where 2 = <MagicMock name='get_job_run' id='140182627981488'>.call_count |
Well, this not yet a problem on our |
0d0f6f2
to
88fde16
Compare
And finally it is green! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fingers crossed 🤞
Shall we? Or should be better that someone else look on this changes? |
And I also run some tests with ❯ breeze shell --backend mysql -M 8 --db-reset
...
root@1b371384f7b3:/opt/airflow# pytest tests/operators/test_generic_transfer.py::TestMySql
============================================================== test session starts ===============================================================
platform linux -- Python 3.7.16, pytest-7.2.1, pluggy-1.0.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow, configfile: pytest.ini
plugins: xdist-3.2.0, anyio-3.6.2, time-machine-2.9.0, instafail-0.4.2, httpx-0.21.3, timeouts-1.2.1, requests-mock-1.10.0, asyncio-0.20.3, capture-warnings-0.0.4, rerunfailures-11.1.1, cov-4.0.0
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 3 items
tests/operators/test_generic_transfer.py::TestMySql::test_mysql_to_mysql[mysqlclient] PASSED [ 33%]
tests/operators/test_generic_transfer.py::TestMySql::test_mysql_to_mysql[mysql-connector-python] PASSED [ 66%]
tests/operators/test_generic_transfer.py::TestMySql::test_mysql_to_mysql_replace PASSED [100%] |
Looks cool! |
The change apache#29519 added experimental MariaDB client support for MySQL for ARM images but we missed that in the changelog.
The change #29519 added experimental MariaDB client support for MySQL for ARM images but we missed that in the changelog.
According to https://mariadb.com/kb/en/mariadb-clientserver-tcp-protocol/ MariaDB client use the same protocol as use MySQL.
This PoC use binaries which included in standard Debian repository only for ARM64 images, AMD64/x86_64 still use Oracle MySQL repository, see all supported options
Airflow starts almost fine in breeze with MySQL 8.0 backend, except:
Some additionals: