Build tiledbvcf-py on Windows #507
Conversation
|
The following patch will allow that final setup.py to complete without errors, but is not a correct solution (won't be appropriate for any msys build if there is, or ever will be, one) |
@dhoke4tdb Thanks! I'm testing it locally now
It's my understanding that we only intend to support an MSVC build of libtiledbvcf and tiledbvcf-py, at least for now |
|
Alright, I have it working locally! python -c "import tiledbvcf; print(tiledbvcf.version)"
rem 0.22.1.dev11+dirty |
|
@jdblischak here is a patch that has a few additional changes for msvc, this is to be applied to the commit -before- you applied my earlier patch... |
It's no problem to do an interactive rebase to update your patch. I'm working on other things at the moment, but I'll return to this first thing Monday |
|
@dhoke4tdb I rebased and applied your updated patch |
|
I also created a GitHub Actions workflow to test the build on Windows. The good news is that we can now build and install the Python client on Windows. The bad news is that both locally and on the GitHub Actions runner, the Python client is non-functional. When I run Windows fatal exception: access violationAnd when I try to simply read an array, Python immediately crashes without any error message. The exit code is -1073741819 (note that the pytest run also exits with the same exit code). Here is the quick python test code: import tiledbvcf
ds = tiledbvcf.Dataset(
uri = "libtiledbvcf/test/inputs/arrays/v3/ingested_2samples",
mode = "r"
)And here is what happens when I run it: python ci\test-tiledbvcf-py.py
echo %ERRORLEVEL%
rem -1073741819 |
|
Also note that libtiledbvcf itself appears to be functional, so the problem is with the Python client tiledbvcf stat --uri libtiledbvcf\test\inputs\arrays\v3\ingested_2samples
rem Statistics for dataset 'libtiledbvcf\test\inputs\arrays\v3\ingested_2samples':
rem - Version: 3
rem - Row tile extent: 10
rem - Tile capacity: 10,000
rem - Anchor gap: 1,000
rem - Number of samples: 2
rem - Extracted attributes: none
tiledbvcf list --uri libtiledbvcf\test\inputs\arrays\v3\ingested_2samples
rem HG00280
rem HG01762 |
|
The 'top' problem is that the tiledbvcf.exe winds up being linked/used as a .dll by the libtiledbvcf extension module instead of tiledbvcf.dll. |
|
Thanks David! After applying your patch, I have things working well locally. My simple examples runs fine: import tiledbvcf
ds = tiledbvcf.Dataset(
uri = "libtiledbvcf/test/inputs/arrays/v3/ingested_2samples",
mode = "r"
)
[[ed for my future reference:
x = ds.read(attrs=["sample_name"])]
]]
print(x["sample_name"].unique())
## ['HG00280' 'HG01762']And the tests are passing. The only failure was one test tries to call the cli tool bcftools, which isn't available as a conda package for Windows (if someone requests it on Windows, then we'd have to go through the same process as we did for htslib; bcftools install instructions for msys2). Below are the truncated test results pytest apis\python\tests\test_tiledbvcf.py
rem ================================================= test session starts =================================================
rem platform win32 -- Python 3.11.0, pytest-7.2.2, pluggy-1.0.0
rem rootdir: C:\Users\john\repos\TileDB-VCF\apis\python
rem collected 39 items
rem
rem apis\python\tests\test_tiledbvcf.py .........s............s.............F.. [100%]
rem
rem ------------------------------------------------ Captured stderr call -------------------------------------------------
rem 'bcftools' is not recognized as an internal or external command,
rem operable program or batch file.
rem
rem =============================================== short test summary info ===============================================
rem FAILED apis\python\tests\test_tiledbvcf.py::test_ingest_with_stats - subprocess.CalledProcessError: Command 'bcftools view --no-version -Oz -o C:\Users\john\AppData\Local\Temp\pytest-o...
rem =========================== 1 failed, 36 passed, 2 skipped, 2 warnings in 110.96s (0:01:50) ===========================
pytest apis\python\tests\test_dask.py
rem ================================================= test session starts =================================================
rem platform win32 -- Python 3.11.0, pytest-7.2.2, pluggy-1.0.0
rem rootdir: C:\Users\john\repos\TileDB-VCF\apis\python
rem collected 4 items
rem
rem apis\python\tests\test_dask.py .... [100%]
rem
rem ============================================ 4 passed, 4 warnings in 5.33s ============================================
|
|
@ihnorton This PR is ready for review. The Python client now builds on Windows. Some notes:
|
|
Also, since I'm a first-time contributor to this repo, my GitHub Action workflow has to be re-approved |
|
My latest commit passed in the build from my fork |
|
@awenocur Could you please approve the GitHub Actions workflow to run and review the PR for any remaining items you would like me to address? |
This is a follow-up to PR #484 from @dhoke4tdb. That merged PR enabled building libtiledbvcf on Windows by linking against a customized build of htslib on msys2. The goal of this PR is to enable building the Python package on Windows as well.
Current status: Python client builds on Windows, tests pass (link to passing build on my fork)
Local installation
Steps to build locally
Checkout my branch
Install the conda env:
Build libtiledbvcf
Build tiledbvcf-py
Run tests
Warning messages from tests
The warning messages from running the tests on Windows are different from the Azure build. Copy-pasting them below in case they are of concern
pytest warnings on Windows