Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ctypes (libffi) is no longer required on linux #455

Merged
merged 1 commit into from
Sep 27, 2024

Conversation

epmog
Copy link
Contributor

@epmog epmog commented Sep 26, 2024

What was the problem/requirement? (What/Why)

A recent submitter installer release for Linux operating systems had a deadline executable that when run with anything would have a stack trace that looks like:

$ deadline help
Traceback (most recent call last):
  File "client/cli/deadline_cli_main.py", line 19, in <module>
  File "client/cli/deadline_cli_main.py", line 10, in main
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "client/cli/__init__.py", line 5, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "client/cli/deadline_dev_gui_main.py", line 8, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "client/config/__init__.py", line 19, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "client/config/config_file.py", line 25, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/models.py", line 14, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/asset_manifests/__init__.py", line 3, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/asset_manifests/base_manifest.py", line 10, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/asset_manifests/hash_algorithms.py", line 9, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/exceptions.py", line 10, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/progress_tracker.py", line 13, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "job_attachments/_utils.py", line 8, in <module>
  File "PyInstaller/loader/pyimod02_importers.py", line 385, in exec_module
  File "ctypes/__init__.py", line 8, in <module>
ImportError: libffi.so.6: cannot open shared object file: No such file or directory
[3159774] Failed to execute script 'deadline_cli_main' due to unhandled exception!

Effectively, a new system library dependency was now required by the deadline executable since we only allowlist 3 libraries to bundle with the pyinstaller build. Thus, Linux distros that use the new version of libffi (libffi.so.8) would be unable to run the executable.

This was inadvertently introduced in a recent innocuous change when a ctypes import was added to the top of a file instead of being behind an is_windows check.

What was the solution? (How)

There's some forward looking items we can do to build on each operating system we support and choose to install the right executable for the distro. In addition to automation around testing the installer automatically on those various platforms

However, this quick fix is to just remove the newly added libffi dependency for Linux. We do this everywhere else we use ctypes, so we're just pattern-matching here.

What is the impact of this change?

The submitter installer builds a deadline executable that works on newer linux distros.

How was this change tested?

See DEVELOPMENT.md for information on running tests.

hatch run fmt
hatch build
hatch run lint
hatch run test
  • Have you run the unit tests?
    • yes
  • Have you run the integration tests?
    • yes
test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_input_files_all_assets_in_cas[2023-03-03] 
[gw0] [  5%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_input_files_all_assets_in_cas[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_no_job_attachment_settings_in_job[2023-03-03] 
[gw0] [ 11%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_no_job_attachment_settings_in_job[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_no_job_attachment_s3_settings[2023-03-03] 
[gw0] [ 17%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_no_job_attachment_s3_settings[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_with_symlink[2023-03-03] 
[gw0] [ 23%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_with_symlink[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_with_step_dependencies[2023-03-03] 
[gw0] [ 29%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_with_step_dependencies[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_with_job_id_step_id_task_id_and_download_directory[2023-03-03] 
[gw0] [ 35%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_with_job_id_step_id_task_id_and_download_directory[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_with_job_id_step_id_and_download_directory[2023-03-03] 
[gw0] [ 41%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_with_job_id_step_id_and_download_directory[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_with_job_id_and_download_directory[2023-03-03] 
[gw0] [ 47%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_with_job_id_and_download_directory[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_input_files_no_download_paths[2023-03-03] 
[gw0] [ 52%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_input_files_no_download_paths[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_no_inputs[2023-03-03] 
[gw0] [ 58%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_no_inputs[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_bucket_wrong_account[2023-03-03] 
[gw0] [ 64%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_bucket_wrong_account[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_bucket_wrong_account[2023-03-03] 
[gw0] [ 70%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_bucket_wrong_account[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_bucket_wrong_account[2023-03-03] 
[gw0] [ 76%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_bucket_wrong_account[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_bucket_wrong_account[2023-03-03] 
[gw0] [ 82%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_bucket_wrong_account[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_windows_max_file_path_length_exception[2023-03-03] 
[gw0] [ 88%] SKIPPED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_windows_max_file_path_length_exception[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_no_outputs_dir[2023-03-03] 
[gw0] [ 94%] PASSED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_no_outputs_dir[2023-03-03] 
test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_windows_file_path_UNC[2023-03-03] 
[gw0] [100%] SKIPPED test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_windows_file_path_UNC[2023-03-03] 

===================================================================== warnings summary =====================================================================
test/integ/deadline_job_attachments/test_job_attachments.py:1287
  /Users/morgane/dev/github/deadline-cloud/test/integ/deadline_job_attachments/test_job_attachments.py:1287: PytestUnknownMarkWarning: Unknown pytest.mark.cross_account - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.cross_account

test/integ/deadline_job_attachments/test_job_attachments.py:1337
  /Users/morgane/dev/github/deadline-cloud/test/integ/deadline_job_attachments/test_job_attachments.py:1337: PytestUnknownMarkWarning: Unknown pytest.mark.cross_account - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.cross_account

test/integ/deadline_job_attachments/test_job_attachments.py:1386
  /Users/morgane/dev/github/deadline-cloud/test/integ/deadline_job_attachments/test_job_attachments.py:1386: PytestUnknownMarkWarning: Unknown pytest.mark.cross_account - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.cross_account

test/integ/deadline_job_attachments/test_job_attachments.py:1470
  /Users/morgane/dev/github/deadline-cloud/test/integ/deadline_job_attachments/test_job_attachments.py:1470: PytestUnknownMarkWarning: Unknown pytest.mark.cross_account - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.cross_account

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================== slowest 5 durations ====================================================================
6.31s setup    test/integ/deadline_job_attachments/test_job_attachments.py::test_upload_input_files_all_assets_in_cas[2023-03-03]
3.07s setup    test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_inputs_with_step_dependencies[2023-03-03]
2.45s teardown test/integ/deadline_job_attachments/test_job_attachments.py::test_download_outputs_windows_file_path_UNC[2023-03-03]
2.14s setup    test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_no_job_attachment_settings_in_job[2023-03-03]
1.73s setup    test/integ/deadline_job_attachments/test_job_attachments.py::test_sync_outputs_with_symlink[2023-03-03]
======================================================== 15 passed, 2 skipped, 4 warnings in 30.40s ========================================================
  • Have you made changes to the download or asset_sync modules? If so, then it is highly recommended
    that you ensure that the docker-based unit tests pass.
    • yes
============================= slowest 5 durations ==============================
1.20s setup    test/unit/deadline_job_attachments/test_download.py::TestFullDownload::test_download_job_output[2023-03-03]
1.11s setup    test/unit/deadline_job_attachments/test_download.py::TestFullDownload::test_get_job_outputs_paths_by_asset_root_when_no_asset_root[2023-03-03]
1.08s setup    test/unit/deadline_job_attachments/test_download.py::TestFullDownload::test_OutputDownloader_get_output_paths_by_root[2023-03-03]
1.08s setup    test/unit/deadline_job_attachments/test_download.py::TestFullDownload::test_download_files_from_manifests_with_fs_permission_settings_posix[2023-03-03]
1.08s setup    test/unit/deadline_job_attachments/test_download.py::TestFullDownload::test_download_task_output[2023-03-03]
======================== 32 passed, 9 skipped in 13.45s ========================

Was this change documented?

N/A

Is this a breaking change?

Nope

Does this change impact security?

Nope


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@epmog epmog added bug Something isn't working dependencies Pull requests that update a dependency file labels Sep 26, 2024
@epmog epmog force-pushed the make_ctypes_windows_only branch 4 times, most recently from 0ef248a to bd4d1f5 Compare September 26, 2024 17:41
leongdl
leongdl previously approved these changes Sep 26, 2024
Copy link
Contributor

@leongdl leongdl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fill out the PR template?

Copy link

@epmog epmog marked this pull request as ready for review September 26, 2024 21:43
@epmog epmog requested a review from a team as a code owner September 26, 2024 21:43
@@ -94,13 +93,16 @@ def _is_relative_to(path1: Union[Path, str], path2: Union[Path, str]) -> bool:


def _is_windows_file_path_limit() -> bool:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Not from this PR, but this name appears to be logically reversed from the function it calls. A name like _windows_are_long_paths_enabled, matching the Windows API call very closely, could be more clear.

@epmog epmog merged commit 6446b66 into aws-deadline:mainline Sep 27, 2024
18 checks passed
@epmog epmog deleted the make_ctypes_windows_only branch September 27, 2024 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants