Added `python.DetectInterpreters` and other utils #805

nfx · 2023-09-26T11:31:28Z

This PR adds a few utilities related to Python interpreter detection:

python.DetectInterpreters to detect all Python versions available in $PATH by executing every matched binary name with --version flag.
python.DetectVirtualEnvPath to detect if there's any child virtual environment in src directory
python.DetectExecutable to detect if there's python3 installed either by which python3 command or by calling python.DetectInterpreters().AtLeast("v3.8")

To be merged after #804, as one of the steps to get #637 in, as previously discussed.

nfx · 2023-09-26T11:37:03Z

Optionally, bca2bc2 could be reverted

hamza-db · 2023-09-26T11:59:45Z

one thing that can be nice to have.
is not not run any python binary that sits in a writable folder by all users.
this is mainly to avoid breaking the security model on multi-user system .. I know if the PATH is pointing somewhere untrusted it is the user fault. but we can help here

hamza-db · 2023-09-26T12:01:29Z

test test :
give a binary 777 and we shouldn't run it if it is 777 or inside a folder that is 777
but the main thing we check for for example in linux is : o=w

nfx · 2023-09-26T12:07:07Z

Windows tests fail:

TestFilteringInterpreters
TestDetectFailsNoMinimalVersion
TestDetectsViaListing

need to re-test this from a windows VM or codespace.

andrewnester · 2023-09-26T12:16:26Z

Overall, LGTM. What's unclear is why do we need this change exactly? Why do we need to know all Python interpreters available in the PATH?

nfx · 2023-09-26T13:05:41Z

@andrewnester two cases:

a lot of dev environments have more than one Python interpreter (like our dev machines at Databricks) and frequently it's unclear/non-usable to get the right one consistently
in labs, we have projects that do require python3.10, and we want to fail the installation if a customer has only 3.7 or 3.6 installed.

libs/python/detect.go

libs/python/interpreters.go

libs/python/venv.go

nfx · 2023-10-02T09:50:17Z

Will probably fall back to giving recommendation to install official python binaries instead of ms-windows-store...

nfx · 2023-10-02T10:01:15Z

And Python interpreters from Microsoft Store are detected:

nfx · 2023-10-02T10:09:20Z

users have to make sure that every installation is in PATH:

nfx · 2023-10-02T10:19:46Z

and we do support a wild mix of different python interpreter versions from Microsoft Store and Official Python downloads:

nfx · 2023-10-02T11:00:49Z

Stock Azure Windows VM is having all the important folders writeable as returned by os.Stat(dir).Mode().Perm():

see also: https://github.com/golang/go/blob/master/src/os/exec/lp_windows.go#L134-L206

hamza-db · 2023-10-02T13:50:50Z

I think the writability test was a "nice to have" measure but not a "must have".
so we can skip the checks if the Python setup is not secure by default on Windows.

pietern

Nice work. This will make working with Python on Windows much, much easier.

libs/python/detect_unix_test.go

libs/python/detect_win_test.go

libs/python/interpreters.go

libs/python/interpreters_unix_test.go

libs/python/interpreters_win_test.go

This PR adds a few utilities related to Python interpreter detection: - `python.DetectInterpreters` to detect all Python versions available in `$PATH` by executing every matched binary name with `--version` flag. - `python.DetectVirtualEnvPath` to detect if there's any child virtual environment in `src` directory - `python.DetectExecutable` to detect if there's python3 installed either by `which python3` command or by calling `python.DetectInterpreters().AtLeast("v3.8")` Code coverage is 95% Remove unused code some comments don't execute from world-writeable locations more tests make things work on windows shuffle some code around

andrewnester · 2023-10-03T13:41:18Z

Fixes #818

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

This PR adds a few utilities related to Python interpreter detection: - `python.DetectInterpreters` to detect all Python versions available in `$PATH` by executing every matched binary name with `--version` flag. - `python.DetectVirtualEnvPath` to detect if there's any child virtual environment in `src` directory - `python.DetectExecutable` to detect if there's python3 installed either by `which python3` command or by calling `python.DetectInterpreters().AtLeast("v3.8")` To be merged after #804, as one of the steps to get #637 in, as previously discussed.

CLI: * Refactor change computation for sync ([#785](#785)). Bundles: * Allow digits in the generated short name ([#820](#820)). * Emit an error when incompatible all purpose cluster used with Python wheel tasks ([#823](#823)). * Use normalized short name for tag value in development mode ([#821](#821)). * Added `python.DetectInterpreters` and other utils ([#805](#805)). * Mark artifacts properties as optional ([#834](#834)). * Added support for glob patterns in pipeline libraries section ([#833](#833)). Internal: * Run tests to verify backend tag validation behavior ([#814](#814)). * Library to validate and normalize cloud specific tags ([#819](#819)). * Added test to submit and run various Python tasks on multiple DBR versions ([#806](#806)). * Create a release PR in setup-cli repo on tag push ([#827](#827)). API Changes: * Changed `databricks account metastore-assignments list` command to return . * Changed `databricks jobs cancel-all-runs` command with new required argument order. * Added `databricks account o-auth-published-apps` command group. * Changed `databricks serving-endpoints query` command . New request type is . * Added `databricks serving-endpoints patch` command. * Added `databricks credentials-manager` command group. * Added `databricks settings` command group. * Changed `databricks clean-rooms list` command to require request of . * Changed `databricks statement-execution execute-statement` command with new required argument order. OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02) Dependency updates: * Bump github.com/google/uuid from 1.3.0 to 1.3.1 ([#825](#825)). * Updated Go SDK to 0.22.0 ([#831](#831)).

nfx requested review from pietern and andrewnester September 26, 2023 11:31

nfx force-pushed the libs/python branch from bca2bc2 to 703cb48 Compare September 26, 2023 11:40

nfx added do-not-merge and removed do-not-merge labels Sep 26, 2023

pietern reviewed Sep 26, 2023

View reviewed changes

libs/python/detect.go Show resolved Hide resolved

libs/python/detect.go Outdated Show resolved Hide resolved

libs/python/interpreters.go Outdated Show resolved Hide resolved

libs/python/interpreters.go Outdated Show resolved Hide resolved

libs/python/venv.go Show resolved Hide resolved

nfx mentioned this pull request Sep 27, 2023

Encountering an error when running ./install.sh databrickslabs/ucx#281

Closed

Base automatically changed from libs/process to main September 27, 2023 09:11

nfx force-pushed the libs/python branch 2 times, most recently from 8ed14e6 to 3063818 Compare September 29, 2023 16:51

nfx requested a review from pietern October 2, 2023 11:38

pietern reviewed Oct 3, 2023

View reviewed changes

nfx requested a review from pietern October 3, 2023 09:27

nfx added the Enhancement New feature or request label Oct 3, 2023

pietern approved these changes Oct 3, 2023

View reviewed changes

nfx enabled auto-merge October 3, 2023 10:38

nfx force-pushed the libs/python branch from c2d6bdb to 32a5001 Compare October 3, 2023 10:39

nfx changed the title ~~Added python.DetectInterpreters other utils~~ Added python.DetectInterpreters and other utils Oct 3, 2023

nfx added this pull request to the merge queue Oct 3, 2023

Merged via the queue into main with commit 7d0f170 Oct 3, 2023

nfx deleted the libs/python branch October 3, 2023 10:53

This was referenced Oct 3, 2023

Asset Bundles deploy does not work on windows #818

Closed

Release v0.207.0 #839

Merged

JCZuurmond mentioned this pull request Dec 8, 2023

Configure which Python interpreter assest bundels uses #1048

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added `python.DetectInterpreters` and other utils #805

Added `python.DetectInterpreters` and other utils #805

nfx commented Sep 26, 2023 •

edited

Loading

nfx commented Sep 26, 2023

hamza-db commented Sep 26, 2023

hamza-db commented Sep 26, 2023

nfx commented Sep 26, 2023

andrewnester commented Sep 26, 2023

nfx commented Sep 26, 2023

nfx commented Oct 2, 2023 •

edited

Loading

nfx commented Oct 2, 2023

nfx commented Oct 2, 2023

nfx commented Oct 2, 2023

nfx commented Oct 2, 2023 •

edited

Loading

hamza-db commented Oct 2, 2023 •

edited

Loading

pietern left a comment

andrewnester commented Oct 3, 2023

Added python.DetectInterpreters and other utils #805

Added python.DetectInterpreters and other utils #805

Conversation

nfx commented Sep 26, 2023 • edited Loading

nfx commented Sep 26, 2023

hamza-db commented Sep 26, 2023

hamza-db commented Sep 26, 2023

nfx commented Sep 26, 2023

andrewnester commented Sep 26, 2023

nfx commented Sep 26, 2023

nfx commented Oct 2, 2023 • edited Loading

nfx commented Oct 2, 2023

nfx commented Oct 2, 2023

nfx commented Oct 2, 2023

nfx commented Oct 2, 2023 • edited Loading

hamza-db commented Oct 2, 2023 • edited Loading

pietern left a comment

Choose a reason for hiding this comment

andrewnester commented Oct 3, 2023

Added `python.DetectInterpreters` and other utils #805

Added `python.DetectInterpreters` and other utils #805

nfx commented Sep 26, 2023 •

edited

Loading

nfx commented Oct 2, 2023 •

edited

Loading

nfx commented Oct 2, 2023 •

edited

Loading

hamza-db commented Oct 2, 2023 •

edited

Loading