Allow selection of python version when starting managed Galaxy #874

mvdbeek · 2018-09-29T19:42:33Z

Make sure python 3 is on PATH and enjoy planemo {test, serve} --galaxy_branch release_18.09 --galaxy_python_version 3.

mvdbeek · 2018-09-29T20:17:30Z

Hmm, this is Paste failing now due to deleting from a dict while iterating over it. The fix is simple and I've done it locally, but Paste looks pretty much unmaintained.

mvdbeek · 2018-10-01T08:07:30Z

Hmm, so for some reason when installing dependencies by running common_startup.sh we get the wrong shebang (it points to the python in the virtualenv in which planemo was installed), which is a problem for the gunicorn entrypoint. I haven't figured out what's going on there, seems very strange.

mvdbeek · 2018-10-01T09:03:28Z

So I'm having __PYVENV_LAUNCHER__ set in my python 3 virtualenv. This gets propagated to new commands lauched by planemo because in galaxy-lib we create a new env with os.environ.copy and update it. Just setting env['__PYVENV_LAUNCHER__'] = '' seems to solve the issue. Seems this problem might occur more frequently in the future based on pypa/virtualenv#845 (comment)

galaxyproject#874 (comment)

common_prefix fails with set.

mvdbeek · 2018-10-06T14:40:56Z

yay, all green!

.travis.yml

nsoranzo · 2018-10-17T16:57:58Z

So I'm having __PYVENV_LAUNCHER__ set in my python 3 virtualenv. This gets propagated to new commands lauched by planemo because in galaxy-lib we create a new env with os.environ.copy and update it. Just setting env['__PYVENV_LAUNCHER__'] = '' seems to solve the issue. Seems this problem might occur more frequently in the future based on pypa/virtualenv#845 (comment)

@mvdbeek A bugfix for this has been proposed in python/cpython#9516

nsoranzo · 2018-10-17T16:35:42Z

planemo/galaxy/config.py

@@ -400,7 +401,7 @@ def config_join(*args):
        template_args = dict(
            port=port,
            host=kwds.get("host", "127.0.0.1"),
-            server_name=server_name,
+            server_name=server_name if float(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) < 3 else 'main',  # gunicorn needs main


What about:

server_name=server_name if parse_version(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) < parse_version('3') else 'main', # gunicorn needs main

after adding from pkg_resources import parse_version at the top.

nsoranzo · 2018-10-17T16:38:58Z

planemo/galaxy/config.py

@@ -941,6 +942,14 @@ def startup_command(self, ctx, **kwds):
            run_script += " --server-name %s" % shlex_quote(self.server_name)
        server_ini = os.path.join(self.config_directory, "galaxy.ini")
        self.env["GALAXY_CONFIG_FILE"] = server_ini
+        if float(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) >= 3:


Suggested change

if float(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) >= 3:

if parse_version(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) >= parse_version('3'):

planemo/galaxy/config.py

nsoranzo · 2018-10-18T16:35:38Z

planemo/galaxy/config.py

@@ -1240,6 +1254,7 @@ def _install_with_command(ctx, config_directory, command, env, kwds):
    else:
        pip_install_command = ""


I just noticed that, since commit 9867e56, pip_installs and pip_install_command could be removed, if desired.

nsoranzo · 2018-10-18T16:54:57Z

planemo/io.py

@@ -182,6 +182,7 @@ def kill_pid_file(pid_file):

    pid = int(open(pid_file, "r").read())
    kill_posix(pid)
+    os.unlink(pid_file)


This should probably be in a try/except Exception: pass, since the killed process can remove the pid file before exiting, in which case os.remove would raise OSError or FileNotFoundError.

Good point, thanks!

nsoranzo · 2018-10-18T18:33:42Z

Thanks @mvdbeek!

Should we enable this for tools-iuc testing? 😈

mvdbeek · 2018-10-18T19:53:59Z

Sure! We can start with a blacklist and work our way down like we did with condafying tools.

Otherwise the `pid_file` is set to the wrong filename, preventing `LocalGalaxyConfig.kill()` from working when running Galaxy under Python 3. Follow-up to galaxyproject#874 .

mvdbeek added the enhancement label Sep 29, 2018

mvdbeek force-pushed the allow_python_version_selection branch from 302b226 to f1b9c9d Compare September 29, 2018 19:43

mvdbeek force-pushed the allow_python_version_selection branch 9 times, most recently from 4e963af to 7b9a4b9 Compare September 29, 2018 21:45

mvdbeek closed this Sep 29, 2018

mvdbeek reopened this Sep 29, 2018

mvdbeek force-pushed the allow_python_version_selection branch 7 times, most recently from 6f4d748 to a4d78b9 Compare September 29, 2018 22:04

mvdbeek mentioned this pull request Sep 30, 2018

[18.09] Allow starting Galaxy under gunicorn when using galaxy.ini galaxyproject/galaxy#6805

Merged

mvdbeek added a commit to mvdbeek/planemo that referenced this pull request Oct 1, 2018

Fix wrong shebang on python3 virtualenv

2334277

galaxyproject#874 (comment)

mvdbeek force-pushed the allow_python_version_selection branch 2 times, most recently from d3b676e to 2394309 Compare October 1, 2018 11:42

mvdbeek added a commit to mvdbeek/planemo that referenced this pull request Oct 1, 2018

Fix wrong shebang on python3 virtualenv

09732ac

galaxyproject#874 (comment)

mvdbeek force-pushed the allow_python_version_selection branch from 2394309 to 35eb2c1 Compare October 1, 2018 11:46

mvdbeek added a commit to mvdbeek/planemo that referenced this pull request Oct 1, 2018

Fix wrong shebang on python3 virtualenv

3f16ee6

galaxyproject#874 (comment)

mvdbeek added 2 commits October 1, 2018 14:51

Temporarily test against my branch

9cef47e

Return cmd list

5675c8e

mvdbeek force-pushed the allow_python_version_selection branch 2 times, most recently from 48d9807 to fafb98b Compare October 1, 2018 21:41

Fix tests?

c784925

mvdbeek force-pushed the allow_python_version_selection branch from fafb98b to c784925 Compare October 6, 2018 08:48

mvdbeek added 5 commits October 6, 2018 10:54

Fix linting throughout codebase

57578e3

Click seems to not return -1 exit codes anymore ?

37f8588

Remove PID file after killing instance

a402de1

Don't test python3 on python2 and vice versa

a7824ee

python3 shed2tap fixes

f3ad9df

common_prefix fails with set.

mvdbeek mentioned this pull request Oct 14, 2018

Conda tutorial fixes. Enable --conda_use_local option for planemo test #876

Merged

Remove extra empty line

6e4d964

nsoranzo reviewed Oct 16, 2018

View reviewed changes

.travis.yml Show resolved Hide resolved

Restore py36-lint and move up py36-quick

dbfec70

mvdbeek force-pushed the allow_python_version_selection branch from 05f66a8 to dbfec70 Compare October 16, 2018 13:01

nsoranzo reviewed Oct 17, 2018

View reviewed changes

Use parse_version for version handling

496ded7

mvdbeek commented Oct 18, 2018

View reviewed changes

planemo/galaxy/config.py Show resolved Hide resolved

Simply if condition

d540aed

nsoranzo reviewed Oct 18, 2018

View reviewed changes

mvdbeek added 2 commits October 18, 2018 19:33

Drop unused pip_install_command

d99275b

Safe removal of pid file

fd1a9e8

nsoranzo merged commit 7ca2b40 into galaxyproject:master Oct 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow selection of python version when starting managed Galaxy #874

Allow selection of python version when starting managed Galaxy #874

mvdbeek commented Sep 29, 2018

mvdbeek commented Sep 29, 2018

mvdbeek commented Oct 1, 2018

mvdbeek commented Oct 1, 2018 •

edited by nsoranzo

Loading

mvdbeek commented Oct 6, 2018

nsoranzo commented Oct 17, 2018

nsoranzo Oct 17, 2018

nsoranzo Oct 17, 2018

nsoranzo Oct 18, 2018

nsoranzo Oct 18, 2018

mvdbeek Oct 18, 2018

nsoranzo commented Oct 18, 2018

mvdbeek commented Oct 18, 2018

	if float(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) >= 3:
	if parse_version(kwds.get('galaxy_python_version', DEFAULT_PYTHON_VERSION)) >= parse_version('3'):

		@@ -1240,6 +1254,7 @@ def _install_with_command(ctx, config_directory, command, env, kwds):
		else:
		pip_install_command = ""

Allow selection of python version when starting managed Galaxy #874

Allow selection of python version when starting managed Galaxy #874

Conversation

mvdbeek commented Sep 29, 2018

mvdbeek commented Sep 29, 2018

mvdbeek commented Oct 1, 2018

mvdbeek commented Oct 1, 2018 • edited by nsoranzo Loading

mvdbeek commented Oct 6, 2018

nsoranzo commented Oct 17, 2018

nsoranzo Oct 17, 2018

Choose a reason for hiding this comment

nsoranzo Oct 17, 2018

Choose a reason for hiding this comment

nsoranzo Oct 18, 2018

Choose a reason for hiding this comment

nsoranzo Oct 18, 2018

Choose a reason for hiding this comment

mvdbeek Oct 18, 2018

Choose a reason for hiding this comment

nsoranzo commented Oct 18, 2018

mvdbeek commented Oct 18, 2018

mvdbeek commented Oct 1, 2018 •

edited by nsoranzo

Loading