[10sp] Make test (and other python code) Python3 compatible #20

kostja · 2016-01-07T19:58:33Z

Python3 is the next major version and some Linux distros (e.g. Linux Arch) are already use it as a default. It makes sense to make the code python3 compatible. In theory conversion should be simple especially if you use tools like py2to3.

Moved from tarantool/tarantool#56

anatol · 2020-02-10T18:11:48Z

Python2 is EOL. Python3 is a default version at major OS distributions. Could you please prioritize this effort and make tarantool project python2-free?

ligurio · 2020-11-20T15:52:16Z

I would try to decompose ticket to subtasks:

remove relative module imports
fix module imports for modules changed in Python 3 (for example ConfigParser)
fix printing to make it compatible with 2 and 3 versions
support Python 3 in Tarantool tests (Support Python 3 in tests tarantool#5538)

Part of #20

In a Python 3 'print' becomes a function, see [1]. Patch makes print calls compatible with Python 3. 1. https://docs.python.org/3/whatsnew/3.0.html#print-is-a-function Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

Closes #20

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

Closes #20

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

Closes #20

Part of #20

Unlike Python 2, Python 3 does not treat `import bar` from a module in a package `foo` as importing of `foo.bar`. Absolute import that is relative to a test-run root (like `import lib.utils`) still work, because `sys.path` contains a script directory (after symlinks resolution) on both Python versions. https://www.python.org/dev/peps/pep-0328/ Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

What's New In Python 3.0 [1]: The function attributes named func_X have been renamed to use the __X__ form, freeing up these names in the function attribute namespace for user-defined attributes. To wit, func_closure, func_code, func_defaults, func_dict, func_doc, func_globals, func_name were renamed to __closure__, __code__, __defaults__, __dict__, __doc__, __globals__, __name__, respectively. 1. https://docs.python.org/3/whatsnew/3.0.html#operators-and-special-methods Part of #20

Fix Tarantool connection liveness and box-py/call.test.py test. Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

gevent 1.2 and below does not support Python 3.7 and above, see [1]. 1. gevent/gevent#1297 Part of #20

In Python 3 start method 'spawn' in multiprocessing module becomes default on Mac OS [1]. The 'spawn' method causes re-execution of some code, which is already executed in the main process. At least it is seen on the lib/__init__.py code, which removes the 'var' directory. Some other code may have side effects too, it requires investigation. The method also requires object serialization that doesn't work when objects use lambdas, whose for example used in class TestSuite (lib/test_suite.py). The latter problem is easy to fix, but the former looks more fundamental. So we stick to the 'fork' method now. The new start method is available on Python 3 only: Traceback (most recent call last): File "../../test/test-run.py", line 227, in <module> multiprocessing.set_start_method('fork') AttributeError: 'module' object has no attribute 'set_start_method' 1. https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods Fixes #265 Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

In Python 2 default string type (`<str>`) is a binary string, non-unicode. We receive data from a socket, from a Popen stream, from a file as a string and operate on those strings without any conversions. Python 3 draws a line here. We usually operate on unicode strings in the code (because this is the default string type, `<str>`), but receive bytes from a socket and a Popen stream. We can use unicode or binary streams for files (unicode by default[^1]). This commit decouples bytes and strings. In most cases it means that we convert data from bytes to a string after receiving from a socket / Popen stream and convert it back from a string to bytes before writting to a socket. Those operations are no-op on Python 2. So, the general rule for our APIs is to accept and return `<str>` disregarding Python version. Not `<bytes>`, not `<unicode>`. The only non-trivial change is around `FilteredStream` and writes into `sys.stdout`. The `FilteredStream` instance replaces `sys.stdout` during execution of a test, so it should follow the usual convention and accept `<str>` in the `write()` method. This is both intuitive and necessary, because `*.py` tests rely on `print('bla bla')` to write into a result file. However the stream should also accept `<bytes>`, because we have a unit test (`unit/json.test`), which produces a binary output, which does not conform UTF-8 encoding. The separate `write_bytes()` method was introduced for this sake. UnittestServer and AppServer write tests output as bytes directly, TarantoolServer rely on the usual string output. We also use bytes directly, when write from one stream to another one: in `app_server.py` for stderr (writting to a log file), in `tarantool_server.py` for log destination property (because it is the destination for Popen). [^1]: Technically it depends on a Python version and a system locale. There are situations, when default text file streams encoding is not 'utf-8'. They will be handled in the next commit. Part of #20 Co-authored-by: Sergey Bronnikov <[email protected]>

The problem: test files, result files contain UTF-8 symbols, which are out of ASCII range. Just `open(file, 'r').read()` without the encoding='utf-8' argument fails to decode them, when the default encoding for file text streams is not 'utf-8'. We meet this situation on Python 3.6.8 (provided by CentOS 7 and CentOS 8), when the POSIX locale is set (`LC_ALL=C`). The solution is described in the code comment: replace `open()` built-in function and always set `encoding='utf-8'`. That's hacky way, but it looks better than change every `open()` call across the code and don't forget to do that in all future code (and keep Python 2 compatibility in the mind). But maybe we'll revisit the approach later. There is another way to hack the `open()` behaviour that works for me on Python 3.6.8: | import _bootlocale | _bootlocale.getpreferredencoding = (lambda *args: 'utf8') However it leans on Python internals and looks less reliable than the implemented one. Part of #20

When Tarantool unit tests runs using test-run error with output like below may happen: [027] small/small_class.test [027] Test.run() received the following error: [027] Traceback (most recent call last): [027] File "/__w/tarantool/tarantool/test-run/lib/test.py", line 192, in run [027] self.execute(server) [027] File "/__w/tarantool/tarantool/test-run/lib/unittest_server.py", line 20, in execute [027] proc = Popen(execs, cwd=server.vardir, stdout=PIPE, stderr=STDOUT) [027] File "/usr/lib/python3.5/subprocess.py", line 676, in __init__ [027] restore_signals, start_new_session) [027] File "/usr/lib/python3.5/subprocess.py", line 1282, in _execute_child [027] raise child_exception_type(errno_num, err_msg) [027] FileNotFoundError: [Errno 2] No such file or directory: '../test/small/small_class.test' [027] [ fail ] The root cause of error is changed behaviour of Popen in Python 3 in comparison to Python 2. One should explicitly set path to a current work dir: Python 2 [1]: "If cwd is not None, the child’s current directory will be changed to cwd before it is executed. Note that this directory is not considered when searching the executable, so you can’t specify the program’s path relative to cwd." Python 3 [2]: "If cwd is not None, the function changes the working directory to cwd before executing the child. <...> In particular, the function looks for executable (or for the first item in args) relative to cwd if the executable path is a relative path." 1. https://docs.python.org/2/library/subprocess.html#subprocess.Popen 2. https://docs.python.org/3/library/subprocess.html#subprocess.Popen Part of #20

To make test box-py/call.test.py compatible with Python 2 and Python 3 it must have the same output when running with both versions of interpeter. With running on Python 2.7 output of called commands concluded to round brackets while on Python 3 it is not. [001] @@ -20,7 +20,7 @@ [001] - true [001] - null [001] ... [001] -call f1 () [001] +('call ', 'f1', ((),)) [001] - 'testing' [001] - 1 [001] - False To fix it print should use formatted string in print() call. Part of #20

Related issues and changes: - Support Python 3 in Tarantool test suite tarantool/tarantool#5538 - Python 3 in tarantool-python tarantool/tarantool-python#181 tarantool/tarantool-python#186 - Use Python 3 in a test infrastructure tarantool/tarantool#5652 Closes #20

Follows up tarantool/test-run#20

Patch follows up commit 395edeb ('python3: decouple bytes and strings') and it is a part of task with switching to Python 3. Follows up: #20

Since Python 3.6+ in json.loads() object that should be deserialized can be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32 [1]. Patch follows up commit 395edeb ('python3: decouple bytes and strings') and it is a part of task with switching to Python 3. 1. https://docs.python.org/3/library/json.html Follows up: #20

Python 2 support is EOL [1] and we should support to run our python tests using Python 3.x. test-run used as test-runner for memcached already supports Python 3, see [2]. Patch adds support of running tests with Python 3 without breaking compatibility with Python 2. Testing on CI switched to Python 3. 1. https://www.python.org/doc/sunset-python-2/ 2. tarantool/test-run#20 Closes #82

Python 2 support is EOL [1] and we should support to run our python tests using Python 3.x. test-run used as test-runner for memcached already supports Python 3, see [2]. Patch enable running tests using Python 3 on CI and removes test/var/memcached symlink. 1. https://www.python.org/doc/sunset-python-2/ 2. tarantool/test-run#20 Closes #82

kostja mentioned this issue Jan 7, 2016

Make test (and other python code) Python3 compatible tarantool/tarantool#56

Closed

rtsisyk added the feature A new functionality label Dec 16, 2016

Totktonada self-assigned this Sep 5, 2019

Totktonada mentioned this issue Sep 24, 2020

There is no python2 on OSX (it has python2.7) #15

Closed

Totktonada assigned ligurio Nov 20, 2020

ligurio added a commit that referenced this issue Nov 20, 2020

Enable using Python 3 in a project infrastructure

4726f10

Part of #20

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix path import lib/box_connection.py

b5c2d50

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix mode format in chmod() call

a701f61

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix import of multiprocessing module

506514c

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix configparser module import

0fabc06

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: convert byte to string

64ff31b

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: decode to utf8

8c128ed

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: replace xrange() with xrange()

20b4d1d

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

Enable using Python 3 in a project infrastructure

8a84b2f

Closes #20

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix path import lib/box_connection.py

b1d7ebd

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix import of StringIO module

b802dfe

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix mode format in chmod() call

bacd064

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix import of multiprocessing module

a06d84a

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: fix configparser module import

7f1f613

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: convert byte to string

24cd430

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: decode to utf8

cdc7d73

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: replace xrange() with xrange()

4d2350d

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

python3: using six to iteritems()

83497c6

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

Enable using Python 3 in a project infrastructure

cc98571

Closes #20

ligurio added a commit that referenced this issue Nov 21, 2020

python3: relative imports

c0e91a2

Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

ligurio added a commit that referenced this issue Nov 21, 2020

Enable using Python 3 in a project infrastructure

78cdef2

Closes #20

ligurio added a commit that referenced this issue Nov 21, 2020

Enable using Python 3 in a project infrastructure

76bd3d9

Closes #20

Totktonada pushed a commit that referenced this issue Mar 13, 2021

python3: conversion iteritems() -> items()

67be420

Part of #20

Totktonada added a commit that referenced this issue Mar 13, 2021

python3: use string byte literal

110f681

Fix Tarantool connection liveness and box-py/call.test.py test. Part of #20 Co-authored-by: Alexander Turenko <[email protected]>

Totktonada pushed a commit that referenced this issue Mar 13, 2021

python3: bump gevent version

e9e8bee

gevent 1.2 and below does not support Python 3.7 and above, see [1]. 1. gevent/gevent#1297 Part of #20

ligurio added a commit to tarantool/vshard that referenced this issue Mar 15, 2021

test-run bump version (Python 3 support)

d22eab8

Follows up tarantool/test-run#20

ligurio added a commit that referenced this issue Apr 30, 2021

lib: use bytes_to_str() in xlog_rows()

7c68241

Patch follows up commit 395edeb ('python3: decouple bytes and strings') and it is a part of task with switching to Python 3. Follows up: #20

ligurio added a commit that referenced this issue May 1, 2021

lib: use bytes_to_str() in xlog_rows()

140ea2e

Patch follows up commit 395edeb ('python3: decouple bytes and strings') and it is a part of task with switching to Python 3. Follows up: #20

ligurio mentioned this issue Dec 1, 2021

Update tests for Python 3 tarantool/memcached#82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[10sp] Make test (and other python code) Python3 compatible #20

[10sp] Make test (and other python code) Python3 compatible #20

kostja commented Jan 7, 2016

anatol commented Feb 10, 2020

ligurio commented Nov 20, 2020 •

edited

Loading

[10sp] Make test (and other python code) Python3 compatible #20

[10sp] Make test (and other python code) Python3 compatible #20

Comments

kostja commented Jan 7, 2016

anatol commented Feb 10, 2020

ligurio commented Nov 20, 2020 • edited Loading

ligurio commented Nov 20, 2020 •

edited

Loading