From d595a5083f909d1351eb5ffe8513700be9e1577a Mon Sep 17 00:00:00 2001 From: KOLANICH Date: Sun, 2 Aug 2020 01:11:42 +0300 Subject: [PATCH] Python packaging-related fixes: * improved the docs on installing python runtime * fixed the links to kaitaiStructCompile.setuptools * added more examples of usage of kaitaiStructCompile.setuptools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Petr Pučil <47499687+generalmimon@users.noreply.github.com> --- lang_python.adoc | 158 +++++++++++++++++++++++++++-------------------- 1 file changed, 92 insertions(+), 66 deletions(-) diff --git a/lang_python.adoc b/lang_python.adoc index c924bb4..f951417 100644 --- a/lang_python.adoc +++ b/lang_python.adoc @@ -13,130 +13,156 @@ To generate Python modules, Kaitai Struct compiler should be invoked with `-t python` option: [source,shell] -kaitai-struct-compiler -t python my_spec.ksy +kaitai-struct-compiler -t python [--python-package .] my_spec.ksy This normally generates `my_spec.py` parser module, which is compatible -with both Python 2 and 3. +with both Python 2 and 3. It is strongly recommended to use Python 3 +because Python 2 is EOL since 2020-01-01. -TODO: document `--python-package` +`--python-package` is an optional CLI arg allows you to select a parent package +name used by one generated file to import symbols from another one. -[[add-runtime]] -=== Adding runtime library +It is recommended to just follow the algorithm: + +* Determine if you need `--python-package`: + +** If you **don't intend to embed** the KSC-generated files and **just want to +play a bit** in a REPL shell (IPython/Jupyter) in the same dir, **omit** this arg. + +** If you **intend to embed** the compilation result into an own package, +**use `--python-package .`**. + +* Put all the KSC-generated code into an own subpackage, in example `kaitai`. +If your build process regenerates the files on every build, you may want +(and probably should, if you don't care about users being unable to build your +package without KSC) `.gitignore` that dir in order not to junk VCS history. + +If you don't want to follow the algorithm blindly, just read about the actions +involved in the Net and read the discussion in the +https://github.com/kaitai-io/kaitai_struct_doc/pull/32#pullrequestreview-459665339[PR]. +Don't send us issues about the recommendations/generated code contradicting PEP 8 - +we don't stick to it. If you want PEP 8 compliant code, you can apply external tools like +`autopep8` and `black` to generated code. + +[[install-runtime]] +=== Installing runtime library Code generated by Kaitai Struct compiler will require additional runtime -library (https://pypi.org/project/kaitaistruct/[kaitaistruct at PyPi], +library (https://pypi.org/project/kaitaistruct/[image:https://img.shields.io/pypi/v/kaitaistruct.svg[kaitaistruct at PyPI]], https://github.com/kaitai-io/kaitai_struct_python_runtime[sources at GitHub]). Runtime library API version must match compiler version that you're using -(i.e. if you use ksc v0.8, then you must use compatible v0.8 runtime). +(i.e. if you use ksc v0.9, then you must use compatible v0.9 runtime). + +If you are using snapshot KSC, you should also use snapshot runtime. -[[add-runtime-release]] +[[install-runtime-release]] ==== Using release version -With stable version of Kaitai Struct compiler, one can use use following +With stable version of Kaitai Struct compiler, one can use the following methods: * On Debian / Ubuntu, one can install it from apt repositories: -** For Python 2.x: +** For Python 3.x: + [source,shell] -sudo apt-get install python-kaitaistruct +sudo apt-get install -y python3-kaitaistruct + -Reference: https://packages.debian.org/buster/python-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python-kaitaistruct[package info for Ubuntu] +Reference: https://packages.debian.org/testing/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_testing/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/focal/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_20_04/python:python-kaitaistruct.svg[package info for Ubuntu]] -** For Python 3.x: +** For Python 2.x: + [source,shell] -sudo apt-get install python3-kaitaistruct +sudo apt-get install -y python-kaitaistruct + -Reference: https://packages.debian.org/buster/python3-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python3-kaitaistruct[package info for Ubuntu] +Reference: https://packages.debian.org/stable/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_stable/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/focal/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_20_04/python:python-kaitaistruct.svg[package info for Ubuntu]] -* If your project is using *requirements.txt* to manage your - dependencies, add the following line to it: + +* Add the following dependency specifier into the package configuration +of the packaging tool you use: + ---- kaitaistruct ---- + -and then run `pip install -r requirements.txt` to update all your -dependencies. +and then run `python3 -m pip install --upgrade -e ./` in the dir with your project +in order to install your package and all its dependencies in +an editable way. If you don't want to package your software, +but only install its dependencies, add `kaitaistruct` into `requirements.txt`. -* Otherwise, you can just install the package manually using *pip* +* Otherwise, you can just install the package manually using `pip` (Python package manager): + [source,shell] -pip install kaitaistruct +python3 -m pip install --upgrade kaitaistruct -[[add-runtime-snapshot]] +[[install-runtime-snapshot]] ==== Using latest (snapshot) version -If you're using latest (unreleased AKA "unstable" AKA "snapshot") build -of Kaitai Struct compiler, that will require latest runtime libraries as -well: +If you're using latest (unreleased aka "bleeding edge" aka "unstable" +aka "snapshot" aka "nightly") build of Kaitai Struct compiler, that will +require latest runtime libraries as well: -* If your project is using *requirements.txt* to manage your - dependencies, add the following line to it: +* You can a make `pip` to use a snapshot version of the runtime by + adding a https://www.python.org/dev/peps/pep-0508/[PEP 508] + dependency specifier into the config of your favorite build tool: + ---- kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git ---- -+ -and then run `pip install -r requirements.txt` to update all your -dependencies. + * Otherwise, you can just install the package manually using *pip* (Python package manager): + [source,shell] -pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git +python3 -m pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git -[[add-dependencies]] -=== Adding dependencies -<> implementation in Python requires a enum -support as described in +[[install-dependencies]] +=== Installing dependencies for legacy Pythons manually + +<> implementation in Python requires +a enum support as described in https://www.python.org/dev/peps/pep-0435/[PEP-0435]. This support is available out of the box in standard libraries since Python v3.4, or -since v2.4 using https://pypi.org/project/enum34/[enum34] PyPi -compatibility layer. +since v2.4 using https://pypi.org/project/enum34/[enum34] polyfill. -* On Debian / Ubuntu, this library can be installed with `sudo apt-get - install python-enum34` -* Otherwise, one can use just `sudo pip install enum34` +* One usually doesn't need to install this lib manually, `pip` will install it + automatically if it is needed. +* One can also install it manually: +** One can use just `python -m pip install enum34`. Don't do it on + later Pythons or the apps using just `enum` will be broken! +** On Debian / Ubuntu, this library can be installed with + `sudo apt-get install -y python-enum34`. -[[automation]] +[[build-automation]] === Automation of compilation process https://setuptools.readthedocs.io/en/latest/[Setuptools] is the de-facto -standard way of building and installing python packages. This library -allows extension with -https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins]. -https://github.com/KOLANICH have developed -https://github.com/kaitaiStructCompile/kaitaiStructCompile.py[an extension to -setuptools], allowing you to automate compilation of `*.ksy` files as the -part of python module build process. The library provides a mostly -declarative syntax to facilitate installation: you add a parameter -`kaitai` to your `setup.py` where you enumerate files that you want -compiled, output directory, compilation flags, and source code -post-compilation transformations. Here are some examples: -https://github.com/KOLANICH/NTMDTRead/blob/master/setup.py[1], -https://github.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2]. +standard way of building Python packages. This library allows extension +with https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins]. +https://gitlab.com/KOLANICH has developed +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools[an extension to setuptools] +(https://github.com/kaitaiStructCompile/kaitaiStructCompile.setuptools[GitHub mirror is also present]), +allowing you to automate compilation of `*.ksy` files as the +part of Python module build process. The library provides a mostly +declarative syntax to facilitate installation: you add a section +`[tool.kaitai]` to your `pyproject.toml` where you enumerate the files that you want +compiled, output directory, compilation flags, source code. +Post-compilation transformations are currently supported only via legacy syntax via `setup.py`. The links to examples can be found in the project docs. Here are -https://github.com/KOLANICH/kaitaiStructCompile.py/blob/master/kaitaiStructCompile/config.schema.json[the -docs] in JSON Schema format. +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.schemas/blob/master/kaitaiStructCompile/schemas/schemas/config.schema.json[the docs] in https://json-schema.org/specification.html[JSON Schema] format. -Please remember that this lib is VERY UNSTABLE and very likely to be -improved in a way that changes API (but due to API simplicity it'd be +Please remember that this lib is **very unstable** and very likely to be +improved in a way that changes its API (but due to the API simplicity it'd be easy to fix your code), but I guess it's much better to use this than to reinvent the wheel. -https://github.com/kaitaiStructCompile/kaitaiStructCompile.py/issues[Contributions -are welcome!] - -Please note that this lib is NOT available on -https://pypi.org/[pypi] and if you want to install it -automatically, you should add -`git+https://github.com/KOLANICH/kaitaiStructCompile.py` into -`dependency_links` in the config passed to `setuptools.setup`. +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.py/issues[Contributions are welcome!] + +Please note that this tool is **not** available on https://pypi.python.org[PyPI] +and should be installed manually, since currently `build_system` of +`pyproject.toml` doesn't support PEP 508 specifiers.