From 845442a40b0732c11c0066c9d670e4ed50646900 Mon Sep 17 00:00:00 2001 From: KOLANICH Date: Sun, 2 Aug 2020 01:11:42 +0300 Subject: [PATCH] Python packaging-related fixes: * improved the docs on installing python runtime * fixed the links to kaitaiStructCompile.setuptools * added more examples of usage of kaitaiStructCompile.setuptools --- lang_python.adoc | 141 +++++++++++++++++++++++++++++------------------ 1 file changed, 88 insertions(+), 53 deletions(-) diff --git a/lang_python.adoc b/lang_python.adoc index c924bb4..f979975 100644 --- a/lang_python.adoc +++ b/lang_python.adoc @@ -13,98 +13,140 @@ To generate Python modules, Kaitai Struct compiler should be invoked with `-t python` option: [source,shell] -kaitai-struct-compiler -t python my_spec.ksy +kaitai-struct-compiler -t python --python-package . my_spec.ksy This normally generates `my_spec.py` parser module, which is compatible -with both Python 2 and 3. +with both Python 2 and 3. It is strongly recommended to use python 3 +because python 2 is EOL since 2020-01-01. You can also install +`python-is-python3` on Ubuntu focal to make `python3` be used via +`python` command. + +`--python-package` optional CLI arg allows you to select a parent package +name used by one generated file to import symbols from another one. + +If you want to embed KSC-generated files into an own package, it is +recommended to set the arg to `.`, which means "current package", +rather than to the real name of the subpackage, because setting it +to a real name will cause breakage when you (or anyone else) want +to reorganize or rename your package. It will cause KSC generate +relative imports instead of absolute ones for the specs imported into +your main spec: + +---- +from .your_imported_spec import YourImportedSpec +---- + +(note the dot we have passed as an arg) instead of + +---- +from your_imported_spec import YourImportedSpec +---- +. + +**Without setting this arg at all you will likely get an `ImportError`, +if any of the specs imports another spec.** You can instead set this +arg to the absolute path of the submodule, but it opens a can of worms. + +It is also recommended to put all the KSC-generated code into an +own subpackage, in example `kaitai`, so, if your build process +regenerates the files on every build, you could just .gitignore that dir. -TODO: document `--python-package` [[add-runtime]] === Adding runtime library Code generated by Kaitai Struct compiler will require additional runtime -library (https://pypi.org/project/kaitaistruct/[kaitaistruct at PyPi], +library (https://pypi.org/project/kaitaistruct/[image:https://img.shields.io/pypi/v/kaitaistruct.svg[kaitaistruct at PyPi]], https://github.com/kaitai-io/kaitai_struct_python_runtime[sources at GitHub]). Runtime library API version must match compiler version that you're using -(i.e. if you use ksc v0.8, then you must use compatible v0.8 runtime). +(i.e. if you use ksc v0.9, then you must use compatible v0.9 runtime). + +If you are using snapshot KSC, you should also use snapshot runtime. [[add-runtime-release]] ==== Using release version -With stable version of Kaitai Struct compiler, one can use use following +With stable version of Kaitai Struct compiler, one can use the following methods: * On Debian / Ubuntu, one can install it from apt repositories: -** For Python 2.x: +** For Python 3.x: + [source,shell] -sudo apt-get install python-kaitaistruct +sudo apt-get install -y python3-kaitaistruct + -Reference: https://packages.debian.org/buster/python-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python-kaitaistruct[package info for Ubuntu] +Reference: https://packages.debian.org/testing/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_testing/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/focal/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_20_04/python:python-kaitaistruct.svg[package info for Ubuntu]] -** For Python 3.x: +** For Python 2.x: + [source,shell] -sudo apt-get install python3-kaitaistruct +sudo apt-get install -y python-kaitaistruct + -Reference: https://packages.debian.org/buster/python3-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python3-kaitaistruct[package info for Ubuntu] +Reference: https://packages.debian.org/stable/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_stable/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/focal/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_20_04/python:python-kaitaistruct.svg[package info for Ubuntu]] -* If your project is using *requirements.txt* to manage your - dependencies, add the following line to it: + +* Add the following dependency specifier into the package configuration +of the packaging tool you use: + ---- kaitaistruct ---- + -and then run `pip install -r requirements.txt` to update all your -dependencies. +and then run `python3 -m pip install --upgrade -e ./` in the dir with your project +in order to install your package and all its dependencies in +an editable way. If you don't want to package your software, +but only install its dependencies, add `kaitaistruct` into `requirements.txt`. -* Otherwise, you can just install the package manually using *pip* +* Otherwise, you can just install the package manually using `pip` (Python package manager): + [source,shell] -pip install kaitaistruct +python3 -m pip install --upgrade kaitaistruct [[add-runtime-snapshot]] ==== Using latest (snapshot) version -If you're using latest (unreleased AKA "unstable" AKA "snapshot") build -of Kaitai Struct compiler, that will require latest runtime libraries as -well: +If you're using latest (unreleased AKA "bleeding edge" AKA "unstable" +AKA "snapshot" AKA "nightly") build of Kaitai Struct compiler, that will +require latest runtime libraries as well: -* If your project is using *requirements.txt* to manage your - dependencies, add the following line to it: +* You can a make `pip` to use a snapshot version of the runtime by + adding a https://www.python.org/dev/peps/pep-0508/[PEP 508] + dependency specifier into the config of your favorite build tool: + ---- kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git ---- + -and then run `pip install -r requirements.txt` to update all your -dependencies. +. * Otherwise, you can just install the package manually using *pip* (Python package manager): + [source,shell] -pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git +python3 -m pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git + [[add-dependencies]] -=== Adding dependencies +=== Adding dependencies manually -<> implementation in Python requires a enum -support as described in +<> implementation in Python requires +a enum support as described in https://www.python.org/dev/peps/pep-0435/[PEP-0435]. This support is available out of the box in standard libraries since Python v3.4, or since v2.4 using https://pypi.org/project/enum34/[enum34] PyPi compatibility layer. -* On Debian / Ubuntu, this library can be installed with `sudo apt-get - install python-enum34` -* Otherwise, one can use just `sudo pip install enum34` +* One usually doesn't need to install this lib manually, `pip` will install it + automatically if it is needed. +* One can also install it manually: +** One can use just `python -m pip install enum34`. Don't do it on + later pythons or the apps using just `enum` will be broken! +** On Debian / Ubuntu, this library can be installed with + `sudo apt-get install -y python-enum34`. [[automation]] === Automation of compilation process @@ -113,30 +155,23 @@ https://setuptools.readthedocs.io/en/latest/[Setuptools] is the de-facto standard way of building and installing python packages. This library allows extension with https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins]. -https://github.com/KOLANICH have developed -https://github.com/kaitaiStructCompile/kaitaiStructCompile.py[an extension to -setuptools], allowing you to automate compilation of `*.ksy` files as the +https://gitlab.com/KOLANICH has developed +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools[an extension to setuptools], allowing you to automate compilation of `*.ksy` files as the part of python module build process. The library provides a mostly -declarative syntax to facilitate installation: you add a parameter -`kaitai` to your `setup.py` where you enumerate files that you want -compiled, output directory, compilation flags, and source code -post-compilation transformations. Here are some examples: -https://github.com/KOLANICH/NTMDTRead/blob/master/setup.py[1], -https://github.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2]. +declarative syntax to facilitate installation: you add a section +`[tool.kaitai]` to your `pyproject.toml` where you enumerate the files that you want +compiled, output directory, compilation flags, source code. +Post-compilation transformations are currently supported only via legacy syntax via `setup.py`. The links to examples can be found in the project docs. Here are -https://github.com/KOLANICH/kaitaiStructCompile.py/blob/master/kaitaiStructCompile/config.schema.json[the -docs] in JSON Schema format. +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.schemas/blob/master/kaitaiStructCompile/schemas/schemas/config.schema.json[the docs] in https://json-schema.org/specification.html[JSON Schema] format. -Please remember that this lib is VERY UNSTABLE and very likely to be -improved in a way that changes API (but due to API simplicity it'd be +Please remember that this lib is **very unstable** and very likely to be +improved in a way that changes its API (but due to the API simplicity it'd be easy to fix your code), but I guess it's much better to use this than to reinvent the wheel. -https://github.com/kaitaiStructCompile/kaitaiStructCompile.py/issues[Contributions -are welcome!] - -Please note that this lib is NOT available on -https://pypi.org/[pypi] and if you want to install it -automatically, you should add -`git+https://github.com/KOLANICH/kaitaiStructCompile.py` into -`dependency_links` in the config passed to `setuptools.setup`. +https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.py/issues[Contributions are welcome!] + +Please note that this lib is NOT available on https://pypi.python.org[pypi] +and should be installed manually, since currently `build_system` of +`pyproject.toml` doesn't support PEP 508 specifiers.