Merge pull request #32 from KOLANICH/python_fixes

Python packaging-related fixes:

lang_python.adoc

@@ -13,130 +13,156 @@ To generate Python modules, Kaitai Struct compiler should be invoked with
 `-t python` option:
 
 [source,shell]
-kaitai-struct-compiler -t python my_spec.ksy
+kaitai-struct-compiler -t python [--python-package .] my_spec.ksy
 
 This normally generates `my_spec.py` parser module, which is compatible
-with both Python 2 and 3.
+with both Python 2 and 3. It is strongly recommended to use Python 3
+because Python 2 is EOL since 2020-01-01.
 
-TODO: document `--python-package`
+`--python-package` is an optional CLI arg allows you to select a parent package
+name used by one generated file to import symbols from another one.
 
-[[add-runtime]]
-=== Adding runtime library
+It is recommended to just follow the algorithm:
+
+* Determine if you need `--python-package`:
+
+** If you **don't intend to embed** the KSC-generated files and **just want to
+play a bit** in a REPL shell (IPython/Jupyter) in the same dir, **omit** this arg.
+
+** If you **intend to embed** the compilation result into an own package,
+**use `--python-package .`**.
+
+* Put all the KSC-generated code into an own subpackage, in example `kaitai`.
+If your build process regenerates the files on every build, you may want
+(and probably should, if you don't care about users being unable to build your
+package without KSC) `.gitignore` that dir in order not to junk VCS history.
+
+If you don't want to follow the algorithm blindly, just read about the actions
+involved in the Net and read the discussion in the
+https://github.com/kaitai-io/kaitai_struct_doc/pull/32#pullrequestreview-459665339[PR].
+Don't send us issues about the recommendations/generated code contradicting PEP 8 -
+we don't stick to it. If you want PEP 8 compliant code, you can apply external tools like
+`autopep8` and `black` to generated code.
+
+[[install-runtime]]
+=== Installing runtime library
 
 Code generated by Kaitai Struct compiler will require additional runtime
-library (https://pypi.org/project/kaitaistruct/[kaitaistruct at PyPi],
+library (https://pypi.org/project/kaitaistruct/[image:https://img.shields.io/pypi/v/kaitaistruct.svg[kaitaistruct at PyPI]],
 https://github.com/kaitai-io/kaitai_struct_python_runtime[sources at
 GitHub]).
 
 Runtime library API version must match compiler version that you're using
-(i.e. if you use ksc v0.8, then you must use compatible v0.8 runtime).
+(i.e. if you use ksc v0.9, then you must use compatible v0.9 runtime).
+
+If you are using snapshot KSC, you should also use snapshot runtime.
 
-[[add-runtime-release]]
+[[install-runtime-release]]
 ==== Using release version
 
-With stable version of Kaitai Struct compiler, one can use use following
+With stable version of Kaitai Struct compiler, one can use the following
 methods:
 
 * On Debian / Ubuntu, one can install it from apt repositories:
 
-** For Python 2.x:
+** For Python 3.x:
 +
 [source,shell]
-sudo apt-get install python-kaitaistruct
+sudo apt-get install -y python3-kaitaistruct
 +
-Reference: https://packages.debian.org/buster/python-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python-kaitaistruct[package info for Ubuntu]
+Reference: https://packages.debian.org/testing/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_testing/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/focal/python3-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_20_04/python:python-kaitaistruct.svg[package info for Ubuntu]]
 
-** For Python 3.x:
+** For Python 2.x:
 +
 [source,shell]
-sudo apt-get install python3-kaitaistruct
+sudo apt-get install -y python-kaitaistruct
 +
-Reference: https://packages.debian.org/buster/python3-kaitaistruct[package info for Debian], https://packages.ubuntu.com/eoan/python3-kaitaistruct[package info for Ubuntu]
+Reference: https://packages.debian.org/stable/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/debian_stable/python:python-kaitaistruct.svg[package info for Debian]], https://packages.ubuntu.com/focal/python-kaitaistruct[image:https://repology.org/badge/version-for-repo/ubuntu_20_04/python:python-kaitaistruct.svg[package info for Ubuntu]]
 
-* If your project is using *requirements.txt* to manage your
-  dependencies, add the following line to it:
+
+* Add the following dependency specifier into the package configuration
+of the packaging tool you use:
 +
 ----
 kaitaistruct
 ----
 +
-and then run `pip install -r requirements.txt` to update all your
-dependencies.
+and then run `python3 -m pip install --upgrade -e ./` in the dir with your project
+in order to install your package and all its dependencies in
+an editable way. If you don't want to package your software,
+but only install its dependencies, add `kaitaistruct` into `requirements.txt`.
 
-* Otherwise, you can just install the package manually using *pip*
+* Otherwise, you can just install the package manually using `pip`
   (Python package manager):
 +
 [source,shell]
-pip install kaitaistruct
+python3 -m pip install --upgrade kaitaistruct
 
-[[add-runtime-snapshot]]
+[[install-runtime-snapshot]]
 ==== Using latest (snapshot) version
 
-If you're using latest (unreleased AKA "unstable" AKA "snapshot") build
-of Kaitai Struct compiler, that will require latest runtime libraries as
-well:
+If you're using latest (unreleased aka "bleeding edge" aka "unstable"
+aka "snapshot" aka "nightly") build of Kaitai Struct compiler, that will
+require latest runtime libraries as well:
 
-* If your project is using *requirements.txt* to manage your
-  dependencies, add the following line to it:
+* You can a make `pip` to use a snapshot version of the runtime by
+  adding a https://www.python.org/dev/peps/pep-0508/[PEP 508]
+  dependency specifier into the config of your favorite build tool:
 +
 ----
 kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git
 ----
-+
-and then run `pip install -r requirements.txt` to update all your
-dependencies.
+
 
 * Otherwise, you can just install the package manually using *pip*
   (Python package manager):
 +
 [source,shell]
-pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git
+python3 -m pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git
 
-[[add-dependencies]]
-=== Adding dependencies
 
-<<ksy_reference#enums,Enums>> implementation in Python requires a enum
-support as described in
+[[install-dependencies]]
+=== Installing dependencies for legacy Pythons manually
+
+<<ksy_reference#enums,Enums>> implementation in Python requires
+a enum support as described in
 https://www.python.org/dev/peps/pep-0435/[PEP-0435]. This support is
 available out of the box in standard libraries since Python v3.4, or
-since v2.4 using https://pypi.org/project/enum34/[enum34] PyPi
-compatibility layer.
+since v2.4 using https://pypi.org/project/enum34/[enum34] polyfill.
 
-* On Debian / Ubuntu, this library can be installed with `sudo apt-get
-  install python-enum34`
-* Otherwise, one can use just `sudo pip install enum34`
+* One usually doesn't need to install this lib manually, `pip` will install it
+   automatically if it is needed.
+* One can also install it manually:
+** One can use just `python -m pip install enum34`. Don't do it on
+     later Pythons or the apps using just `enum` will be broken!
+** On Debian / Ubuntu, this library can be installed with
+    `sudo apt-get install -y python-enum34`.
 
-[[automation]]
+[[build-automation]]
 === Automation of compilation process
 
 https://setuptools.readthedocs.io/en/latest/[Setuptools] is the de-facto
-standard way of building and installing python packages. This library
-allows extension with
-https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins].
-https://github.com/KOLANICH have developed
-https://github.com/kaitaiStructCompile/kaitaiStructCompile.py[an extension to
-setuptools], allowing you to automate compilation of `*.ksy` files as the
-part of python module build process. The library provides a mostly
-declarative syntax to facilitate installation: you add a parameter
-`kaitai` to your `setup.py` where you enumerate files that you want
-compiled, output directory, compilation flags, and source code
-post-compilation transformations. Here are some examples:
-https://github.com/KOLANICH/NTMDTRead/blob/master/setup.py[1],
-https://github.com/KOLANICH/SpecprParser.py/blob/master/setup.py[2].
+standard way of building Python packages. This library allows extension
+with https://setuptools.readthedocs.io/en/latest/setuptools.html#extending-and-reusing-setuptools[plugins].
+https://gitlab.com/KOLANICH has developed
+https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.setuptools[an extension to setuptools]
+(https://github.com/kaitaiStructCompile/kaitaiStructCompile.setuptools[GitHub mirror is also present]),
+allowing you to automate compilation of `*.ksy` files as the
+part of Python module build process. The library provides a mostly
+declarative syntax to facilitate installation: you add a section
+`[tool.kaitai]` to your `pyproject.toml` where you enumerate the files that you want
+compiled, output directory, compilation flags, source code. 
+Post-compilation transformations are currently supported only via legacy syntax via `setup.py`. The links to examples can be found in the project docs.
 
 Here are
-https://github.com/KOLANICH/kaitaiStructCompile.py/blob/master/kaitaiStructCompile/config.schema.json[the
-docs] in JSON Schema format.
+https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.schemas/blob/master/kaitaiStructCompile/schemas/schemas/config.schema.json[the docs] in https://json-schema.org/specification.html[JSON Schema] format.
 
-Please remember that this lib is VERY UNSTABLE and very likely to be
-improved in a way that changes API (but due to API simplicity it'd be
+Please remember that this lib is **very unstable** and very likely to be
+improved in a way that changes its API (but due to the API simplicity it'd be
 easy to fix your code), but I guess it's much better to use this than to
 reinvent the wheel.
-https://github.com/kaitaiStructCompile/kaitaiStructCompile.py/issues[Contributions
-are welcome!]
-
-Please note that this lib is NOT available on
-https://pypi.org/[pypi] and if you want to install it
-automatically, you should add
-`git+https://github.com/KOLANICH/kaitaiStructCompile.py` into
-`dependency_links` in the config passed to `setuptools.setup`.
+https://gitlab.com/kaitaiStructCompile.py/kaitaiStructCompile.py/issues[Contributions are welcome!]
+
+Please note that this tool is **not** available on https://pypi.python.org[PyPI]
+and should be installed manually, since currently `build_system` of
+`pyproject.toml` doesn't support PEP 508 specifiers.