Package directory structure

This python project is structured using a src subdirectory, with tests package placed at root.

The following package structure was selected based on the system requirements for this type of python package, shareable library, and command line tool system.

The key decisions were:

• tests directory at root and configured as a package

  • facilitates use of tests during development
  • developers can run tests via source code, users do not require access to tests after installation
  • subdirectories encourage clear categorisation of test by "type" and ease of visual checks of test coverage during development

• main package nested within a "src" directory

  • This structure is common and noted e.g in the disutils (https://docs.python.org/3/distutils/setupscript.html?highlight=libraries) and setuptools documentation (https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html?highlight=src#using-a-src-layout)
  • The name of the subdirectory is not important, "src" is common, some packages use "lib".
  • The main benefits is that the package is not locally importable from the project root or from the test directory and therefore testing will always focus on installed code, not local source code

• scripts directory for executables (rather than bin)

  • easy to interpret and aids visual search

For further details on impacts of package structure:

• https://github.com/cma-open/cmatools/wiki/Package-structure-and-testing

---

.github
data
docs
logs
scripts
src
  /cmatools
  __init__.py
    subpackage-a
      __init__.py_
      module.py
    subpackage-b
      __init.__py
      anothermodule.py
tests
  __init__.py
  unit-tests
  integration-tests
  end-to-end-tests
  user-interface-tests
  conftest.py
.coveragerc
.gitignore
.nojekyll
bandit.yml
CHANGELOG.rst
codecov.yml
environment.yml
index.html
LICENSE
pyproject.toml
README.md
setup.cfg
setup.py

---

The project structure, with comments on content and purpose

.github  # Workflow actions. Automation of testing, documentation and releases.
data # This acts as a placeholder location for input, intermediate and output data. 
     # For convenience and visibility this is nested within the repo structure. 
     # The files are ignored by github and are not committed to the repo. 
     # In production config would set this on another disk e.g. for inputs, outputs, scratch.
docs # The system documentation. Built via Sphinx. The source files are maintained within the repo. 
     # The build files are frequently regenerated when the documentation is to be viewed. 
     # These files are used by the gh-pages branch to build the system webpages.
logs # This dir and files are ignored by github and will not appear in the repo. 
     # This is visible locally once the system is installed and run. 
     # Holds log files generated by the system. 
     # Used to store info on system status, config, performance etc useful to users or developers.
scripts # All execuatable scripts e.g. shell scripts used for conda builds or to run tests
        # Subdirectories hold collections of scripts.
    _template # Examples and best practice for shell scripting.
    codestyle # Script to run the code style formatter
    conda # Scripts to install miniconda and create or remove conda environments 
    docs # Script to build the sphinx documentation
    tests # Scripts to the tests by each "type", e.g. unit tests, integration tests, etc.
    uninstall # Scripts to fully uninstall the system, for use in testing during development.
src # This “root package” contains the main python package as a subdirectory. 
    # May also holds any modules not in a package.
  package # The python package , e.g. "cmatools".
    __init__.py # This holds docstrings to list the purpose and content of the package.
                # Currently also sets the version via __version__
    subpackage-a # Subpackages help to organise the package structure.
      __init__.py_
      module.py
    subpackage-b
      __init.__py
      anothermodule.py
tests # All tests, stored as a package and subpackages. Organised by test "type".
  __init__.py
  unit-tests # Replicates the package structure.
             # Holds tests for each "unit" of code, e.g. function and class method.
  integration-tests # Tests that examine how different parts of the system work together.
  end-to-end-tests # Tests that replicate a full run of the system.
                   # May include e.g. sourcing inputs, processing, verifying ouputs.
  user-interface-tests # Tests from a user perspective and examining all user modifiable interfaces
                       # These should be tailored to the expected type of user.
                       # Should cover any potential errors or unexpected input by the users.
  conftest.py # Config file for pytest.
              # Used to create useage error if the tests are run before the system is fully installed.
.coveragerc # Config for coverage testing.
.gitignore  # File to list any files not to be stored and committed to the repo.
            # Examples include the logs and data dirs, alos various tests output and system build files.
.nojekyll # Configur for the webpages on GitHub.
bandit.yml # Config for testing, also used by Codacy.
CHANGELOG.rst # File to list the main changes, enhancements, bug fixes etc in the latest release.
codecov.yml # Config for coverage testing, e.g. setting required thresholds.
environment_*.yml # These files hold the required dependencies used to create isolated conda environments.
                  # Used during development and by the automated testing.
index.html # Used to redirect webpages to the docs subdir, for the gh-pages documentation
LICENSE # The chosen license applicable to the current phase of code development.
        # e.g. BSD 3-Clause license.
pyproject.toml # Config file for various setup related settings.
README.md # The initial page that viewers of the GitHub repo will see. 
          # This page also hold the badges "dashboard" that shows the status of the project code, tests, etc.
setup.cfg # Used to list configuration settings for setup.py.
setup.py # This file configures the installation of the package, for both developers and end users
         # Various configuration options here and via the linked setup.cfg file.

---