Skip to content

Commit

Permalink
Rename some properties to be more explicit, add docstrings
Browse files Browse the repository at this point in the history
  • Loading branch information
tomasbedrich committed Jul 24, 2018
1 parent 0d44732 commit 5c7af2f
Show file tree
Hide file tree
Showing 2 changed files with 94 additions and 36 deletions.
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ From top to bottom:

1. Override layer = holds runtime directive overrides, if any.
2. Env layer = directives loaded from environment variables are kept in this layer, if any.
3. File layer = directives loaded from file(s) are kept in this layer, if any.
3. File layer = directives loaded from configuration file(s) are kept in this layer, if any.
4. Default layer = holds a default value for **every** initialized directive.

## Learn by example
Expand All @@ -24,14 +24,14 @@ The behavior is best shown on following example:
```python
from llconfig import Config

c = Config('local/override.cnf.py', '/etc/my_app/conf.d', env_prefix='MY_', files_env_var='CONFIG')
c = Config('local/override.cnf.py', '/etc/my_app/conf.d', env_prefix='MY_', config_files_env_var='CONFIG')
c.init('PORT', int, 80)
c['PORT']
```

The returned value is `80`, given that there is no `MY_PORT` env
variable, no `MY_CONFIG` env variable and no `PORT = 1234` line in any of `local/override.cnf.py` or
`/etc/my_app/conf.d/*.cnf.py` files.
`/etc/my_app/conf.d/*.cnf.py` configuration files.

### Search process

Expand All @@ -49,13 +49,13 @@ Then, if the env variable is not present, the file layer is searched. There can
preserves order (so the leftmost part is always handled first).
2. Files passed to constructor (`local/override.cnf.py` and `/etc/my_app/conf.d` in this example). If there
is a path pointing to directory instead of simple file, the **directory is expanded** (non-recursively). The
expansion lists all files in given directory using `file_glob_pattern` attribute **sorted by file name
expansion lists all files in given directory using `expansion_glob_pattern` attribute **sorted by file name
in reverse order** (you can change this behavior by extending this class and overriding `_expand_dir`
method). The expanded files are used as separate sub-layers in place of original directory.

When all of the file sub-layers are created, each file **is executed** and each file's global namespace is
searched for the `PORT` directive (still preserving the order). If found, the directive is returned as is
(without conversion).
When all of the file sub-layers are created, each configuration file **is executed** and each file's global
namespace is searched for the `PORT` directive (still preserving the order). If found, the directive is returned
as is (without conversion).

The default layer is searched as a last resort. As it contains values from directives' `init`, there is always a
default value (unless a search for non-initialized directive is performed). The default value is returned as is.
Expand All @@ -65,7 +65,7 @@ default value (unless a search for non-initialized directive is performed). The
Directive is initialized using `init` method. It takes directive name, converter function (see bellow) and
a default value (which is `None` by default). It is recommended to name directives using upper-case only.
**Any directive you want to use must be initialized,** otherwise it is ignored (unknown env variables, unknown
directives in files, etc.).
directives in configuration files, etc.).

This means that once you initialize a directive you can safely use it without `KeyError`s or without calling
`c.get('PORT', 'default')`. There will always be at least the default value.
Expand Down Expand Up @@ -123,5 +123,5 @@ dict(c) # => {'DB_HOST': 'localhost', 'DB_PORT': 3306, 'DB_USER': None}

**In short: do not use this library in untrusted environment,** unless you completely understand how it works and
what possible attack vectors are. The main concern is that each file forming the file layer is executed. There
is also a possibility to load files using `files_env_var` environment variable (`APP_CONFIG` by default),
is also a possibility to load files using `config_files_env_var` environment variable (`APP_CONFIG` by default),
unless disabled. On top of that, you can compromise your application using badly written converter.
112 changes: 85 additions & 27 deletions llconfig/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,43 +3,84 @@
import os
from pathlib import Path
import types
from typing import Optional, Union, Any, Callable

_logger = logging.getLogger(__name__)


class Config(MutableMapping):
# TODO docstrings
"""
Main object holding the configuration.
"""

__slots__ = (
'files', 'env_prefix', 'files_env_var', '_loaded', '_converters',
'config_files', 'env_prefix', 'config_files_env_var', '_loaded', '_converters',
'_override_layer', '_env_layer', '_file_layer', '_default_layer'
)

autoload = True

file_glob_pattern = '*.cnf.py'

def __init__(self, *files, env_prefix: str = 'APP_', files_env_var: str = 'CONFIG'):
self.files = files
"""bool: Whether to automatically trigger load() on item access or configuration test (if not loaded yet)."""

expansion_glob_pattern = '*.cnf.py'
"""str: Pattern used to expand a directory, when passed instead of a config file."""

def __init__(
self,
*config_files: Union[str, Path],
env_prefix: str = 'APP_',
config_files_env_var: Optional[str] = 'CONFIG'
):
"""
Create configuration object, init empty layers.
Args:
*config_files: Configuration files to load to the file layer.
env_prefix: Prefix of all env vars handled by this library (set to empty string to disable prefixing).
config_files_env_var: Name of env var containing colon delimited list of files to prepend to `config_files`.
Set to `None` to disable this behavior.
"""
self.config_files = config_files
self.env_prefix = env_prefix
self.files_env_var = files_env_var
self.config_files_env_var = config_files_env_var
self._loaded = False

self._converters = {}
"""Holds converter functions to be called every time when converting env variable."""

self._override_layer = {}
"""Layer holding runtime directive overrides, if any."""

self._env_layer = {}
"""Layer holding directives loaded from environment variables, if any."""

self._file_layer = ChainMap()
"""Layer holding directives loaded from file(s), if any."""

self._default_layer = {}
"""Layer holding default value for every initialized directive."""

def init(self, key: str, converter: Callable[[str], Any], default=None):
"""
Initialize configuration directive.
def init(self, key: str, converter: callable, default=None):
if key == self.files_env_var:
raise KeyError('Conflict between directive name and `files_env_var` name.')
Args:
key: Case-sensitive directive name which is used everywhere (in env vars, in config files, in defaults).
converter: Function, which is called when converting env variable value to Python.
default: Directive default value.
"""
if key == self.config_files_env_var:
raise KeyError('Conflict between directive name and `config_files_env_var` name.')

self._loaded = False
self._default_layer[key] = default
self._converters[key] = converter

def load(self):
"""
Load env layer and file layer.
There is no need to call this explicitly when `autoload` is turned on.
"""
self._load_env_vars()
self._load_files()
self._loaded = True
Expand All @@ -55,46 +96,63 @@ def _load_env_vars(self):
_logger.info('env vars loaded')

def _load_files(self):
_logger.debug('loading files')
_logger.debug('loading config files')

paths = []

if self.files_env_var:
env_var = self.env_prefix + self.files_env_var
_logger.debug('loading files from env var "{}"'.format(env_var))
if self.config_files_env_var:
env_var = self.env_prefix + self.config_files_env_var
_logger.debug('getting list of config files from env var "{}"'.format(env_var))
env_var_val = os.environ.get(env_var)
if env_var_val:
paths.extend(Path(p) for p in env_var_val.split(':'))

if self.files:
paths.extend(Path(p) for p in self.files)
if self.config_files:
paths.extend(Path(p) for p in self.config_files)

files = []
config_files = []
for p in paths:
if p.is_dir():
files.extend(self._expand_dir(p))
config_files.extend(self._expand_dir(p))
else:
files.append(p)
config_files.append(p)

_logger.debug('list of files to load: {}'.format(files))
self._file_layer.maps[:] = [self._load_file(f) for f in files]
_logger.info('files loaded')
_logger.debug('list of config files to load: {}'.format(config_files))
self._file_layer.maps[:] = [self._load_file(f) for f in config_files]
_logger.info('config files loaded')

def _expand_dir(self, path: Path):
files = path.glob(self.file_glob_pattern)
"""
Returns:
List[Path]: Contents of given path non-recursively expanded using `expansion_glob_pattern`, sorted by file
name in reverse order.
"""
files = path.glob(self.expansion_glob_pattern)
files = filter(lambda f: f.is_file(), files)
files = sorted(files, key=lambda f: f.name, reverse=True)
return files
return list(files)

def _load_file(self, file: Path):
"""
Execute given file and parse config directives from it.
Returns:
Dict[str, Any]: Global namespace of executed file filtered to contain only initialized config keys.
"""
_logger.debug('loading file: "{}"'.format(file))
d = types.ModuleType(file.stem)
d.__file__ = file.name
exec(compile(file.read_bytes(), file.name, 'exec'), d.__dict__)
return {key: getattr(d, key) for key in dir(d) if key in self._default_layer}

# TODO naming?
def get_namespace(self, namespace: str, lowercase: bool = True, trim_namespace: bool = True):
"""
Returns:
Dict[str, Any]: Dict containing a subset of configuration options matching the specified namespace.
See Also:
http://flask.pocoo.org/docs/1.0/api/#flask.Config.get_namespace
"""
if not namespace:
raise ValueError('Namespace must not be empty.')

Expand All @@ -113,7 +171,7 @@ def get_namespace(self, namespace: str, lowercase: bool = True, trim_namespace:
return res

def test(self):
# TODO
# TODO try to convert each env variable to raise error as soon as possible
# _logger.debug('testing configuration')
pass

Expand Down

0 comments on commit 5c7af2f

Please sign in to comment.