Update bom master to upstream master

Documentation/source/advanced_config.rst

@@ -254,7 +254,7 @@ in particular where it creates files within it.
                 *.o (compiled object files)
                 *.mod (mod files)
           metrics/
-          my_program.exe
+          my_program
           log.txt
 
 The *project workspace* folder takes its name from the project label passed in to the build configuration.

Documentation/source/glossary.rst

@@ -29,7 +29,7 @@ Glossary
         Fab's built-in steps come with sensible defaults so the user doesn't have to write unnecessary config.
 
         As an example, the Fortran preprocessor has a default artefact getter which reads *".F90"* files
-        from the :term:`Artefact Collection` called ``"all_source"``.
+        from the :term:`Artefact Collection` called ``"INITIAL_SOURCE"``.
 
         Artefact getters are derived from :class:`~fab.artefacts.ArtefactsGetter`.
 
@@ -65,7 +65,7 @@ Glossary
         A folder inside the :term:`Fab Workspace`, containing all source and output from a build config.
 
     Root Symbol
-        The name of a Fortran PROGRAM, or ``"main"`` for C code. Fab builds an exe for every root symbol it's given.
+        The name of a Fortran PROGRAM, or ``"main"`` for C code. Fab builds an executable for every root symbol it's given.
 
     Source Tree
         The :class:`~fab.steps.analyse.analyse` step produces a dependency tree of the entire project source.

Documentation/source/writing_config.rst

@@ -79,7 +79,7 @@ Please see the documentation for :func:`~fab.steps.find_source_files.find_source
 including how to exclude certain source code from the build. More grab steps can be found in the :mod:`~fab.steps.grab`
 module.
 
-After the find_source_files step, there will be a collection called ``"ALL_SOURCE"``, in the artefact store.
+After the find_source_files step, there will be a collection called ``"INITIAL_SOURCE"``, in the artefact store.
 
 .. [1] See :func:`~fab.steps.c_pragma_injector.c_pragma_injector` for an example of a step which
     creates artefacts in the source folder.
@@ -94,7 +94,7 @@ which must happen before we analyse it.
 
 Steps generally create and find artefacts in the :term:`Artefact Store`, arranged into named collections.
 The :func:`~fab.steps.preprocess.preprocess_fortran`
-automatically looks for Fortran source code in a collection named `'ALL_SOURCE'`,
+automatically looks for Fortran source code in a collection named `'INITIAL_SOURCE'`,
 which is the default output from the preceding :funcfind_source_files step.
 It filters just the (uppercase) ``.F90`` files.
 
@@ -293,8 +293,8 @@ it is the user's responsibility to maintain the default artefact sets
   My apologies for the LONG lines, they were the only way I could find
   to have properly indented paragraphs :(
 
-1. :func:`~fab.steps.find_source_files.find_source_files` will add all source files it finds to ``ALL_SOURCE`` (by default, can be overwritten by the user). Any ``.F90`` and ``.f90`` file will also be added to ``FORTRAN_BUILD_FILES``, any ``.c`` file to ``C_BUILD_FILES``, and any ``.x90`` or ``.X90`` file to ``X90_BUILD_FILES``. It can be called several times if files from different root directories need to be added, and it will automatically update the ``*_BUILD_FILES`` sets.
-2. Any user script that creates new files can add files to ``ALL_SOURCE`` if required, but also to the corresponding ``*_BUILD_FILES``. This will happen automatically if :func:`~fab.steps.find_source_files.find_source_files` is called to add these newly created files.
+1. :func:`~fab.steps.find_source_files.find_source_files` will add all source files it finds to ``INITIAL_SOURCE`` (by default, can be overwritten by the user). Any ``.F90`` and ``.f90`` file will also be added to ``FORTRAN_BUILD_FILES``, any ``.c`` file to ``C_BUILD_FILES``, and any ``.x90`` or ``.X90`` file to ``X90_BUILD_FILES``. It can be called several times if files from different root directories need to be added, and it will automatically update the ``*_BUILD_FILES`` sets.
+2. Any user script that creates new files can add files to ``INITIAL_SOURCE`` if required, but also to the corresponding ``*_BUILD_FILES``. This will happen automatically if :func:`~fab.steps.find_source_files.find_source_files` is called to add these newly created files.
 3. If :func:`~fab.steps.c_pragma_injector.c_pragma_injector` is being called, it will handle all files in ``C_BUILD_FILES``, and will replace all the original C files with the newly created ones. For backward compatibility it will also store the new objects in the ``PRAGMAD_C`` set.
 4. If :func:`~fab.steps.preprocess.preprocess_c` is called, it will preprocess all files in ``C_BUILD_FILES`` (at this stage typically preprocess the files in the original source folder, writing the output files to the build folder), and update that artefact set accordingly. For backward compatibility it will also store the preprocessed files in ``PREPROCESSED_C``.
 5. If :func:`~fab.steps.preprocess.preprocess_fortran` is called, it will preprocess all files in ``FORTRAN_BUILD_FILES`` that end on ``.F90``, creating new ``.f90`` files in the build folder. These files will be added to ``PREPROCESSED_FORTRAN``. Then the original ``.F90`` are removed from ``FORTRAN_BUILD_FILES``, and the new preprocessed files (which are in ``PREPROCESSED_FORTRAN``) will be added. Then any ``.f90`` files that are not already in the build folder (an example of this are files created by a user script) are copied from the original source folder into the build folder, and ``FORTRAN_BUILD_FILES`` is updated to use the files in the new location.

run_configs/lfric/lfric_common.py

@@ -1,9 +1,11 @@
 import logging
 import os
 import shutil
+from typing import Optional
 from pathlib import Path
 
 from fab.artefacts import ArtefactSet
+from fab.build_config import BuildConfig
 from fab.steps import step
 from fab.steps.find_source_files import find_source_files
 from fab.tools import Category, Tool
@@ -111,11 +113,11 @@ def fparser_workaround_stop_concatenation(config):
 
 
 # ============================================================================
-def get_transformation_script(fpath, config):
+def get_transformation_script(fpath: Path,
+                              config: BuildConfig) -> Optional[Path]:
     ''':returns: the transformation script to be used by PSyclone.
-    :rtype: Path
-
     '''
+
     optimisation_path = config.source_root / 'optimisation' / 'meto-spice'
     relative_path = None
     for base_path in [config.source_root, config.build_output]:
@@ -132,4 +134,4 @@ def get_transformation_script(fpath, config):
     global_transformation_script = optimisation_path / 'global.py'
     if global_transformation_script.exists():
         return global_transformation_script
-    return ""
+    return None

source/fab/artefacts.py

@@ -27,7 +27,7 @@
 class ArtefactSet(Enum):
     '''A simple enum with the artefact types used internally in Fab.
     '''
-    ALL_SOURCE = auto()
+    INITIAL_SOURCE = auto()
     PREPROCESSED_FORTRAN = auto()
     PREPROCESSED_C = auto()
     FORTRAN_BUILD_FILES = auto()
@@ -42,8 +42,8 @@ class ArtefactSet(Enum):
 
 
 class ArtefactStore(dict):
-    '''This object stores set of artefacts (which can be of any type). Each artefact
-    is indexed by a string.
+    '''This object stores sets of artefacts (which can be of any type).
+    Each artefact is indexed by either an ArtefactSet enum, or a string.
     '''
 
     def __init__(self):
@@ -89,7 +89,8 @@ def update_dict(self, collection: Union[str, ArtefactSet],
         :param key: the key in the dictionary to update.
         :param values: the values to update with.
         '''
-        self[collection][key].update([values] if isinstance(values, str) else values)
+        self[collection][key].update([values] if isinstance(values, str)
+                                     else values)
 
     def copy_artefacts(self, source: Union[str, ArtefactSet],
                        dest: Union[str, ArtefactSet],
@@ -149,7 +150,8 @@ def __call__(self, artefact_store):
 
 class CollectionGetter(ArtefactsGetter):
     """
-    A simple artefact getter which returns one :term:`Artefact Collection` from the artefact_store.
+    A simple artefact getter which returns one :term:`Artefact Collection`
+    from the artefact_store.
 
     Example::
 
@@ -170,33 +172,38 @@ def __call__(self, artefact_store):
 
 class CollectionConcat(ArtefactsGetter):
     """
-    Returns a concatenated list from multiple :term:`Artefact Collections <Artefact Collection>`
-    (each expected to be an iterable).
+    Returns a concatenated list from multiple
+    :term:`Artefact Collections <Artefact Collection>` (each expected to be
+    an iterable).
 
-    An :class:`~fab.artefacts.ArtefactsGetter` can be provided instead of a collection_name.
+    An :class:`~fab.artefacts.ArtefactsGetter` can be provided instead of a
+    collection_name.
 
     Example::
 
-        # The default source code getter for the Analyse step might look like this.
+        # The default source code getter for the Analyse step might look
+        # like this.
         DEFAULT_SOURCE_GETTER = CollectionConcat([
             'preprocessed_c',
             'preprocessed_fortran',
-            SuffixFilter(ArtefactSet.ALL_SOURCE, '.f90'),
+            SuffixFilter(ArtefactSet.INITIAL_SOURCE, '.f90'),
         ])
 
     """
     def __init__(self, collections: Iterable[Union[ArtefactSet, str,
                                                    ArtefactsGetter]]):
         """
         :param collections:
-            An iterable containing collection names (strings) or other ArtefactsGetters.
+            An iterable containing collection names (strings) or
+            other ArtefactsGetters.
 
         """
         self.collections = collections
 
     # todo: ensure the labelled values are iterables
     def __call__(self, artefact_store: ArtefactStore):
-        # todo: this should be a set, in case a file appears in multiple collections
+        # todo: this should be a set, in case a file appears in
+        # multiple collections
         result = []
         for collection in self.collections:
             if isinstance(collection, (str, ArtefactSet)):
@@ -208,13 +215,13 @@ def __call__(self, artefact_store: ArtefactStore):
 
 class SuffixFilter(ArtefactsGetter):
     """
-    Returns the file paths in a :term:`Artefact Collection` (expected to be an iterable),
-    filtered by suffix.
+    Returns the file paths in a :term:`Artefact Collection` (expected to be
+    an iterable), filtered by suffix.
 
     Example::
 
         # The default source getter for the FortranPreProcessor step.
-        DEFAULT_SOURCE = SuffixFilter(ArtefactSet.ALL_SOURCE, '.F90')
+        DEFAULT_SOURCE = SuffixFilter(ArtefactSet.INITIAL_SOURCE, '.F90')
 
     """
     def __init__(self,
@@ -264,6 +271,7 @@ def __call__(self, artefact_store: ArtefactStore):
 
         build_lists: Dict[str, List[AnalysedDependent]] = {}
         for root, tree in build_trees.items():
-            build_lists[root] = filter_source_tree(source_tree=tree, suffixes=self.suffixes)
+            build_lists[root] = filter_source_tree(source_tree=tree,
+                                                   suffixes=self.suffixes)
 
         return build_lists

source/fab/parse/fortran.py

@@ -298,8 +298,9 @@ def _process_comment(self, analysed_file, obj):
         if comment[:2] == "!$":
             # Check if it is a use statement with an OpenMP sentinel:
             # Use fparser's string reader to discard potential comment
-            # TODO #13: once fparser supports reading the sentinels,
+            # TODO #327: once fparser supports reading the sentinels,
             # this can be removed.
+            # fparser issue: https://github.com/stfc/fparser/issues/443
             reader = FortranStringReader(comment[2:])
             try:
                 line = reader.next()

source/fab/steps/analyse.py

@@ -312,7 +312,8 @@ def _gen_symbol_table(analysed_files: Iterable[AnalysedDependent]) -> Dict[str,
             symbols[symbol_def] = analysed_file.fpath
 
     if duplicates:
-        # we don't break the build because these symbols might not be required to build the exe
+        # we don't break the build because these symbols might not be
+        # required to build the executable.
         # todo: put a big warning at the end of the build?
         err_msg = "\n".join(map(str, duplicates))
         warnings.warn(f"Duplicates found while generating symbol table:\n{err_msg}")

source/fab/steps/archive_objects.py

@@ -24,10 +24,12 @@
 DEFAULT_SOURCE_GETTER = CollectionGetter(ArtefactSet.OBJECT_FILES)
 
 
-# todo: two diagrams showing the flow of artefacts in the exe and library use cases
-#       show how the library has a single build target with None as the name.
+# todo: two diagrams showing the flow of artefacts in the executables and
+# library use cases show how the library has a single build target with None
+# as the name.
 
-# todo: all this documentation for such a simple step - should we split it up somehow?
+# todo: all this documentation for such a simple step - should we split it
+# up somehow?
 
 @step
 def archive_objects(config: BuildConfig,
@@ -73,7 +75,7 @@ def archive_objects(config: BuildConfig,
     targets, each with a name. This typically happens when configuring the
     :class:`~fab.steps.analyser.Analyser` step *with* a root symbol(s).
     We can assume each list of object files is sufficient to build each
-    *<root_symbol>.exe*.
+    *<root_symbol>* executable.
 
     In this case you cannot specify an *output_fpath* path because they are
     automatically created from the target name.
@@ -110,14 +112,15 @@ def archive_objects(config: BuildConfig,
     target_objects = source_getter(config.artefact_store)
     assert target_objects.keys()
     if output_fpath and list(target_objects.keys()) != [None]:
-        raise ValueError("You must not specify an output path (library) when there are root symbols (exes)")
+        raise ValueError("You must not specify an output path (library) when "
+                         "there are root symbols (executables)")
     if not output_fpath and list(target_objects.keys()) == [None]:
         raise ValueError("You must specify an output path when building a library.")
 
     for root, objects in target_objects.items():
 
         if root:
-            # we're building an object archive for an exe
+            # we're building an object archive for an executable
             output_fpath = str(config.build_output / f'{root}.a')
         else:
             # we're building a single object archive with a given filename

source/fab/steps/c_pragma_injector.py

@@ -20,23 +20,30 @@
 
 # todo: test
 @step
-def c_pragma_injector(config, source: Optional[ArtefactsGetter] = None, output_name=None):
+def c_pragma_injector(config, source: Optional[ArtefactsGetter] = None,
+                      output_name=None):
     """
-    A build step to inject custom pragmas to mark blocks of user and system include statements.
+    A build step to inject custom pragmas to mark blocks of user and system
+    include statements.
 
-    By default, reads .c files from the *all_source* artefact and creates the *pragmad_c* artefact.
+    By default, reads .c files from the *INITIAL_SOURCE* artefact and creates
+    the *pragmad_c* artefact.
 
-    This step does not write to the build output folder, it creates the pragmad c in the same folder as the c file.
-    This is because a subsequent preprocessing step needs to look in the source folder for header files,
+    This step does not write to the build output folder, it creates the
+    pragmad c in the same folder as the c file. This is because a subsequent
+    preprocessing step needs to look in the source folder for header files,
     including in paths relative to the c file.
 
     :param config:
-        The :class:`fab.build_config.BuildConfig` object where we can read settings
-        such as the project workspace folder or the multiprocessing flag.
+        The :class:`fab.build_config.BuildConfig` object where we can
+        read settings such as the project workspace folder or the
+        multiprocessing flag.
     :param source:
-        An :class:`~fab.artefacts.ArtefactsGetter` which give us our c files to process.
+        An :class:`~fab.artefacts.ArtefactsGetter` which give us our c files
+        to process.
     :param output_name:
-        The name of the artefact collection to create in the artefact store, with a sensible default
+        The name of the artefact collection to create in the artefact store,
+        with a sensible default
 
     """
     source_getter = source or DEFAULT_SOURCE_GETTER

source/fab/steps/find_source_files.py

@@ -40,7 +40,8 @@ def check(self, path):
 
 class Include(_PathFilter):
     """
-    A path filter which includes matching paths, this convenience class improves config readability.
+    A path filter which includes matching paths, this convenience class
+    improves config readability.
 
     """
     def __init__(self, *filter_strings):
@@ -57,7 +58,8 @@ def __str__(self):
 
 class Exclude(_PathFilter):
     """
-    A path filter which excludes matching paths, this convenience class improves config readability.
+    A path filter which excludes matching paths, this convenience class
+    improves config readability.
 
     """
 
@@ -75,17 +77,18 @@ def __str__(self):
 
 @step
 def find_source_files(config, source_root=None,
-                      output_collection=ArtefactSet.ALL_SOURCE,
+                      output_collection=ArtefactSet.INITIAL_SOURCE,
                       path_filters: Optional[Iterable[_PathFilter]] = None):
     """
     Find the files in the source folder, with filtering.
 
     Files can be included or excluded with simple pattern matching.
     Every file is included by default, unless the filters say otherwise.
 
-    Path filters are expected to be provided by the user in an *ordered* collection.
-    The two convenience subclasses, :class:`~fab.steps.walk_source.Include` and :class:`~fab.steps.walk_source.Exclude`,
-    improve readability.
+    Path filters are expected to be provided by the user in an *ordered*
+    collection. The two convenience subclasses,
+    :class:`~fab.steps.walk_source.Include` and
+    :class:`~fab.steps.walk_source.Exclude`, improve readability.
 
     Order matters. For example::
 
@@ -94,14 +97,17 @@ def find_source_files(config, source_root=None,
             Include('my_folder/my_file.F90'),
         ]
 
-    In the above example, swapping the order would stop the file being included in the build.
+    In the above example, swapping the order would stop the file being
+    included in the build.
 
     A path matches a filter string simply if it *contains* it,
-    so the path *my_folder/my_file.F90* would match filters "my_folder", "my_file" and "er/my".
+    so the path *my_folder/my_file.F90* would match filters
+    "my_folder", "my_file" and "er/my".
 
     :param config:
-        The :class:`fab.build_config.BuildConfig` object where we can read settings
-        such as the project workspace folder or the multiprocessing flag.
+        The :class:`fab.build_config.BuildConfig` object where we can read
+        settings such as the project workspace folder or the multiprocessing
+        flag.
     :param source_root:
         Optional path to source folder, with a sensible default.
     :param output_collection:
@@ -120,8 +126,10 @@ def find_source_files(config, source_root=None,
 
     # file filtering
     filtered_fpaths = set()
-    # todo: we shouldn't need to ignore the prebuild folder here, it's not underneath the source root.
-    for fpath in file_walk(source_root, ignore_folders=[config.prebuild_folder]):
+    # todo: we shouldn't need to ignore the prebuild folder here, it's not
+    # underneath the source root.
+    for fpath in file_walk(source_root,
+                           ignore_folders=[config.prebuild_folder]):
 
         wanted = True
         for path_filter in path_filters:

source/fab/steps/preprocess.py

@@ -207,7 +207,7 @@ class DefaultCPreprocessorSource(ArtefactsGetter):
     """
     def __call__(self, artefact_store):
         return CollectionGetter(ArtefactSet.PRAGMAD_C)(artefact_store) \
-               or SuffixFilter(ArtefactSet.ALL_SOURCE, '.c')(artefact_store)
+               or SuffixFilter(ArtefactSet.INITIAL_SOURCE, '.c')(artefact_store)
 
 
 # todo: rename preprocess_c