Extended section on dataset aggregation (#390)

Extended section on dataset aggregation
[ committed by @billschereriii ]
[ reviewed by @ashao @mellis13 ]

doc/advanced_topics.rst

@@ -0,0 +1,126 @@
+***************
+Advanced Topics
+***************
+
+This page of documentation is reserved for advanced topics
+that may not be needed for all users.
+
+.. _advanced_topics_dataset_aggregation:
+
+Dataset Aggregation
+===================
+
+In addition to the ability to work with individual datasets, SmartRedis lets
+users manage lists of datasets and work with them collectively. This is
+particularly useful for producer/consumer relationships.
+
+The DataSet Aggregation API manages references to multiple ``DataSet`` objects
+on one or more database nodes through an interface referred to as
+``aggregation lists``.
+An ``aggregation list`` in SmartRedis stores references to
+``DataSet`` objects that are stored in the database.  The SmartRedis client
+in one application can append ``DataSet`` objects to the ``aggregation list``;
+subsequently, ``SmartRedis`` clients in the same application or a different
+application can retrieve some or all of the ``DataSet`` objects referenced
+in that ``aggregation list``.
+
+Appending to a DataSet aggregation list
+---------------------------------------
+
+The C++ client function to append a reference to a ``DataSet`` to an
+aggregation list is shown below:
+
+.. code-block:: cpp
+
+    # C++ aggregation list append interface
+    void append_to_list(const std::string& list_name,
+                        const DataSet& dataset);
+
+NOTE: The ``DataSet`` must have already been added to the database via
+the ``put_dataset()`` method in the SmartRedis ``Client``.
+
+The above function will append the a reference to the provided ``DataSet`` to the
+``aggregation list``, which can be referenced in all user-facing functions
+by the provided list name.  Note that a list can be appended by
+any client in the same or different application.  Implicitly, all
+``DataSets``, when appended, are added to the end of the list. If the list does not
+already exist, it is automatically created.
+
+Retrieving the DataSets in an aggregation list
+----------------------------------------------
+
+To retrieve the ``DataSet`` referenced in an ``aggregation list``,
+the SmartRedis ``Client`` provides an API function that
+returns an iterable object containing all of ``DataSets``
+appended to the ``aggregation list``.  For example, the C++ client
+function to retrieve the entire ``aggregation list`` contents is shown below:
+
+.. code-block:: cpp
+
+    # C++ aggregation list retrieval interface
+    std::vector<DataSet> get_datasets_from_list(const std::string& list_name);
+
+It is also possible to retrieve a subset of the DataSets within an aggregation
+list:
+
+.. code-block:: cpp
+
+    # C++ aggregation list subset retrieval interface
+    std::vector<DataSet> get_dataset_list_range(const std::string& list_name,
+                                                    const int start_index,
+                                                    const int end_index);
+
+The start_index and end_index may be specified as negative numbers.
+In this case, the offset is from the most recently appended DataSets.
+For example, an offset of -1 is the last element in the list, and -3 is the
+antepenultimate DataSet.
+
+An application can determine how many DataSets are in an aggregation list
+via the following API:
+
+.. code-block:: cpp
+
+    # C++ aggregation list length query
+    int get_list_length(const std::string& list_name);
+
+
+Synchronization
+---------------
+
+A SmartRedis Client can use the following APIs to block until a
+predicate is matched on the length of the list:
+
+.. code-block:: cpp
+
+    # Block until the list reaches a specific length
+    bool poll_list_length(const std::string& name, int list_length,
+                            int poll_frequency_ms, int num_tries);
+
+    # Block until the list reaches or exceeds a specific length
+    bool poll_list_length_gte(const std::string& name, int list_length,
+                                int poll_frequency_ms, int num_tries);
+
+    # Block until the list is no longer than a specific length
+    bool poll_list_length_lte(const std::string& name, int list_length,
+                                int poll_frequency_ms, int num_tries);
+
+Other operations
+----------------
+
+Finally, aggregation lists may be copied, renamed, or deleted. Note
+that there is no synchronization support for these operations; performing
+these operations when multiple applications are accessing the list may
+lead to race conditions:
+
+.. code-block:: cpp
+
+    # Copy an aggregation list
+    void copy_list(const std::string& src_name,
+                    const std::string& dest_name);
+
+    # Rename an aggregation list
+    void rename_list(const std::string& src_name,
+                        const std::string& dest_name);
+
+    # Delete an aggregation list
+    void delete_list(const std::string& list_name);

doc/changelog.rst

@@ -8,6 +8,7 @@ To be released at some future point in time
 
 Description
 
+- Expanded documentation of aggregation lists
 - Updated third-party software dependencies to current versions
 - Updated post-merge tests in CI/CD to work with new test system
 - Enabled static builds of SmartRedis
@@ -20,6 +21,7 @@ Description
 
 Detailed Notes
 
+- Added a new advanced topics documentation page with a section on aggregation lists (PR390_)
 - Updated pybind (2.10.3 => 2.11.1), hiredis (1.1.0 => 1.2.0), and redis++ (1.3.5 => 1.3.10) dependencies to current versions (PR389_)
 - Post-merge tests in CI/CD have been updated to interface cleanly with the new test system that was deployed in the previous release (PR388_)
 - Static builds of SmartRedis can now work with Linux platforms. Fortran is tested with GNU, PGI, Intel compilers (PR386_)
@@ -31,6 +33,7 @@ Detailed Notes
 - Deleted obsolete build and testing files that are no longer needed with the new build and test system (PR366_)
 - Reuse existing redis connection when mapping the Redis cluster (PR364_)
 
+.. _PR390: https://github.com/CrayLabs/SmartRedis/pull/390
 .. _PR389: https://github.com/CrayLabs/SmartRedis/pull/389
 .. _PR388: https://github.com/CrayLabs/SmartRedis/pull/388
 .. _PR386: https://github.com/CrayLabs/SmartRedis/pull/386

doc/data_structures.rst

@@ -327,50 +327,9 @@ was used when constructing the metadata field with
 Aggregating
 -----------
 
-An API is provided to aggregate multiple ``DataSet`` objects that
-are stored on one or more database nodes.  This is accomplished
-through an interface referred to as ``aggregation lists``.
-An ``aggregation list`` in SmartRedis stores references to
-``DataSet`` objects that are stored in the database.  ``DataSet``
-objects can be appended to the ``aggregation list`` and then
-``SmartRedis`` clients in the same application or a different application
-can retrieve all or some of the ``DataSet`` objects referenced in that
-``aggregation list``.
-
-For example, the C++ client function to append a ``DataSet`` to an
-aggregation list is shown below:
-
-.. code-block:: cpp
-
-    # C++ aggregation list append interface
-    void append_to_list(const std::string& list_name,
-                        const DataSet& dataset);
-
-The above function will append the provided ``DataSet`` to the
-``aggregation list``, which can be referenced in all user-facing functions
-by the provided list name.  Note that a list can be appended by
-any client in the same or different application.  Additionally, all
-appends are performed at the end of the list, and if the list does not
-already exist, it is automatically created.
-
-For retrieval of ``aggregation list`` contents,
-the SmartRedis ``Client`` method provides an API function that
-will return an iterable container with all of the ``DataSet`` objects
-that were appended to the ``aggregation list``.  For example, the C++ client
-function to retrieve the ``aggregation list`` contents is shown below:
-
-.. code-block:: cpp
-
-    # C++ aggregation list retrieval interface
-    std::vector<DataSet> get_datasets_from_list(const std::string& list_name);
-
-Additional functions are provided to retrieve only a portion of the
-``aggregation list`` contents, copy an ``aggregation list``, rename
-an ``aggregation list` and retrieve ``aggregation list`` length.
-A blocking method to poll the ``aggregation list`` length is also
-provided as a means to wait for list completion before performing
-another task in the same application or a separate application.
-
+SmartRedis also supports an advanced API for working with aggregate
+lists of DataSets; details may be found
+:ref:`here <_advanced_topics_dataset_aggregation>`.
 
 Model
 =====

doc/index.rst

@@ -13,6 +13,7 @@
    installation
    testing
    runtime
+   advanced_topics
 
 .. toctree::
    :maxdepth: 2