diff --git a/doc/advanced_topics.rst b/doc/advanced_topics.rst new file mode 100644 index 000000000..d21d828a9 --- /dev/null +++ b/doc/advanced_topics.rst @@ -0,0 +1,126 @@ +*************** +Advanced Topics +*************** + +This page of documentation is reserved for advanced topics +that may not be needed for all users. + +.. _advanced_topics_dataset_aggregation: + +Dataset Aggregation +=================== + +In addition to the ability to work with individual datasets, SmartRedis lets +users manage lists of datasets and work with them collectively. This is +particularly useful for producer/consumer relationships. + +The DataSet Aggregation API manages references to multiple ``DataSet`` objects +on one or more database nodes through an interface referred to as +``aggregation lists``. +An ``aggregation list`` in SmartRedis stores references to +``DataSet`` objects that are stored in the database. The SmartRedis client +in one application can append ``DataSet`` objects to the ``aggregation list``; +subsequently, ``SmartRedis`` clients in the same application or a different +application can retrieve some or all of the ``DataSet`` objects referenced +in that ``aggregation list``. + +Appending to a DataSet aggregation list +--------------------------------------- + +The C++ client function to append a reference to a ``DataSet`` to an +aggregation list is shown below: + +.. code-block:: cpp + + # C++ aggregation list append interface + void append_to_list(const std::string& list_name, + const DataSet& dataset); + +NOTE: The ``DataSet`` must have already been added to the database via +the ``put_dataset()`` method in the SmartRedis ``Client``. + +The above function will append the a reference to the provided ``DataSet`` to the +``aggregation list``, which can be referenced in all user-facing functions +by the provided list name. Note that a list can be appended by +any client in the same or different application. Implicitly, all +``DataSets``, when appended, are added to the end of the list. If the list does not +already exist, it is automatically created. + +Retrieving the DataSets in an aggregation list +---------------------------------------------- + +To retrieve the ``DataSet`` referenced in an ``aggregation list``, +the SmartRedis ``Client`` provides an API function that +returns an iterable object containing all of ``DataSets`` +appended to the ``aggregation list``. For example, the C++ client +function to retrieve the entire ``aggregation list`` contents is shown below: + +.. code-block:: cpp + + # C++ aggregation list retrieval interface + std::vector get_datasets_from_list(const std::string& list_name); + +It is also possible to retrieve a subset of the DataSets within an aggregation +list: + +.. code-block:: cpp + + # C++ aggregation list subset retrieval interface + std::vector get_dataset_list_range(const std::string& list_name, + const int start_index, + const int end_index); + +The start_index and end_index may be specified as negative numbers. +In this case, the offset is from the most recently appended DataSets. +For example, an offset of -1 is the last element in the list, and -3 is the +antepenultimate DataSet. + +An application can determine how many DataSets are in an aggregation list +via the following API: + +.. code-block:: cpp + + # C++ aggregation list length query + int get_list_length(const std::string& list_name); + + +Synchronization +--------------- + +A SmartRedis Client can use the following APIs to block until a +predicate is matched on the length of the list: + +.. code-block:: cpp + + # Block until the list reaches a specific length + bool poll_list_length(const std::string& name, int list_length, + int poll_frequency_ms, int num_tries); + + # Block until the list reaches or exceeds a specific length + bool poll_list_length_gte(const std::string& name, int list_length, + int poll_frequency_ms, int num_tries); + + # Block until the list is no longer than a specific length + bool poll_list_length_lte(const std::string& name, int list_length, + int poll_frequency_ms, int num_tries); + +Other operations +---------------- + +Finally, aggregation lists may be copied, renamed, or deleted. Note +that there is no synchronization support for these operations; performing +these operations when multiple applications are accessing the list may +lead to race conditions: + +.. code-block:: cpp + + # Copy an aggregation list + void copy_list(const std::string& src_name, + const std::string& dest_name); + + # Rename an aggregation list + void rename_list(const std::string& src_name, + const std::string& dest_name); + + # Delete an aggregation list + void delete_list(const std::string& list_name); diff --git a/doc/changelog.rst b/doc/changelog.rst index f71d00811..ba69f4576 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -8,6 +8,7 @@ To be released at some future point in time Description +- Expanded documentation of aggregation lists - Updated third-party software dependencies to current versions - Updated post-merge tests in CI/CD to work with new test system - Enabled static builds of SmartRedis @@ -20,6 +21,7 @@ Description Detailed Notes +- Added a new advanced topics documentation page with a section on aggregation lists (PR390_) - Updated pybind (2.10.3 => 2.11.1), hiredis (1.1.0 => 1.2.0), and redis++ (1.3.5 => 1.3.10) dependencies to current versions (PR389_) - Post-merge tests in CI/CD have been updated to interface cleanly with the new test system that was deployed in the previous release (PR388_) - Static builds of SmartRedis can now work with Linux platforms. Fortran is tested with GNU, PGI, Intel compilers (PR386_) @@ -31,6 +33,7 @@ Detailed Notes - Deleted obsolete build and testing files that are no longer needed with the new build and test system (PR366_) - Reuse existing redis connection when mapping the Redis cluster (PR364_) +.. _PR390: https://github.com/CrayLabs/SmartRedis/pull/390 .. _PR389: https://github.com/CrayLabs/SmartRedis/pull/389 .. _PR388: https://github.com/CrayLabs/SmartRedis/pull/388 .. _PR386: https://github.com/CrayLabs/SmartRedis/pull/386 diff --git a/doc/data_structures.rst b/doc/data_structures.rst index e10fd9d9a..2a1b86467 100644 --- a/doc/data_structures.rst +++ b/doc/data_structures.rst @@ -327,50 +327,9 @@ was used when constructing the metadata field with Aggregating ----------- -An API is provided to aggregate multiple ``DataSet`` objects that -are stored on one or more database nodes. This is accomplished -through an interface referred to as ``aggregation lists``. -An ``aggregation list`` in SmartRedis stores references to -``DataSet`` objects that are stored in the database. ``DataSet`` -objects can be appended to the ``aggregation list`` and then -``SmartRedis`` clients in the same application or a different application -can retrieve all or some of the ``DataSet`` objects referenced in that -``aggregation list``. - -For example, the C++ client function to append a ``DataSet`` to an -aggregation list is shown below: - -.. code-block:: cpp - - # C++ aggregation list append interface - void append_to_list(const std::string& list_name, - const DataSet& dataset); - -The above function will append the provided ``DataSet`` to the -``aggregation list``, which can be referenced in all user-facing functions -by the provided list name. Note that a list can be appended by -any client in the same or different application. Additionally, all -appends are performed at the end of the list, and if the list does not -already exist, it is automatically created. - -For retrieval of ``aggregation list`` contents, -the SmartRedis ``Client`` method provides an API function that -will return an iterable container with all of the ``DataSet`` objects -that were appended to the ``aggregation list``. For example, the C++ client -function to retrieve the ``aggregation list`` contents is shown below: - -.. code-block:: cpp - - # C++ aggregation list retrieval interface - std::vector get_datasets_from_list(const std::string& list_name); - -Additional functions are provided to retrieve only a portion of the -``aggregation list`` contents, copy an ``aggregation list``, rename -an ``aggregation list` and retrieve ``aggregation list`` length. -A blocking method to poll the ``aggregation list`` length is also -provided as a means to wait for list completion before performing -another task in the same application or a separate application. - +SmartRedis also supports an advanced API for working with aggregate +lists of DataSets; details may be found +:ref:`here <_advanced_topics_dataset_aggregation>`. Model ===== diff --git a/doc/index.rst b/doc/index.rst index ce76bbf29..2a5f5ed90 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -13,6 +13,7 @@ installation testing runtime + advanced_topics .. toctree:: :maxdepth: 2