From 5aad3bafdfb5617a154441de4f73e5c0e92548f1 Mon Sep 17 00:00:00 2001 From: billschereriii Date: Wed, 9 Aug 2023 14:42:57 -0500 Subject: [PATCH 1/3] Extended section on dataset aggregation --- doc/advanced_data_structures.rst | 117 +++++++++++++++++++++++++++++++ doc/changelog.rst | 3 + doc/data_structures.rst | 47 +------------ doc/index.rst | 1 + 4 files changed, 124 insertions(+), 44 deletions(-) create mode 100644 doc/advanced_data_structures.rst diff --git a/doc/advanced_data_structures.rst b/doc/advanced_data_structures.rst new file mode 100644 index 000000000..080834872 --- /dev/null +++ b/doc/advanced_data_structures.rst @@ -0,0 +1,117 @@ +****************************** +Advanced Data Structure Topics +****************************** + +This page of documentation is reserved for advanced topics in SmartRedis +data structures that may not be needed for all users. + +.. _advanced_data_structures_dataset_aggregation: + +Dataset Aggregation +=================== + +In addition to the ability to work with individual datasets, SmartRedis offers +users the manage lists of datasets and work with them collectively. This is +particularly useful for producer/consumer relationships. + +The DataSet Aggregation API stores multiple ``DataSet`` objects +on one or more database nodes through an interface referred to as +``aggregation lists``. +An ``aggregation list`` in SmartRedis stores references to +``DataSet`` objects that are stored in the database. The SmartRedis client +in one application can append ``DataSet`` objects to the ``aggregation list``; +subsequently, ``SmartRedis`` clients in the same application or a different +application can retrieve some or all of the ``DataSet`` objects referenced +in that ``aggregation list``. + +Appending to a DataSet +---------------------- + +For example, the C++ client function to append a ``DataSet`` to an +aggregation list is shown below: + +.. code-block:: cpp + + # C++ aggregation list append interface + void append_to_list(const std::string& list_name, + const DataSet& dataset); + +The above function will append the provided ``DataSet`` to the +``aggregation list``, which can be referenced in all user-facing functions +by the provided list name. Note that a list can be appended by +any client in the same or different application. Implicitly, all +DataSets, when appended, are added to the end of the list. If the list does not +already exist, it is automatically created. + +Retrieving DataSets from an aggregation list +-------------------------------------------- + +To retrieve ``aggregation list`` contents, +the SmartRedis ``Client`` method provides an API function that +will return an iterable container with all of the ``DataSet`` objects +appended to the ``aggregation list``. For example, the C++ client +function to retrieve the entire ``aggregation list`` contents is shown below: + +.. code-block:: cpp + + # C++ aggregation list retrieval interface + std::vector get_datasets_from_list(const std::string& list_name); + +It is also possible to retrieve a subset of the DataSets within an aggregation +list: + +.. code-block:: cpp + + # C++ aggregation list subset retrieval interface + std::vector get_dataset_list_range(const std::string& list_name, + const int start_index, + const int end_index); + +An application can determine how many DataSets are in an aggregation List +via the following API: + +.. code-block:: cpp + + # C++ aggregation list length query + int get_list_length(const std::string& list_name); + +Synchronization +--------------- + +A SmartRedis Client can use the following APIs to block until a +predicate is matched on the length of the list: + +.. code-block:: cpp + + # Block until the list reaches a specific length + bool poll_list_length(const std::string& name, int list_length, + int poll_frequency_ms, int num_tries); + + # Block until the list reaches or exceeds a specific length + bool poll_list_length_gte(const std::string& name, int list_length, + int poll_frequency_ms, int num_tries); + + # Block until the list is no longer than a specific length + bool poll_list_length_lte(const std::string& name, int list_length, + int poll_frequency_ms, int num_tries); + +Other operations +---------------- + +Finally, aggregation lists may be copied, renamed, or deleted. Note +that there is no synchronization support for these operations; performing +these operations when multiple applications are accessing the list may +lead to race conditions: + +.. code-block:: cpp + + # Copy an aggregation list + void copy_list(const std::string& src_name, + const std::string& dest_name); + + # Rename an aggregation list + void rename_list(const std::string& src_name, + const std::string& dest_name); + + # Delete an aggregation list + void delete_list(const std::string& list_name); diff --git a/doc/changelog.rst b/doc/changelog.rst index f71d00811..ba69f4576 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -8,6 +8,7 @@ To be released at some future point in time Description +- Expanded documentation of aggregation lists - Updated third-party software dependencies to current versions - Updated post-merge tests in CI/CD to work with new test system - Enabled static builds of SmartRedis @@ -20,6 +21,7 @@ Description Detailed Notes +- Added a new advanced topics documentation page with a section on aggregation lists (PR390_) - Updated pybind (2.10.3 => 2.11.1), hiredis (1.1.0 => 1.2.0), and redis++ (1.3.5 => 1.3.10) dependencies to current versions (PR389_) - Post-merge tests in CI/CD have been updated to interface cleanly with the new test system that was deployed in the previous release (PR388_) - Static builds of SmartRedis can now work with Linux platforms. Fortran is tested with GNU, PGI, Intel compilers (PR386_) @@ -31,6 +33,7 @@ Detailed Notes - Deleted obsolete build and testing files that are no longer needed with the new build and test system (PR366_) - Reuse existing redis connection when mapping the Redis cluster (PR364_) +.. _PR390: https://github.com/CrayLabs/SmartRedis/pull/390 .. _PR389: https://github.com/CrayLabs/SmartRedis/pull/389 .. _PR388: https://github.com/CrayLabs/SmartRedis/pull/388 .. _PR386: https://github.com/CrayLabs/SmartRedis/pull/386 diff --git a/doc/data_structures.rst b/doc/data_structures.rst index e10fd9d9a..3de3253ba 100644 --- a/doc/data_structures.rst +++ b/doc/data_structures.rst @@ -327,50 +327,9 @@ was used when constructing the metadata field with Aggregating ----------- -An API is provided to aggregate multiple ``DataSet`` objects that -are stored on one or more database nodes. This is accomplished -through an interface referred to as ``aggregation lists``. -An ``aggregation list`` in SmartRedis stores references to -``DataSet`` objects that are stored in the database. ``DataSet`` -objects can be appended to the ``aggregation list`` and then -``SmartRedis`` clients in the same application or a different application -can retrieve all or some of the ``DataSet`` objects referenced in that -``aggregation list``. - -For example, the C++ client function to append a ``DataSet`` to an -aggregation list is shown below: - -.. code-block:: cpp - - # C++ aggregation list append interface - void append_to_list(const std::string& list_name, - const DataSet& dataset); - -The above function will append the provided ``DataSet`` to the -``aggregation list``, which can be referenced in all user-facing functions -by the provided list name. Note that a list can be appended by -any client in the same or different application. Additionally, all -appends are performed at the end of the list, and if the list does not -already exist, it is automatically created. - -For retrieval of ``aggregation list`` contents, -the SmartRedis ``Client`` method provides an API function that -will return an iterable container with all of the ``DataSet`` objects -that were appended to the ``aggregation list``. For example, the C++ client -function to retrieve the ``aggregation list`` contents is shown below: - -.. code-block:: cpp - - # C++ aggregation list retrieval interface - std::vector get_datasets_from_list(const std::string& list_name); - -Additional functions are provided to retrieve only a portion of the -``aggregation list`` contents, copy an ``aggregation list``, rename -an ``aggregation list` and retrieve ``aggregation list`` length. -A blocking method to poll the ``aggregation list`` length is also -provided as a means to wait for list completion before performing -another task in the same application or a separate application. - +SmartRedis also supports an advanced API for working with aggregate +lists of DataSets; details may be found +:ref:`here <_advanced_data_structures_dataset_aggregation>`. Model ===== diff --git a/doc/index.rst b/doc/index.rst index ce76bbf29..6f4c85ce6 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -9,6 +9,7 @@ overview data_structures + advanced_data_structures dataset_conversions installation testing From 068e688db9d767865828cb6ab796558bb7f3298c Mon Sep 17 00:00:00 2001 From: billschereriii Date: Mon, 14 Aug 2023 16:31:16 -0500 Subject: [PATCH 2/3] Address review feedback --- ...ata_structures.rst => advanced_topics.rst} | 39 ++++++++++++------- doc/data_structures.rst | 2 +- doc/index.rst | 2 +- 3 files changed, 26 insertions(+), 17 deletions(-) rename doc/{advanced_data_structures.rst => advanced_topics.rst} (76%) diff --git a/doc/advanced_data_structures.rst b/doc/advanced_topics.rst similarity index 76% rename from doc/advanced_data_structures.rst rename to doc/advanced_topics.rst index 080834872..ce4a37dc6 100644 --- a/doc/advanced_data_structures.rst +++ b/doc/advanced_topics.rst @@ -1,11 +1,11 @@ -****************************** -Advanced Data Structure Topics -****************************** +*************** +Advanced Topics +*************** This page of documentation is reserved for advanced topics in SmartRedis data structures that may not be needed for all users. -.. _advanced_data_structures_dataset_aggregation: +.. _advanced_topics_dataset_aggregation: Dataset Aggregation =================== @@ -14,7 +14,7 @@ In addition to the ability to work with individual datasets, SmartRedis offers users the manage lists of datasets and work with them collectively. This is particularly useful for producer/consumer relationships. -The DataSet Aggregation API stores multiple ``DataSet`` objects +The DataSet Aggregation API manages references to multiple ``DataSet`` objects on one or more database nodes through an interface referred to as ``aggregation lists``. An ``aggregation list`` in SmartRedis stores references to @@ -24,10 +24,10 @@ subsequently, ``SmartRedis`` clients in the same application or a different application can retrieve some or all of the ``DataSet`` objects referenced in that ``aggregation list``. -Appending to a DataSet ----------------------- +Appending to a DataSet aggregation list +--------------------------------------- -For example, the C++ client function to append a ``DataSet`` to an +The C++ client function to append a reference to a ``DataSet`` to an aggregation list is shown below: .. code-block:: cpp @@ -36,19 +36,22 @@ aggregation list is shown below: void append_to_list(const std::string& list_name, const DataSet& dataset); -The above function will append the provided ``DataSet`` to the +NOTE: The ``DataSet`` must have already been added to the database via +the ``put_dataset()`` method in the SmartRedis ``Client``. + +The above function will append the a reference to the provided ``DataSet`` to the ``aggregation list``, which can be referenced in all user-facing functions by the provided list name. Note that a list can be appended by any client in the same or different application. Implicitly, all -DataSets, when appended, are added to the end of the list. If the list does not +``DataSets``, when appended, are added to the end of the list. If the list does not already exist, it is automatically created. -Retrieving DataSets from an aggregation list --------------------------------------------- +Retrieving the DataSets in an aggregation list +---------------------------------------------- -To retrieve ``aggregation list`` contents, -the SmartRedis ``Client`` method provides an API function that -will return an iterable container with all of the ``DataSet`` objects +To retrieve the ``DataSet`` referenced in an ``aggregation list``, +the SmartRedis ``Client`` provides an API function that +returns an iterable object containing all of ``DataSets`` appended to the ``aggregation list``. For example, the C++ client function to retrieve the entire ``aggregation list`` contents is shown below: @@ -67,6 +70,11 @@ list: const int start_index, const int end_index); +The start_index and end_index may be specified as negative numbers. +In this case, the offset is from the most recently appended DataSets. +For example, an offset of -1 is the last element in the list, and -3 is the +antepenultimate DataSet. + An application can determine how many DataSets are in an aggregation List via the following API: @@ -75,6 +83,7 @@ via the following API: # C++ aggregation list length query int get_list_length(const std::string& list_name); + Synchronization --------------- diff --git a/doc/data_structures.rst b/doc/data_structures.rst index 3de3253ba..2a1b86467 100644 --- a/doc/data_structures.rst +++ b/doc/data_structures.rst @@ -329,7 +329,7 @@ Aggregating SmartRedis also supports an advanced API for working with aggregate lists of DataSets; details may be found -:ref:`here <_advanced_data_structures_dataset_aggregation>`. +:ref:`here <_advanced_topics_dataset_aggregation>`. Model ===== diff --git a/doc/index.rst b/doc/index.rst index 6f4c85ce6..2a5f5ed90 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -9,11 +9,11 @@ overview data_structures - advanced_data_structures dataset_conversions installation testing runtime + advanced_topics .. toctree:: :maxdepth: 2 From da576058755c67014088c2649566d48a22624a4c Mon Sep 17 00:00:00 2001 From: billschereriii Date: Tue, 15 Aug 2023 10:06:59 -0500 Subject: [PATCH 3/3] One more set of typos --- doc/advanced_topics.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/advanced_topics.rst b/doc/advanced_topics.rst index ce4a37dc6..d21d828a9 100644 --- a/doc/advanced_topics.rst +++ b/doc/advanced_topics.rst @@ -2,16 +2,16 @@ Advanced Topics *************** -This page of documentation is reserved for advanced topics in SmartRedis -data structures that may not be needed for all users. +This page of documentation is reserved for advanced topics +that may not be needed for all users. .. _advanced_topics_dataset_aggregation: Dataset Aggregation =================== -In addition to the ability to work with individual datasets, SmartRedis offers -users the manage lists of datasets and work with them collectively. This is +In addition to the ability to work with individual datasets, SmartRedis lets +users manage lists of datasets and work with them collectively. This is particularly useful for producer/consumer relationships. The DataSet Aggregation API manages references to multiple ``DataSet`` objects @@ -75,7 +75,7 @@ In this case, the offset is from the most recently appended DataSets. For example, an offset of -1 is the last element in the list, and -3 is the antepenultimate DataSet. -An application can determine how many DataSets are in an aggregation List +An application can determine how many DataSets are in an aggregation list via the following API: .. code-block:: cpp