Replace KVStorePersister with KVStore

[tnull]

So far, the main persistence interface implemented by the user was the KVStorePersister which provided a number of blanket implementations for convenience. However, as we'll need additional functionalities soon (namely, the capability of removing previously persisted data items), this interface was rather limited.

Here, we upstream the KVStore interface with slight modifications from LDK Node and add the FilesystemStore implementation as part of a new lightning-storage crate that supersedes lightning-persister.

[codecov-commenter]

Codecov Report

Patch coverage: 76.19% and project coverage change: +1.12% 🎉

Comparison is base (073f078) 90.59% compared to head (af2bd74) 91.72%.
Report is 24 commits behind head on main.

❗ Current head af2bd74 differs from pull request most recent head 7a65671. Consider uploading reports for the commit 7a65671 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2472      +/-   ##
==========================================
+ Coverage   90.59%   91.72%   +1.12%     
==========================================
  Files         110      112       +2     
  Lines       57509    66839    +9330     
  Branches    57509    66839    +9330     
==========================================
+ Hits        52101    61307    +9206     
- Misses       5408     5532     +124     

Files Changed	Coverage Δ
lightning/src/util/test_utils.rs	68.70% <0.00%> (-4.91%)	⬇️
lightning-persister/src/utils.rs	71.42% <71.42%> (ø)
lightning/src/util/persist.rs	84.78% <80.55%> (-15.22%)	⬇️
lightning-background-processor/src/lib.rs	81.41% <82.05%> (-0.91%)	⬇️
lightning-persister/src/fs_store.rs	87.71% <87.71%> (ø)
lightning-persister/src/lib.rs	100.00% <100.00%> (+10.83%)	⬆️
lightning-persister/src/test_utils.rs	100.00% <100.00%> (ø)

... and 13 files with indirect coverage changes

☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.

[tnull]

Ran a few bench iterations with n = 1000 sample size each to check there is no regression in write performance. Unsurprisingly (as FilesystemStore is in good part the same code as FilesystemPersister), I saw no significant change or at times even a slight improvement.

For example some results (Linux on Intel(R) Xeon(R) CPU E3-1226 v3 @ 3.30GHz):

Benchmarking bench_filesystem_persisted_sends: Collecting 1000 samples in estimated 235.35 s (1000 iterations)
Benchmarking bench_filesystem_persisted_sends: Analyzing
bench_filesystem_persisted_sends
                        time:   [210.07 ms 217.72 ms 225.39 ms]
                        change: [-16.944% -12.610% -8.2165%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking bench_filesystem_persisted_sends: Collecting 1000 samples in estimated 114.87 s (1000 iterations)
Benchmarking bench_filesystem_persisted_sends: Analyzing
bench_filesystem_persisted_sends
                        time:   [234.67 ms 243.50 ms 252.42 ms]
                        change: [-2.5607% +2.5055% +7.8026%] (p = 0.34 > 0.05)
                        No change in performance detected.

[tnull]

(Not exactly sure why the BP test is stuck on windows but no other platform, will need to setup a windows box to debug this further)

[tnull]

Rebased on main.

[tnull]

(Not exactly sure why the BP test is stuck on windows but no other platform, will need to setup a windows box to debug this further)

Windows issues are fixed, CI failure is #2438.

[tnull]

feel free to squash

Squashed fixups and included the following changes (as I decided to move the key length to a pub const to be in line with the alphabet):

> git diff-tree -U2 09b9fa8c 9e89a53b
diff --git a/lightning-persister/src/utils.rs b/lightning-persister/src/utils.rs
index bf042ff9..ae83aa47 100644
--- a/lightning-persister/src/utils.rs
+++ b/lightning-persister/src/utils.rs
@@ -1,6 +1,5 @@
-use lightning::util::persist::KVSTORE_NAMESPACE_KEY_ALPHABET;
+use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_MAX_KEY_LEN};
 use lightning::util::string::PrintableString;

-const KVSTORE_MAX_KEY_LEN: usize = 120;

 pub(crate) fn is_valid_kvstore_str(key: &str) -> bool {
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index d1f70e24..4f00d3de 100644
--- a/lightning/src/util/persist.rs
+++ b/lightning/src/util/persist.rs
@@ -32,4 +32,7 @@ use crate::util::ser::{ReadableArgs, Writeable};
 pub const KVSTORE_NAMESPACE_KEY_ALPHABET: &str = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-";

+/// The maximum number of characters keys may have.
+pub const KVSTORE_MAX_KEY_LEN: usize = 120;
+
 /// The namespace under which the [`ChannelManager`] will be persisted.
 pub const CHANNEL_MANAGER_PERSISTENCE_NAMESPACE: &str = "";
@@ -66,6 +69,6 @@ pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
 ///
 /// Keys and namespaces are required to be valid ASCII strings in the range of
-/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than 120 characters. Empty namespaces and
-/// sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
+/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_MAX_KEY_LEN`]. Empty namespaces
+/// and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
 /// `sub_namespace` is required to be empty, too. This means that concerns should always be
 /// separated by namespace first, before sub-namespaces are used. While the number of namespaces

[tnull]

So, I grew paranoid about hitting the max path length limitations on windows and added a corresponding test case. Low and behold there of course was an issue, so I now included a fix (canonicalizating paths on Windows):

g> git diff-tree -U2 9e89a53b af2bd741
diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index 7e5c2568..77c03a64 100644
--- a/lightning-persister/src/fs_store.rs
+++ b/lightning-persister/src/fs_store.rs
@@ -67,4 +67,26 @@ impl FilesystemStore {
                }
        }
+
+       fn get_dest_dir_path(&self, namespace: &str, sub_namespace: &str) -> std::io::Result<PathBuf> {
+               let mut dest_dir_path = {
+                       #[cfg(target_os = "windows")]
+                       {
+                               let data_dir = self.data_dir.clone();
+                               fs::create_dir_all(data_dir.clone())?;
+                               fs::canonicalize(data_dir)?
+                       }
+                       #[cfg(not(target_os = "windows"))]
+                       {
+                               self.data_dir.clone()
+                       }
+               };
+
+               dest_dir_path.push(namespace);
+               if !sub_namespace.is_empty() {
+                       dest_dir_path.push(sub_namespace);
+               }
+
+               Ok(dest_dir_path)
+       }
 }

@@ -73,9 +95,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, Some(key), "read")?;

-               let mut dest_file_path = self.data_dir.clone();
-               dest_file_path.push(namespace);
-               if !sub_namespace.is_empty() {
-                       dest_file_path.push(sub_namespace);
-               }
+               let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
                dest_file_path.push(key);

@@ -100,9 +118,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, Some(key), "write")?;

-               let mut dest_file_path = self.data_dir.clone();
-               dest_file_path.push(namespace);
-               if !sub_namespace.is_empty() {
-                       dest_file_path.push(sub_namespace);
-               }
+               let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
                dest_file_path.push(key);

@@ -191,9 +205,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, Some(key), "remove")?;

-               let mut dest_file_path = self.data_dir.clone();
-               dest_file_path.push(namespace);
-               if !sub_namespace.is_empty() {
-                       dest_file_path.push(sub_namespace);
-               }
+               let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
                dest_file_path.push(key);

@@ -284,10 +294,5 @@ impl KVStore for FilesystemStore {
                check_namespace_key_validity(namespace, sub_namespace, None, "list")?;

-               let mut prefixed_dest = self.data_dir.clone();
-               prefixed_dest.push(namespace);
-               if !sub_namespace.is_empty() {
-                       prefixed_dest.push(sub_namespace);
-               }
-
+               let prefixed_dest = self.get_dest_dir_path(namespace, sub_namespace)?;
                let mut keys = Vec::new();

diff --git a/lightning-persister/src/test_utils.rs b/lightning-persister/src/test_utils.rs
index 718b8ef1..91557500 100644
--- a/lightning-persister/src/test_utils.rs
+++ b/lightning-persister/src/test_utils.rs
@@ -1,3 +1,3 @@
-use lightning::util::persist::{KVStore, read_channel_monitors};
+use lightning::util::persist::{KVStore, KVSTORE_NAMESPACE_KEY_MAX_LEN, read_channel_monitors};
 use lightning::ln::functional_test_utils::{connect_block, create_announced_chan_between_nodes,
        create_chanmon_cfgs, create_dummy_block, create_network, create_node_cfgs, create_node_chanmgrs,
@@ -39,4 +39,20 @@ pub(crate) fn do_read_write_remove_list_persist<K: KVStore + RefUnwindSafe>(kv_s
        let listed_keys = kv_store.list(namespace, sub_namespace).unwrap();
        assert_eq!(listed_keys.len(), 0);
+
+       // Ensure we have no issue operating with namespace/sub_namespace/key being KVSTORE_NAMESPACE_KEY_MAX_LEN
+       let max_chars: String = std::iter::repeat('A').take(KVSTORE_NAMESPACE_KEY_MAX_LEN).collect();
+       kv_store.write(&max_chars, &max_chars, &max_chars, &data).unwrap();
+
+       let listed_keys = kv_store.list(&max_chars, &max_chars).unwrap();
+       assert_eq!(listed_keys.len(), 1);
+       assert_eq!(listed_keys[0], max_chars);
+
+       let read_data = kv_store.read(&max_chars, &max_chars, &max_chars).unwrap();
+       assert_eq!(data, &*read_data);
+
+       kv_store.remove(&max_chars, &max_chars, &max_chars, false).unwrap();
+
+       let listed_keys = kv_store.list(&max_chars, &max_chars).unwrap();
+       assert_eq!(listed_keys.len(), 0);
 }

diff --git a/lightning-persister/src/utils.rs b/lightning-persister/src/utils.rs
index ae83aa47..54ec230d 100644
--- a/lightning-persister/src/utils.rs
+++ b/lightning-persister/src/utils.rs
@@ -1,8 +1,8 @@
-use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_MAX_KEY_LEN};
+use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_NAMESPACE_KEY_MAX_LEN};
 use lightning::util::string::PrintableString;


 pub(crate) fn is_valid_kvstore_str(key: &str) -> bool {
-       key.len() <= KVSTORE_MAX_KEY_LEN && key.chars().all(|c| KVSTORE_NAMESPACE_KEY_ALPHABET.contains(c))
+       key.len() <= KVSTORE_NAMESPACE_KEY_MAX_LEN && key.chars().all(|c| KVSTORE_NAMESPACE_KEY_ALPHABET.contains(c))
 }

diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index 4f00d3de..c476edd7 100644
--- a/lightning/src/util/persist.rs
+++ b/lightning/src/util/persist.rs
@@ -32,6 +32,6 @@ use crate::util::ser::{ReadableArgs, Writeable};
 pub const KVSTORE_NAMESPACE_KEY_ALPHABET: &str = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-";

-/// The maximum number of characters keys may have.
-pub const KVSTORE_MAX_KEY_LEN: usize = 120;
+/// The maximum number of characters namespaces and keys may have.
+pub const KVSTORE_NAMESPACE_KEY_MAX_LEN: usize = 120;

 /// The namespace under which the [`ChannelManager`] will be persisted.
@@ -69,7 +69,7 @@ pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
 ///
 /// Keys and namespaces are required to be valid ASCII strings in the range of
-/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_MAX_KEY_LEN`]. Empty namespaces
-/// and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
-/// `sub_namespace` is required to be empty, too. This means that concerns should always be
+/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_NAMESPACE_KEY_MAX_LEN`]. Empty
+/// namespaces and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is
+/// empty, `sub_namespace` is required to be empty, too. This means that concerns should always be
 /// separated by namespace first, before sub-namespaces are used. While the number of namespaces
 /// will be relatively small and is determined at compile time, there may be many sub-namespaces

[tnull]

Addressed the feedback and force-pushed with the following changes:

> git diff-tree -U2 af2bd741 7a656719
diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index 77c03a64..56d071da 100644
--- a/lightning-persister/src/fs_store.rs
+++ b/lightning-persister/src/fs_store.rs
@@ -240,5 +240,5 @@ impl KVStore for FilesystemStore {
                                        //
                                        // In order to assert we permanently removed the file in question we therefore
-                                       // call `fsync` on the parent directory on platforms that support it,
+                                       // call `fsync` on the parent directory on platforms that support it.
                                        dir_file.sync_all()?;
                                }
@@ -328,5 +328,5 @@ impl KVStore for FilesystemStore {
                        // If we otherwise don't find a file at the given path something went wrong.
                        if !metadata.is_file() {
-                               debug_assert!(false, "Failed to list keys of {}/{}: file coulnd't be accessed.",
+                               debug_assert!(false, "Failed to list keys of {}/{}: file couldn't be accessed.",
                                        PrintableString(namespace), PrintableString(sub_namespace));
                                let msg = format!("Failed to list keys of {}/{}: file couldn't be accessed.",
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index c476edd7..ca0605c9 100644
--- a/lightning/src/util/persist.rs
+++ b/lightning/src/util/persist.rs
@@ -61,5 +61,5 @@ pub const SCORER_PERSISTENCE_SUB_NAMESPACE: &str = "";
 pub const SCORER_PERSISTENCE_KEY: &str = "scorer";

-/// Provides an interface that allows to store and retrieve persisted values that are associated
+/// Provides an interface that allows storage and retrieval of persisted values that are associated
 /// with given keys.
 ///
@@ -147,5 +147,5 @@ impl<'a, A: KVStore, M: Deref, T: Deref, ES: Deref, NS: Deref, SP: Deref, F: Der
                L::Target: 'static + Logger,
 {
-       /// Persist the given ['ChannelManager'] to disk, returning an error if persistence failed.
+       /// Persist the given [`ChannelManager`] to disk, returning an error if persistence failed.
        fn persist_manager(&self, channel_manager: &ChannelManager<M, T, ES, NS, SP, F, R, L>) -> Result<(), io::Error> {
                self.write(CHANNEL_MANAGER_PERSISTENCE_NAMESPACE,

[valentinewallace]

This seems like it's had a lot of detailed review already so I'll ack in case we want to merge today and take another glance tomorrow. Can't see there being any blocking feedback.

[G8XSU]

Ship it !

[TheBlueMatt]

One thing that needs addressing in a followup, but given the stack of PRs waiting, gonna go ahead and land this.