Expose historical bucket data via new accessors #1961
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportBase: 90.72% // Head: 90.78% // Increases project coverage by
📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more Additional details and impacted files@@ Coverage Diff @@
## main #1961 +/- ##
==========================================
+ Coverage 90.72% 90.78% +0.06%
==========================================
Files 97 98 +1
Lines 50579 51541 +962
Branches 50579 51541 +962
==========================================
+ Hits 45888 46793 +905
- Misses 4691 4748 +57
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
tnull
reviewed
Jan 17, 2023
alecchendev
reviewed
Jan 20, 2023
lightning/src/routing/scoring.rs
Outdated
Comment on lines
810
to
809
| /// For example, a value of `[0, 0, 0, 0, 0, 0, 32]` indicates that we belive the probability | ||
| /// of a bound being in the top octile to be 100%, and have never (recently) seen it in any | ||
| /// other octiles. A value of `[31, 0, 0, 0, 0, 0, 0, 32]` indicates we've seen the bound being | ||
| /// both in the top and bottom octile, and roughly with similar (recent) frequency. |
There was a problem hiding this comment.
This part was super helpful for understanding how the historical buckets work/what they represent. I initially started reading the documentation around the HistoricalBucketRangeTracker and didn't really get it until I read this example.
That being said, I think without having seen that implementation, it's not super clear why these numbers are 32/31. I guess it doesn't really matter for calculating the probabilities/interpreting the buckets relative to each other, and in the previous paragraphs it mentions they're on an arbitrary scale, but that was just something that stood out to me. Otherwise I thought this made sense.
There was a problem hiding this comment.
Yea, I think its on purpose here - its an arbitrary scale so we don't want to try to explain the scale (and have it go out of date).
G8XSU
reviewed
Jan 21, 2023
| // The "it failed" increment is 32, where the probability should lie fully in the first | ||
| // octile. | ||
| assert_eq!(scorer.historical_estimated_channel_liquidity_probabilities(42, &target), | ||
| Some(([32, 0, 0, 0, 0, 0, 0, 0], [32, 0, 0, 0, 0, 0, 0, 0]))); | ||
|
|
||
| // Even after we tell the scorer we definitely have enough available liquidity, it will | ||
| // still remember that there was some failure in the past, and assign a non-0 penalty. | ||
| scorer.payment_path_failed(&payment_path_for_amount(1000).iter().collect::<Vec<_>>(), 43); |
There was a problem hiding this comment.
doubt/confused, shouldn't we call payment_path_successful here?
There was a problem hiding this comment.
See payment_for_amount - if we say payment_path_failed at hop 43 (which is after 42) then we're telling the scorer "we were able to pay the required amount at hop 42". If we were to switch to payment_path_successful then we're telling the scorer "we were able to and did pay the required amount at hop 42, so the available liquidity is 1000 less than you think, cause we just pushed funds over it".
`NodeId` is a public key, there's not much reason to not implement `Display` for it and only `Debug`.
TheBlueMatt
force-pushed
the
2023-01-expose-hist-buckets
branch
from
b9d946d to
8c8e18e
Compare
tnull
reviewed
Jan 23, 2023
tnull
reviewed
Jan 25, 2023
TheBlueMatt
force-pushed
the
2023-01-expose-hist-buckets
branch
from
d3ec643 to
a2d763a
Compare
|
Squashed, adding one further doc update from g8xsu: $ git diff-tree -U1 d3ec6435 a2d763ab
diff --git a/lightning/src/routing/scoring.rs b/lightning/src/routing/scoring.rs
index 61071056a..57e784ca7 100644
--- a/lightning/src/routing/scoring.rs
+++ b/lightning/src/routing/scoring.rs
@@ -813,2 +813,5 @@ impl<G: Deref<Target = NetworkGraph<L>>, L: Deref, T: Time> ProbabilisticScorerU
/// both in the top and bottom octile, and roughly with similar (recent) frequency.
+ ///
+ /// Because the datapoints are decayed slowly over time, values will eventually return to
+ /// `Some(([0; 8], [0; 8]))`.
pub fn historical_estimated_channel_liquidity_probabilities(&self, scid: u64, target: &NodeId) |
tnull
previously approved these changes
Jan 26, 2023
TheBlueMatt
force-pushed
the
2023-01-expose-hist-buckets
branch
from
a2d763a to
a21bb65
Compare
When we're calculating if, once we apply the unupdated decays, the historical data tracker has enough data to assign a score, we previously calculated the decayed points while walking the buckets as we don't use the decayed buckets anyway (to avoid losing precision). That is fine, except that as written it decayed individual buckets additional times. Instead, here we actually calculate the full set of decayed buckets and use those to decide if we have valid points. This adds some additional stack space and may in fact be slower, but will be useful in the next commit and shouldn't be a huge change.
Users should be able to view the data we use to score channels, so this exposes that data in new accessors. Fixes lightningdevkit#1854.
TheBlueMatt
force-pushed
the
2023-01-expose-hist-buckets
branch
from
a21bb65 to
3f32f60
Compare
|
Done. |
tnull
approved these changes
Jan 30, 2023
wpaulino
approved these changes
Jan 30, 2023
k0k0ne
pushed a commit
to bitlightlabs/rust-lightning
that referenced
this pull request
Sep 30, 2024
0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1854.