[RFC] Expand the precision of our log10 lookup tables + add precision #1406
Conversation
Codecov Report
@@ Coverage Diff @@
## main #1406 +/- ##
==========================================
- Coverage 90.83% 90.83% -0.01%
==========================================
Files 73 73
Lines 41266 41927 +661
Branches 41266 41927 +661
==========================================
+ Hits 37486 38084 +598
- Misses 3780 3843 +63
Continue to review full report at Codecov.
|
Technically, it is the 4 or 6 bits after the most significant bit (i.e,
Should that say failure probability? I take it the non-linearity occurs when failure probability is less than
I'm in favor of increasing the table size. |
| // Shows the scores of "realistic" sends of 100k sats over channels of 1-10m sats (with a | ||
| // 50k sat reserve). |
There was a problem hiding this comment.
Do we take into account the reserve? Maybe for first hops only?
There was a problem hiding this comment.
What do you mean? The capacity amounts are all 50k sats short of the round number. We probably don't need to care, I just did that case I was seeing strange scores for some channels and copied the amounts over to the test while exploring.
There was a problem hiding this comment.
As in, the capacity passed to the scorer by get_route doesn't deduct the reserve, IIRC.
There was a problem hiding this comment.
Ah, yes, I believe nodes announce their HTLC max without the reserve.
lightning/src/routing/scoring.rs
Outdated
| let negative_log10_times_2048 = | ||
| approx::negative_log10_times_2048(numerator, denominator); | ||
| self.combined_penalty_msat(amount_msat, negative_log10_times_2048, params) | ||
| if amount_msat - min_liquidity_msat < denominator / 64 { |
There was a problem hiding this comment.
Is 64 the same as LOWER_BITS_BOUND?
There was a problem hiding this comment.
Hmm, uhmmmmm, I guess? I'll go ahead and change it to reference the constant cause they're tightly tied, but it was more of an experimental thing. Honestly we should probably do LOWER_BITS_BOUND / 2 here but you end up hitting the cutoff before the penalty gets close to 500, creating a new non-linearity there.
When we send values over channels of rather substantial size, the imprecision of our log lookup tables creates a rather substantial non-linearity between values that round up or down one bit. For example, with the default scoring values, sending 100k sats over channels with 1m, 2m, 3m, and 4m sats of capacity score rather drastically differently: 3645, 2512, 500, and 1442 msat. Here we expand the precision of our log lookup tables rather substantially by: (a) making the multiplier 2048 instead of 1024, which still fits inside a u16, and (b) quadrupling the size of the lookup table to look at the top 6 bits after the most-significant bit of an input instead of the top 4. This makes the scores of the same channels substantially more linear, with values of 3613, 1977, 1474, and 1223 msat. The same channels would be scored at 3611, 1972, 1464, and 1216 msat with a non-approximating scorer.
Hmm, good question! I think kinda no - its non-linear because we quantize too much, ultimately, but of course if the failure probability is relatively high, the quantization error is small relative to the total amount. I think ultimately its a question of how the quantization error compares to the base penalty of 500msat, less the absolute value of it. |
TheBlueMatt
force-pushed
the
2022-04-scorer-precision
branch
from
0b725f2 to
cdaa016
Compare
When we start getting a numerator and divisor particularly close to each other, the log approximation starts to get very noisy. In order to avoid applying scores that are basically noise (and can range upwards of 2x the default per-hop penalty), simply consider such cases as having a success probability of 100%.
TheBlueMatt
force-pushed
the
2022-04-scorer-precision
branch
from
b92cfe4 to
30e1922
Compare
When we send values over channels of rather substantial size, the
imprecision of our log lookup tables creates a rather substantial
non-linearity between values that round up or down one bit.
For example, with the default scoring values, sending 100k sats
over channels with 1m, 2m, 3m, and 4m sats of capacity score
rather drastically differently: 3645, 2512, 500, and 1442 msat.
Here we expand the precision of our log lookup tables rather
substantially by: (a) making the multiplier 2048 instead of 1024,
which still fits inside a u16, and (b) quadrupling the size of the
lookup table to look at the top 6 bits of an input instead of the
top 4.
This makes the scores of the same channels substantially more
linear, with values of 3613, 1977, 1474, and 1223 msat.
The same channels would be scored at 3611, 1972, 1464, and 1216
msat with a non-approximating scorer.
Further, below 1.5625% failure probability (1/64), we simply treat the failure probability as 0 and drop the log-based multipliers entirely.
That's the compelling case for this. On the other hand, one could reasonably argue that the extra precision isn't all that important and we should instead be happy with non-linearity above a reasonable percentage. This would imply something like a non-linearity at 6.25% or 3.125% success probability, where we consider anything below those as 0. That way we'd avoid the lookup table expansion.
Honestly I don't have a strong feeling either way, the lookup table is pretty small in either case, I just started down that path before I decided I needed to bound it, so that code ended up here first.