CIP-???? | Coin Selection - Optimized Random Improve #785
Conversation
cardano-dev4
changed the title
rphair
changed the title
rphair
added
the
Category: Wallets
There was a problem hiding this comment.
Thanks @NicholasMaselli & if we can address the basic formatting & CIP structure issues before the next CIP meeting (https://hackmd.io/@cip-editors/84) we might include it on the agenda (for Triage = introduction, not a review of the proposal or its algorithm).
Note since this CIP addresses a CPS (CPS-0009) remember there is a heading preamble item Solution-To which will point your CIP at this CPS. If & when this CIP is confirmed this metadata link should also point from that CPS to your CIP. 🤓
| Category: Wallets | ||
| Authors: | ||
| - Nicholas Maselli <[email protected]> | ||
| Implementors: CardanoSharp 7, SaturnNFT, Levvy Finance |
There was a problem hiding this comment.
These also need to be a bullet list, formatted like the Authors. From reading the Implementations below I can't be sure if the second 2 are implementations of your algorithm... if there is any subjectivity about that I would include your reference implementation alone for now.
| <b>Cardano needs to scale.</b> As the ecosystem grows and new users come in, blocks are regular at 100% load with 88KB per block allocation. Recently, events such as the Lil Sappys NFT mint have pushed the Cardano blockchain load to over 100% for several hours, preventing anyone from using the chain and providing an awful user experience to new users who complain on social media and give the ecosystem a bad reputation. | ||
|
|
||
| <b>This can't continue if Cardano is to succeed.</b> |
There was a problem hiding this comment.
| <b>Cardano needs to scale.</b> As the ecosystem grows and new users come in, blocks are regular at 100% load with 88KB per block allocation. Recently, events such as the Lil Sappys NFT mint have pushed the Cardano blockchain load to over 100% for several hours, preventing anyone from using the chain and providing an awful user experience to new users who complain on social media and give the ecosystem a bad reputation. | |
| <b>This can't continue if Cardano is to succeed.</b> | |
| As the Cardano ecosystem grows and new users come in, blocks are regular at 100% load with 88KB per block allocation. Recently, events such as the Lil Sappys NFT mint have pushed the Cardano blockchain load to over 100% for several hours, preventing anyone from using the chain. |
In general we need to avoid subjective & persuasive language because people cannot generally agree upon these statements at the present time.... let alone years down the road when this document will still persist while these arguments may not.
The Motivation is to outline technical imperatives... it's a courtesy to your readers & potential implementors to simply point out the facts that indicate what needs to be changed & allow readers to draw their own conclusions about the subjective impact of the problem.
| - After looping through each asset we have a list of change outputs. We want to calculate the ada minUtxos required for all of those change outputs. | ||
| - Using this ada minUtxo number along with the amount of ada in the balance we attempt to evenly distribute ada accross the existing change outputs, with the remainder going to the final change output. | ||
|
|
||
| ## Path to Active |
There was a problem hiding this comment.
@NicholasMaselli can you reformat this section clearly with checkboxes on each item, like you see for example in CIP-0095, with the common sections grouped into headings which each have their own checkbox to represent completion of all items in that group?
Editors, reviewers & implementors should be able to scan down the list & quickly see what's been done & pending. Any confirmation or qualification of how external implementations relate to the algorithm you are proposing should be documented in the Rationale section instead.
In fact, your CIP doesn't have a ## Rationale section, which is mandatory. So I would recommend moving the SaturnNFT and Levvy material into Rationale (immediately before) & addressing how the prior art relates to the algorithm that you put into your own reference implementation & want to bring to Cardano at large.
See Rationale: how does this CIP achieve its goals? (CIP-0001 > Structure)
|
if @NicholasMaselli as per #785 (comment) you're fixing up the content issues above on Sunday or Monday then it'll be ready for Triage at our next meeting so I've added it to the upcoming agenda (https://hackmd.io/@cip-editors/84). Please keep mind we will just introduce this PR and your CIP idea since the main thread of the meeting will be to review some related Plutus proposals in a group and try to clear a complicated field of very similar Plutus CIPs. Also please @SIDANWhatever @twwu123 @nicolasayotte @siegfried @fallen-icarus as authors & reviewers of CPS-0009 your opinion on this proposal would be welcome here. |
|
|
||
| ## Motivation: why is this CIP necessary? | ||
|
|
||
| <b>Cardano needs to scale.</b> As the ecosystem grows and new users come in, blocks are regular at 100% load with 88KB per block allocation. Recently, events such as the Lil Sappys NFT mint have pushed the Cardano blockchain load to over 100% for several hours, preventing anyone from using the chain and providing an awful user experience to new users who complain on social media and give the ecosystem a bad reputation. |
There was a problem hiding this comment.
Hi @NicholasMaselli
Many thanks for creating this CIP!
I have some questions about this event:
Recently, events such as the Lil Sappys NFT mint have pushed the Cardano blockchain load to over 100% for several hours, preventing anyone from using the chain and providing an awful user experience to new users who complain on social media and give the ecosystem a bad reputation.
Would you be able to give a more complete description of how you believe this particular event is related to the problem of coin selection?
In other words, can you point to specific problems (related to coin selection) that caused or contributed to the situation you describe?
I'd be particularly interested in:
- specific examples of coin selection attempts that failed (for whatever reasons); or
- specific examples of coin selection attempts that produced transactions that were larger than necessary (thus requiring higher fees than necessary)
One problem I can think of is that in a few years, it may be hard for readers to piece together the details of this event, and how they relate to coin selection. However, this CIP will remain. So including more context here may help readers to better understand the historical motivation.
|
|
||
| <b>This can't continue if Cardano is to succeed.</b> | ||
|
|
||
| Currently the existing <a href="https://github.com/cardano-foundation/CIPs/tree/master/CIP-0002">CIP-2</a> for coin selection is not sufficient as it was developed in a time before native assets and smart contracts were live on the Cardano Blockchain. Many existing dapps are using extremely poor coin selection resulting in transactions that are 2,3, sometimes 5 times larger then neccesary while also creating outputs that are difficult to handle in future coin selections. |
There was a problem hiding this comment.
Many existing dapps are using extremely poor coin selection resulting in transactions that are 2,3, sometimes 5 times larger then neccesary while also creating outputs that are difficult to handle in future coin selections.
Are you able to include some worked examples of transactions that are 2, 3, and sometimes 5 times larger than necessary?
|
Overall I think its nice to see this proposal. It would be nicer to
|
Yes! I will do all that. Unfortunately I am coding 24/7 for a product release at the moment, but once that is out I will clean up and showcase data for this |
There was a problem hiding this comment.
- Use UTxO consistently throughout the document. Since UTxO is an abbreviation for Unspent Transaction Output; UTxO is the correct capitalization.
This was a first draft mostly looking at grammar and structure of the CIP overall. I think there is strong promise and the methodology described makes sense to me. Please see individual comments for some proposed changes and suggestions.
|
|
||
| ## Abstract | ||
|
|
||
| Coin Selection algorithms have become a critical part of the Cardano ecosystem. These algorithms choose a set of input Utxos for a given set of new Outputs. |
There was a problem hiding this comment.
| Coin Selection algorithms have become a critical part of the Cardano ecosystem. These algorithms choose a set of input Utxos for a given set of new Outputs. | |
| Coin Selection algorithms are a critical part of any UTxO-based ecosystem. These algorithms choose a set of input UTxOs for a given set of new Outputs. |
|
|
||
| Coin Selection algorithms have become a critical part of the Cardano ecosystem. These algorithms choose a set of input Utxos for a given set of new Outputs. | ||
|
|
||
| This CIP aims to present optimized coin selection algorithms to enable Wallets and Dapps to interact with smart contracts and native assets such as tokens and NFTs in a more efficient way. The goal of these algorithms is to primarily reduce the size of transactions as well as reduce the mem and step required for smart contract executions. |
There was a problem hiding this comment.
| This CIP aims to present optimized coin selection algorithms to enable Wallets and Dapps to interact with smart contracts and native assets such as tokens and NFTs in a more efficient way. The goal of these algorithms is to primarily reduce the size of transactions as well as reduce the mem and step required for smart contract executions. | |
| This CIP presents an optimized coin selection algorithm to enable Wallets and dApps to interact with smart contracts and native assets in a more efficient way. The primary goal of this algorithm is to reduce the size of transactions as well as the memory and CPU steps required for smart contract executions. |
There was a problem hiding this comment.
agreed: better English & technical term usage
|
|
||
| Currently the existing <a href="https://github.com/cardano-foundation/CIPs/tree/master/CIP-0002">CIP-2</a> for coin selection is not sufficient as it was developed in a time before native assets and smart contracts were live on the Cardano Blockchain. Many existing dapps are using extremely poor coin selection resulting in transactions that are 2,3, sometimes 5 times larger then neccesary while also creating outputs that are difficult to handle in future coin selections. | ||
|
|
||
| This CIP aims to provide at least 1 algorithm and open the conversation for multiple algorithms to exponentially improve coin selection which will decrease transaction sizes accross the board and increase Cardano's throughput. |
There was a problem hiding this comment.
| This CIP aims to provide at least 1 algorithm and open the conversation for multiple algorithms to exponentially improve coin selection which will decrease transaction sizes accross the board and increase Cardano's throughput. | |
| This CIP describes an improved algorithm and opens the conversation for multiple future algorithms to exponentially improve coin selection which will decrease transaction sizes across the board and increase network throughput. |
There was a problem hiding this comment.
... as with last 2 review facets, in agreement with @Crypto2099's grammar & usage changes as they continue below.
|
|
||
| - Transaction size reduction. | ||
| - Change Output size reduction for future transactions. | ||
| - Dust manage to prevent large small dust Utxos from piling up. |
There was a problem hiding this comment.
| - Dust manage to prevent large small dust Utxos from piling up. | |
| - Dust management to prevent large numbers of small value (dust) UTxOs from piling up. |
There was a problem hiding this comment.
Dust is not necessarily a word that everyone understands explicitly, better to help provide a bit of an explainer here before we continue to use the word "dust" further down in the document.
|
|
||
| ### Algorithms | ||
|
|
||
| Random improve was an original algorithm presented in <a href="https://github.com/cardano-foundation/CIPs/tree/master/CIP-0002">CIP-2</a>, this algorithm had a series of good ideas that can be expanded upon here. The first proposed algorithm will simply be an Optimized Random Improve implementation. |
There was a problem hiding this comment.
| Random improve was an original algorithm presented in <a href="https://github.com/cardano-foundation/CIPs/tree/master/CIP-0002">CIP-2</a>, this algorithm had a series of good ideas that can be expanded upon here. The first proposed algorithm will simply be an Optimized Random Improve implementation. | |
| Random Improve was an original algorithm presented in <a href="https://github.com/cardano-foundation/CIPs/tree/master/CIP-0002">CIP-2</a>, this algorithm had a series of good ideas that can be expanded upon. The proposed algorithm will be an Optimized Random Improve implementation. |
There was a problem hiding this comment.
One small point: the original author of this algorithm (Edsko de Vries) called it "Random-Improve" (with a hyphen).
I would propose to use the same style in this document, if possible.
Source: Self Organisation in Coin Selection.
There was a problem hiding this comment.
@NicholasMaselli - unless you can demonstrate that your algorithm is an optimisation of some other "Random Improve" algorithm besides the one @jonathanknowles refers to, editorially we would insist upon changing this term everywhere.
Also: a link to this paper, e.g. as a reference implementation, should be added to the mandatory but currently non-existent Rationale section (see #785 (comment)) Again, if it's a derivation of that algorithm, it's too integral to the CIP itself to be simply included in a References section at the end.
p.s. the already included reference to CIP-0002 technically includes Random-Improve but doesn't link to it specifically, so the prior art is still likely to get missed unless specifically linked to de Vries' paper and/or this section: https://github.com/cardano-foundation/CIPs/tree/master/CIP-0002#random-improve
| Example Definition (C#): | ||
|
|
||
| public void SelectInputs( | ||
| CoinSelection coinSelection, | ||
| List<Utxo> availableUtxos, | ||
| long amount, | ||
| Asset? asset = null, | ||
| List<Utxo>? requiredUtxos = null, | ||
| int limit = 20 | ||
| ) { } | ||
|
|
There was a problem hiding this comment.
| Example Definition (C#): | |
| public void SelectInputs( | |
| CoinSelection coinSelection, | |
| List<Utxo> availableUtxos, | |
| long amount, | |
| Asset? asset = null, | |
| List<Utxo>? requiredUtxos = null, | |
| int limit = 20 | |
| ) { } | |
| Example Definition (C#): | |
| ```csharp | |
| public void SelectInputs( | |
| CoinSelection coinSelection, | |
| List<Utxo> availableUtxos, | |
| long amount, | |
| Asset? asset = null, | |
| List<Utxo>? requiredUtxos = null, | |
| int limit = 20 | |
| ) { } | |
| \``` |
|
|
||
| - <b>Step 1a)</b> | ||
|
|
||
| - The first step is to create an object to store the <b>Selected Utxos</b> and the <b>Change Outputs</b> which will be evaluated repeatedly throughout the algorithm. |
There was a problem hiding this comment.
| - The first step is to create an object to store the <b>Selected Utxos</b> and the <b>Change Outputs</b> which will be evaluated repeatedly throughout the algorithm. | |
| - The aggregator function creates an object to store the **Selected UTxOs** and the **Change Outputs** which will be evaluated repeatedly throughout the algorithm. |
| - <b>Step 1a)</b> | ||
|
|
||
| - The first step is to create an object to store the <b>Selected Utxos</b> and the <b>Change Outputs</b> which will be evaluated repeatedly throughout the algorithm. | ||
| - Next, we need to combined all of our assets from all of the outputs into a single list of assets, these should take into account minted assets as well as a fee buffer which will allow the transaction to work for any fee below the buffer. |
There was a problem hiding this comment.
| - Next, we need to combined all of our assets from all of the outputs into a single list of assets, these should take into account minted assets as well as a fee buffer which will allow the transaction to work for any fee below the buffer. | |
| - All assets from the **Change Outputs** are combined into a single list of assets. This should include minted assets as well as a *Fee Buffer* which will allow the transaction to work for any fee amount less than or equal to the buffer. |
There was a problem hiding this comment.
Shouldn't minted assets be kept separate from the "All Assets" list so that we don't run into an error where we can't select a newly minted NFT (for example) because it does not exist yet? I understand how/why it's needed when generating the change outputs from the transaction but I'm not sure it makes sense to reference it here when it is most likely added later on after input selection is finished if I'm not mistaken.
| Example Definition (C#): | ||
|
|
||
| public void CalculateChange(CoinSelection coinSelection, Balance outputBalance, string changeAddress, ulong feeBuffer = 0) { } | ||
|
|
There was a problem hiding this comment.
| Example Definition (C#): | |
| public void CalculateChange(CoinSelection coinSelection, Balance outputBalance, string changeAddress, ulong feeBuffer = 0) { } | |
| Example Definition (C#): | |
| ```csharp | |
| public void CalculateChange( | |
| CoinSelection coinSelection, | |
| Balance outputBalance, | |
| string changeAddress, | |
| ulong feeBuffer = 0 | |
| ) { } | |
| \``` |
keep formatting consistent with that used for the earlier functions
|
|
||
| The idea behind this function is to create output Utxos with change that limit the size of future transactions by splitting large Utxos or Utxos with a lot of assets. For this we introduce 2 constant parameters that can be adjusted to fit your algorithm needs, the maxChangeOuputs which we set to 4, and the idealMaxAssetsPerOutput which we set to 30. | ||
|
|
||
| The maxChangeOutputs parameter indicates that we have a maximum of 4 change outputs unless we require additional outputs due to those ideal 4 reaching their size limits. The idealMaxAssetsPerOutput is the ideal maximum number of assets in an output Utxo before we create a new output. This value is set to 30 meaning we try to output at max 30 assets in a change output. Having more then 30 is fine but the algorithm attempts to balance Utxos such that it does not exceed 30 if we can avoid it. |
There was a problem hiding this comment.
| The maxChangeOutputs parameter indicates that we have a maximum of 4 change outputs unless we require additional outputs due to those ideal 4 reaching their size limits. The idealMaxAssetsPerOutput is the ideal maximum number of assets in an output Utxo before we create a new output. This value is set to 30 meaning we try to output at max 30 assets in a change output. Having more then 30 is fine but the algorithm attempts to balance Utxos such that it does not exceed 30 if we can avoid it. | |
| The *maxChangeOutputs* parameter indicates that we have a maximum of 4 change outputs unless we require additional outputs due to those ideal 4 reaching their size limits. The *idealMaxAssetsPerOutput* is the ideal maximum number of assets in an output UTxO before we create a new output. This value is set to 30 meaning we try to output at max 30 assets in a change output. Having more then 30 is fine but the algorithm attempts to balance UTxOs such that it does not exceed 30 if we can avoid it. |
These parameters:
- maxChangeOutputs
- idealMaxAssetsPerOutput
Are not actually defined in the given function definition?
|
@NicholasMaselli after your #785 (comment), without any response to abundant review in the last 5 months we have to consider this as (potentially) If this does close after a period of being marked as such, and you become able to work on it further (beginning with addressing all the pending feedback, criticism & requests for clarification), you'd be welcome to re-open it and we'd then enter it into the review process accordingly. |
rphair
added
the
State: Likely Abandoned
|
@Avatar-Nick I can see that you changed your GitHub handle so I'm making one more attempt to ping you before we close this as |
This PR presents a CIP for an optimized random improve algorithm. Recently, Cardano blocks have been at 100% load for several hours creating a need for small transactions. This algorithm implementation has dramatically improved transaction sizes on multiple dapps such as Levvy Finance (https://levvy.fi/) and Saturn NFT (https://saturnnft.io/)
Note: This is the first time I am writing a serious CIP proposal so I apologize if some of the formatting and structure is incorrect.
Below is the abstract and motivation for this CIP:
Abstract
Coin Selection algorithms have become a critical part of the Cardano ecosystem. These algorithms choose a set of input Utxos for a given set of new Outputs.
This CIP aims to present optimized coin selection algorithms to enable Wallets and Dapps to interact with smart contracts and native assets such as tokens and NFTs in a more efficient way. The goal of these algorithms is to primarily reduce the size of transactions as well as reduce the mem and step required for smart contract executions.
Motivation: why is this CIP necessary?
Cardano needs to scale. As the ecosystem grows and new users come in, blocks are regular at 100% load with 88KB per block allocation. Recently, events such as the Lil Sappys NFT mint have pushed the Cardano blockchain load to over 100% for several hours, preventing anyone from using the chain and providing an awful user experience to new users who complain on social media and give the ecosystem a bad reputation.
(rendered version in PR branch)