Optimize user state queries using new API endpoints

[dib542]

This work should follow on from the changes made to use the user state endpoints in:

• Update user state queries to new API endpoints for v0.4.1 #488 (hopefully completed at the point of Duality API v0.3.3)

Discussion

With the introduction of the endpoint /dualitylabs/duality/dex/pool/{pairID}/{tickIndex}/{fee} in duality-labs/duality#446 (which is included in Duality v0.4.0)

User queries from the front end can now look up their own data from

• /dualitylabs/duality/dex/user/positions/{address}
• /dualitylabs/duality/dex/user/deposits/{address}
• /dualitylabs/duality/dex/user/limit_orders/{address}

and use these known center tick indexes and fees to find the reserves of both lower_tick0 and upper_tick1 of a center tick at the same time (and avoid having a 404 not found error on empty upper or lower tick requests).

This should help the front end:

• reduce the number of queries made to find a user's current position worth (in terms of absolute reserves and USD)
• ensure that the calculated user reserves in the app is accurate

Unexpected developments

When requesting this feature I was asking for (importantly this data is from the same block height) an endpoint to return a pool's

• lowerTickReserves
• upperTickReserves
• totalShares

This I wanted to combine with the user data of /dualitylabs/duality/dex/user/deposits/{address} which returns the user's shares in sharesOwned, to calculate a user's holding in terms of reserves, ie.

• userReserves0 = (sharesOwned / totalShares) * lowerTickReserves
• userReserves1 = (sharesOwned / totalShares) * upperTickReserves

However the new endpoint returns only

message QueryPoolResponse {
	Pool pool = 1 [(gogoproto.nullable) = true];
}

message Pool {
    PoolReserves lower_tick0 = 1;
    PoolReserves upper_tick1 = 2;
}

Which does not have enough information to piece together the amount of reserves that a user holds at a certain block in time. Because the requests for this information must still be made in at least two separate requests we cannot guarantee that the data is from the same height, so the calculated user reserves remain an estimation.

Conclusion

So while this endpoint does serve to reduce the number of network requests from 3 to 2 per user deposit, It is not particularly helpful.

requests used previously per user deposit:

• totalShares from /cosmos/bank/v1beta1/supply/{denom}
• lowerReserves from /dualitylabs/duality/dex/pool_reserves/{pairID}/{tokenIn}/{tickIndex}/{fee}
• upperReserves from /dualitylabs/duality/dex/pool_reserves/{pairID}/{tokenIn}/{tickIndex}/{fee}

requests used after new endpoint per user deposit:

• totalShares from /cosmos/bank/v1beta1/supply/{denom}
• lowerReserves and upperReserves from /dualitylabs/duality/dex/pool/{pairID}/{tickIndex}/{fee}

The endpoint as is may be used to reduce network calls, but I don't think this is a priority compared to other issues such as the guarantee that this calculation is accurate by guaranteeing that the data is from the same block height.

[dib542]

Suggestions for improvements to the API:

New logic

On the app I would like to make a distinction between "accurateUserReserves" and "estimatedUserReserves" with an implementation of maybe both using React hooks.

Accurate user reserves

"accurateUserReserves" is for pages that require accurate detail such as the Pool Management pages with show the user deposits in real-time.

"accurateUserReserves" would require knowing the users reserves calculated correctly at each block, as we currently lack this same-block guarantee, accurate reserves cannot be calculated.

Estimated user reserves

"estimatedUserReserves" can be for pages that do not show individual user deposits but aggregated user deposit values such as the Portfolio page which sums a user's reserve value in each pair.

"estimatedUserReserves" requires knowing the number of user's reserves (exact or approximate) at a recent point in time, and then assuming what the reserves are at the current point in time assuming that the token pair pools are perfectly arbitraged at the current point in time (which is likely not true but the calculated result should be approximately correct for pools with enough liquidity)

Indicative user reserves

both "accurateUserReserves" and "estimatedUserReserves" can be calculated from a new concept "indicativeUserReserves" which contains for each pool:

• pairID
• tickIndex
• fee
• indicativeReservesAsToken0
• indicativeReservesAsToken1

where the indicativeReserves here is the pre-calculated (userReserves = (sharesOwned / totalShares) * tickReserves) value of the users shares if they were wholly as reserves0 or reserves1.

Estimated user reserves calculations

Using indicativeReserves values and the real-world (ie. third party CoinGecko API) current price of the tokens, the front end can make an estimation of the user's reserves and their monetary value by using the assumption that the the pair price reflects the real world price, and that there are no "behind enemy lines" deposits (ie. that the token pair is perfectly arbitraged). By knowing the token prices and determining the tickIndex that assumedly separates reserves from the token0 side and token1 side, the reserves of each side can be estimated.

Accurate user reserves calculations

Using accurate indicativeReserves values and real-time liquidity value provided by the indexer on /liquidity/pair/{tokenA}/{tokenB} the proportion of reserves for each pool of a pair is known for each block and therefore the accurate number of user's reserves can be known at each block.

Indicative user reserves calculations

"indicativeReserves" can be calculated from userReserves = (sharesOwned / totalShares) * tickReserves at any point in time (but will only be "accurate" when totalShares and tickReserves are fetched from the same block height as they may change every/any block height, sharesOwned only changes with user actions).

These values can be currently collected from:

• sharesOwned from /cosmos/bank/v1beta1/balances/{address}
  • this could be fetched from /dualitylabs/duality/dex/user/deposits/{address} if poolID was returned on DepositRecord, but it is not so this is not recommend after the update of duality-labs/duality@9ede0f2
• totalShares from /cosmos/bank/v1beta1/supply/{denom}
• lowerReserves from /dualitylabs/duality/dex/pool_reserves/{pairID}/{tokenIn}/{tickIndex}/{fee}
• upperReserves from /dualitylabs/duality/dex/pool_reserves/{pairID}/{tokenIn}/{tickIndex}/{fee}

but if would be better if they could be collected from one place, allowing the app to have an "accurate" (numbers from the same height) version of the user's indicative reserves:

• The endpoint /duality/dex/user/deposits/{address}:
  • could add either a calculated indicativeReservesAsToken0 and indicativeReservesAsToken1 directly
  • or list all the properties required to calculated it all at once, i.e the endpoint should return for each pool all of
    • sharesOwned
    • totalShares
    • lowerReserves0
    • upperReserves1
  • the difference is that the first option returns the same values whenever called (if the user did not do an action), and the second option can change with the actions of other users trading on the pools that the user has deposits in. so the first option is more REST-like from the user's deposits point of view, and the second option is more REST-like from the specific pools point of view.
• the endpoints /duality/dex/pool/{poolID} and/or /duality/dex/pool/{pairID}/{tickIndex}/{fee}:
  • could return totalShares on QueryPoolResponse or the Pool type
  • this would provide enough information for each pool in question
  • but of course this mean that instead of fetching one just endpoint for all the user's deposit data, this data must be fetched for each user's deposit (which may be a significant number) after fetching the user's deposit pools from /duality/dex/user/deposits/{address}
    • /duality/dex/user/deposits/{address} should return poolID if only the /duality/dex/pool/{poolID} endpoint will contain the totalShares information.
    • it seems plausible to fetch all of a user's dex pools from the user's balances: /cosmos/bank/v1beta1/balances/{address} but this endpoint will no longer give an indication of which pools belong to which token pair (without fetching metadata for each pool)
• due to the number of possible network requests I think adding information to the /duality/dex/user/deposits/{address} would be more beneficial.
• A new endpoint /duality/dex/user/deposits/{address}/{poolID}
  • could allow the app to fetch and update in its store the indicativeReserves for a single pool. which would be a typical request after the users makes a Dex action that causes a bank transfer action to occur against their wallet address.
  • this is not really needed if we don't add indicativeReserves to the /duality/dex/user/deposits/{address} endpoint. If we add totalShares there instead the required data to update the app state for a single pool will be available on the /duality/dex/pool/{poolID} and/or /duality/dex/pool/{pairID}/{tickIndex}/{fee} endpoints if the totalShares information is added there.

Conclusion

Therefore I think my recommendation is to add:

• to the Pool type (https://github.com/neutron-org/neutron/blob/main/proto/neutron/dex/pool.proto):
  • totalShares
  • it already has lowerReserves0 and upperReserves1
  • this will make the Pool type suitable for calculating accurate indicative reserves for the app
• to endpoint /duality/dex/user/deposits/{address}:
  • pool (with new totalShares info)

This allows the app to

• request /duality/dex/user/deposits/{address} on app start / address change
  • to get the entire user's indicativeReserves (and therefore future accurateUserReserves and estimatedUserReserves) in one network request.
• then request /duality/dex/pool/{poolID} upon any bank.transfer updates to the user's bank
  • to update relevant portions of the user's indicativeReserves

Additionally for other applications: the ability to continually fetch /duality/dex/user/deposits/{address} whenever to get an accurate total of the user's reserves at the current point in time could be very helpful in building simple applications.

[dib542]

The previous comment has been expanded out into separate issues:

• Optimization: add an estimatedUserReserves state to know approximate user reserves #492
• Improve accuracy of userReserves calculations #493