Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Config: AssetEnabled upgrade #1735

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

gbjk
Copy link
Collaborator

@gbjk gbjk commented Nov 29, 2024

Config:

  • Moves AssetEnabled from being a bool pointer to just being a bool

Exchanges:

  • Rename StoreAssetPairFormat to StoreAssetPairStore
  • StorePairs(enabled) set assetEnabled if pairs list is not empty
  • StorePairs(enabled) add pairs to Available as well
    • This is a convenience for all the test that have to do both, as well as users. Can be dropped if contentious
  • Fix error formatting for DisableAssetWebsocketSupport logging
  • Various assertifications in tests
  • Simplify exchange's defaultPairFormatting handling and make it somewhat consistent.
    • Could do more, but there's a larger overhaul of this coming
  • SetAssetEnabled will not error on no-op (already disabled or enabled)

Bitfinex:

  • Fix tests for MarginFunding subs

Type of change

Please delete options that are not relevant and add an x in [] as item is complete.

  • New feature (non-breaking change which adds functionality)

How has this been tested

  • go test ./... -race
  • golangci-lint run

@gbjk gbjk self-assigned this Nov 29, 2024
@gbjk gbjk requested a review from thrasher- November 29, 2024 06:19
@gbjk gbjk added the review me This pull request is ready for review label Nov 29, 2024
@gbjk gbjk force-pushed the feature/config_assetenabled branch 2 times, most recently from 3697ede to 5fb08d2 Compare November 30, 2024 06:05
@gloriousCode gloriousCode added the nomerge requires dependency This pull request is dependent on another, so it can't be merged until the dependent one is merged label Dec 1, 2024
@gbjk gbjk force-pushed the feature/config_assetenabled branch from 026f88f to 272827a Compare December 2, 2024 01:28
@gbjk gbjk mentioned this pull request Dec 4, 2024
3 tasks
@gbjk gbjk force-pushed the feature/config_assetenabled branch from 942b49b to e2d81d0 Compare December 9, 2024 08:13
@gbjk gbjk removed the nomerge requires dependency This pull request is dependent on another, so it can't be merged until the dependent one is merged label Dec 9, 2024
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick test on gateio, when I set everything but spot assets to false for asset enabled it still fetches and subscribes all assets. Doesn't seem to be happening on master, once resolved I will do a more in-depth check.

@gbjk gbjk force-pushed the feature/config_assetenabled branch from c092bfa to 65c05a0 Compare December 13, 2024 03:28
@gbjk
Copy link
Collaborator Author

gbjk commented Dec 13, 2024

Quick test on gateio, when I set everything but spot assets to false for asset enabled it still fetches and subscribes all assets. Doesn't seem to be happening on master, once resolved I will do a more in-depth check.

Fixed gbjk@65c05a063

@gbjk gbjk force-pushed the feature/config_assetenabled branch from 65c05a0 to 4a4fb99 Compare December 13, 2024 03:44
@gbjk gbjk mentioned this pull request Dec 17, 2024
5 tasks
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great cleanup! tested and runs great. Just some minor nits.

if err != nil {
log.Errorln(log.ExchangeSys, err)
log.Errorf(log.ExchangeSys, "%s error storing `%s` default asset formats: %s", {{.Variable}}.Name, asset.Spot, err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you wrap all these details on the error paths in StoreAssetPairStore? That way we don't need to have this.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole direction here was the opposite of this change request.
The ideology I'm following is that functions return the error they encounter, without their own context.
If you call a go standard library function, you'll get the same thing.
It's down to the caller to decide if it wants to give context to that error, and what that context looks like from it's perspective.
From the perspective of the function caller, they know what they called, and so the error returned should be undecorated.

Case in point: StoreAssetPairStore is not default asset formats so these callers all needed to clarify the context they were calling that function in.

This is why you'll often see me decorate errors one level above where GCT has previously been doing it.
Sometimes it can feel slightly weighty, in this example where we copy the same context everywhere.

We could definitely add a common error for errStoringDefaultAssetFormats though, and I think that's warranted.
I'll circle back to this after the others and see if you've replied :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding one more bit of context:
This implies that functions must also add their own context.
e.g. If StoreAssetPairStore encountered some error interacting with a database, like "Record not found", We'd expect that error to come without context.
So StoreAssetPairStore should add the context error searching for glove size in database: %w.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From an implementation perspective, maintaining this standard can introduce toil during implementation and reviews, especially since any failure in SetDefaults would inherently cause the exchange to malfunction. Differentiating between errors in this context seems redundant. Still suggest wrapping asset and exchange for the exchange method.
But I will leave this open for other peoples perspective. @thrasher- @gloriousCode

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at what both of you saying, I don't think there should be a hard rule about either. There will be times I will absolutely argue for errors returned to contain the data which it is complaining about + some extra context tidbits. Things like "they know what they called" in a large, semi-automated system like GCT doesn't work as well as you'd hope and I wouldn't want that universally applied.

Another issue which depends on the situation is the onus on the caller, so for some functions that would mean decorating the returned error in a large amount of areas of code (which you have done in 11 places) could have just been done in the function once.

Like Shazbert says, the onus is then put on the reviewer to say "hey decorate that error" when a new call is added. If everything relies on reviewers catching people not doing things on top of all the things they are doing, then we are going to get unhelpful, undecorated errors. An unlabelled asset is invalid in the logs is useless which is why we do what we do.

Given that this function is only called within SetDefaults() and gk has copy pasted the same message everywhere, I'm okay with it staying - even if I find it strange to advocate for copy and pasting strings in 11 places. You won't be able to use this text to get out of all future instances 😄

Copy link
Collaborator Author

@gbjk gbjk Feb 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an interesting point about how GCT is a system.
I'd be taking a completely different perspective if GCT was more closed, and the paths to these functions were all private, and callers knew to log it without decoration.
Also: logged with double decoration is better than logged with no context. That's an argument against my change.

Copy link

codecov bot commented Dec 27, 2024

Codecov Report

Attention: Patch coverage is 71.68142% with 64 lines in your changes missing coverage. Please review.

Project coverage is 37.38%. Comparing base (105591b) to head (bfa72e2).
Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
exchanges/exchange.go 44.44% 10 Missing and 15 partials ⚠️
exchanges/binance/binance_wrapper.go 33.33% 4 Missing and 2 partials ⚠️
exchanges/bitfinex/bitfinex_wrapper.go 66.66% 3 Missing and 2 partials ⚠️
exchanges/bitmex/bitmex_wrapper.go 66.66% 3 Missing and 2 partials ⚠️
exchanges/kraken/kraken_wrapper.go 66.66% 3 Missing and 2 partials ⚠️
exchanges/deribit/deribit_wrapper.go 42.85% 2 Missing and 2 partials ⚠️
exchanges/btse/btse_wrapper.go 76.92% 2 Missing and 1 partial ⚠️
exchanges/bybit/bybit_wrapper.go 25.00% 2 Missing and 1 partial ⚠️
exchanges/huobi/huobi_wrapper.go 80.00% 2 Missing and 1 partial ⚠️
exchanges/kucoin/kucoin_wrapper.go 80.00% 2 Missing and 1 partial ⚠️
... and 1 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1735      +/-   ##
==========================================
+ Coverage   37.13%   37.38%   +0.24%     
==========================================
  Files         414      416       +2     
  Lines      180198   178480    -1718     
==========================================
- Hits        66925    66723     -202     
+ Misses     105413   104001    -1412     
+ Partials     7860     7756     -104     
Files with missing lines Coverage Δ
backtester/engine/setup.go 58.59% <100.00%> (+0.70%) ⬆️
config/config.go 87.30% <ø> (+0.25%) ⬆️
config/versions/v3.go 100.00% <100.00%> (ø)
currency/manager.go 93.66% <100.00%> (-0.35%) ⬇️
exchanges/sharedtestvalues/sharedtestvalues.go 76.59% <100.00%> (+11.28%) ⬆️
exchanges/binanceus/binanceus_wrapper.go 41.04% <60.00%> (+0.15%) ⬆️
exchanges/btse/btse_wrapper.go 44.63% <76.92%> (+0.85%) ⬆️
exchanges/bybit/bybit_wrapper.go 46.63% <25.00%> (+3.42%) ⬆️
exchanges/huobi/huobi_wrapper.go 39.60% <80.00%> (-0.37%) ⬇️
exchanges/kucoin/kucoin_wrapper.go 31.86% <80.00%> (+0.12%) ⬆️
... and 6 more

... and 45 files with indirect coverage changes

@gbjk gbjk requested a review from shazbert December 27, 2024 02:52
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current changes tACK. 🚀

@gloriousCode gloriousCode added the szrc shazbert review complete label Dec 29, 2024
@gbjk gbjk force-pushed the feature/config_assetenabled branch from 5293a16 to c1cbeff Compare January 8, 2025 08:27
@gbjk gbjk force-pushed the feature/config_assetenabled branch from c1cbeff to bfa72e2 Compare January 10, 2025 06:53
@gbjk gbjk requested a review from gloriousCode January 10, 2025 06:54
@gbjk
Copy link
Collaborator Author

gbjk commented Jan 10, 2025

I'm going to rebase this onto #1770, because I think that's kinda necessary for Downgrade to be testable.

@gbjk gbjk force-pushed the feature/config_assetenabled branch from bfa72e2 to 65d4423 Compare January 10, 2025 07:10
@gbjk gbjk added the nomerge requires dependency This pull request is dependent on another, so it can't be merged until the dependent one is merged label Jan 18, 2025
@gbjk
Copy link
Collaborator Author

gbjk commented Jan 18, 2025

Clarification: 1770 makes this manually testable for Downgrade, which is what caught @gloriousCode out.
The unit tests work before and after.

@gbjk gbjk mentioned this pull request Jan 24, 2025
15 tasks
@gbjk gbjk removed the nomerge requires dependency This pull request is dependent on another, so it can't be merged until the dependent one is merged label Feb 20, 2025
@gbjk gbjk force-pushed the feature/config_assetenabled branch from 65d4423 to 79a651c Compare February 20, 2025 02:22
Copy link
Collaborator

@gloriousCode gloriousCode left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for the most part. Looking forward to it being rebased so that I can verify with the downgrade bugfix

if err != nil {
log.Errorln(log.ExchangeSys, err)
log.Errorf(log.ExchangeSys, "%s error storing `%s` default asset formats: %s", {{.Variable}}.Name, asset.Spot, err)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at what both of you saying, I don't think there should be a hard rule about either. There will be times I will absolutely argue for errors returned to contain the data which it is complaining about + some extra context tidbits. Things like "they know what they called" in a large, semi-automated system like GCT doesn't work as well as you'd hope and I wouldn't want that universally applied.

Another issue which depends on the situation is the onus on the caller, so for some functions that would mean decorating the returned error in a large amount of areas of code (which you have done in 11 places) could have just been done in the function once.

Like Shazbert says, the onus is then put on the reviewer to say "hey decorate that error" when a new call is added. If everything relies on reviewers catching people not doing things on top of all the things they are doing, then we are going to get unhelpful, undecorated errors. An unlabelled asset is invalid in the logs is useless which is why we do what we do.

Given that this function is only called within SetDefaults() and gk has copy pasted the same message everywhere, I'm okay with it staying - even if I find it strange to advocate for copy and pasting strings in 11 places. You won't be able to use this text to get out of all future instances 😄

@gloriousCode gloriousCode removed the review me This pull request is ready for review label Feb 26, 2025
@gbjk
Copy link
Collaborator Author

gbjk commented Feb 27, 2025

Looks good for the most part. Looking forward to it being rebased so that I can verify with the downgrade bugfix

It was already rebased for you, when I removed restructuring 😄 At least I hope.
But I'll rebase once more to resolve conflicts. And sniff it's still working correctly.

@gbjk gbjk force-pushed the feature/config_assetenabled branch 2 times, most recently from 497c276 to 1827edb Compare February 27, 2025 11:04
@gbjk gbjk requested a review from gloriousCode March 2, 2025 02:18
@gbjk gbjk requested a review from shazbert March 3, 2025 01:10
Copy link
Collaborator

@shazbert shazbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing more from me!

@thrasher- thrasher- added the review me This pull request is ready for review label Mar 4, 2025
@shazbert shazbert added the szrc shazbert review complete label Mar 5, 2025
@gbjk gbjk force-pushed the feature/config_assetenabled branch from d1fb399 to 30ca6a9 Compare March 6, 2025 03:42
@gbjk gbjk force-pushed the feature/config_assetenabled branch 2 times, most recently from d1fb399 to 91e3344 Compare March 6, 2025 05:42
gbjk added 3 commits March 6, 2025 12:50
Previously we were calling it "Format", but accepting everything from
the PairStore.
We were also defaulting to turning the Asset on.

Now callers need to get their AssetEnabled set as they want it, so
there's no magic

This change also moves responsibility for error wrapping outside to the
caller.
Previously we ignored the field and just turned on everything.
I think that was because we couldn't get at the old value.
In either case, we have the option to do better, and respect the
assetEnabled value
@gbjk gbjk force-pushed the feature/config_assetenabled branch from b1821d4 to 9d516e9 Compare March 6, 2025 05:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review me This pull request is ready for review szrc shazbert review complete
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants