-
Notifications
You must be signed in to change notification settings - Fork 262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Static Metadata Validation #478
Merged
Merged
Changes from 12 commits
Commits
Show all changes
173 commits
Select commit
Hold shift + click to select a range
12313d7
metadata: Implement MetadataHashable for deterministic hashing
lexnv 707a5e5
metadata: Hash `scale_info::Field`
lexnv 3353583
metadata: Hash `scale_info::Variant`
lexnv ebc588a
metadata: Hash `scale_info::TypeDef`
lexnv dc727fc
metadata: Hash pallet metadata
lexnv 4690ac0
metadata: Avoid data representation collision via unique identifiers
lexnv 8178351
metadata: Finalize hashing on recursive types
lexnv b60ebdf
metadata: Cache recursive calls
lexnv a9b1447
metadata: Move `MetadataHashable` to codegen to avoid cyclic dependency
lexnv 3ece9e3
codegen: Add pallet unique hash
lexnv c64cc60
metadata: Wrap metadata as owned
lexnv a24da9a
subxt: Use MetadataHashable wrapper for clients
lexnv 05f1bc8
subxt: Generate runtime pallet uid from metadata
lexnv b98a93c
Validate metadata compatibility at the pallet level
lexnv 73befba
Update polkadot.rs
lexnv e576ab2
Modify examples and tests for the new API
lexnv 76f2c9c
codegen: Implement metadata uid
lexnv 58f743c
Update polkadot with TryFrom implementation
lexnv ea9e233
client: Change `to_runtime_api` to reflect TryFrom changes
lexnv f9d9956
client: Skip full metadata validation option
lexnv 0ed1f43
codegen: Add option to skip pallet validation for TransactionApi
lexnv f5665f7
codegen: Add option to skip pallet validation for StorageApi
lexnv b36555f
Update polkadot.rs with ability to skip pallet validation
lexnv e8e8792
codegen: Change `MetadataHashable` to per function implementation
lexnv 86b1c7e
codegen: Use metadata hashes functions
lexnv c0ecf32
subxt: Use metadata hashes functions
lexnv 4431c22
codegen: Make `get_type_uid` private
lexnv 1fa9db4
codegen, subxt: Rename metadata functions `*_uid` to `*_hash`
lexnv 2d6e906
codegen: Update `get_field_hash` to use `codec::Encode`
lexnv 4da2e0e
subxt: Update polkadot.rs
lexnv ce7a6d7
codegen, subxt: Move metadata check from client to subxt::Metadata
lexnv 7033233
codegen, subxt: Rename metadata check functions to follow `*_hash` na…
lexnv ed51cd8
codegen: Update polkadot.rs to reflect naming changes
lexnv 0d500eb
codegen: Use `encode_to` for metadata generation
lexnv 1c2f986
codegen: Update polkadot.rs to reflect `encode_to` changes
lexnv 02bb231
codegen: Specific name for visited set
lexnv 13e6112
metadata: Provide cache to hashing functions
lexnv d388598
metadata: Compute metadata hash by sorted pallets
lexnv 5e6a21d
metadata: Get extrinsic hash
lexnv 45c6453
metadata: Extend metadata hash with extrinsic and metadata type
lexnv 5eb41c9
metadata: Add cache as metadata parameter
lexnv e33f33a
codegen, subxt: Update metadata hash to use cache
lexnv c5d89a3
metadata: Implement Default trait for MetadataHasherCache
lexnv e7af28f
metadata: Add cache for pallets
lexnv e0779bd
metadata: Move functionality to metadata crate
lexnv 953c031
codegen, subxt: Use subxt-metadata crate
lexnv b865f18
codegen: Remove metdata hashing functionality
lexnv 26216fa
metadata: Add documentation
lexnv aad7d2d
metadata: Fix vector capacity to include extrinisc and type hash
lexnv 5de8e83
metadata: Add empty CLI
lexnv 3877505
metadata-cli: Fetch metadata from substrate nodes
lexnv 1809cc7
metadata-cli: Log metadata hashes of provided nodes
lexnv 1957e56
metadata-cli: Group compatible nodes by metadata
lexnv 9e32361
metadata-cli: Simplify hash map insertion
lexnv 70aa7c1
metadata-cli: Move full metadata check to function
lexnv bfa9c8f
metadata-cli: Group metadata validation at the pallet level
lexnv c7a1e27
subxt: Persist metadata cache
lexnv ca8c22c
metadata: Move compatibility cli from subxt-metadata to subxt-cli
lexnv 776250e
metadata: Remove cli from subxt-metadata
lexnv 1c600ee
cli: Fix clippy
lexnv 95a6134
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv beee3f1
metadata: Fix compatible metadata when pallets are registered in diff…
lexnv 84e049e
tests: Handle result of pallet hashing
lexnv 6635423
metadata: Remove type cache for deterministic hashing
lexnv 1fe787d
metadata: Add test assets from `substrate-node-template` tag `polkado…
lexnv 20055e5
metadata-tests: Check cache hashing for Balances pallet
lexnv 7a9b8e5
metadata: Fix `get_type_hash` clippy issue
lexnv 931fcdb
metadata-tests: Compare one time cache with persistent cache
lexnv c9ed131
metadata-test: Check metadata hash populates cache for pallets
lexnv 43ae4db
metadata-tests: Simplify `cache_deterministic_hash` test
lexnv f9ca06d
metadata-tests: Check deterministic metadata for different order pallets
lexnv 09e74d2
metadata: Fix clippy
lexnv 57771e8
codegen: Implement TransactionApiUnchecked for skipping pallet valida…
lexnv f3f4d72
codegen: Implement StorageApiUnchecked for skipping pallet validation
lexnv 9e4c62c
codegen: Remove skip_pallet_validation boolean
lexnv f92383c
subxt: Implement ClientUnchecked for skipping metadata validation
lexnv 07cf5f2
Update polkadot.rs
lexnv 6baa8e8
Update examples of rpc_call to skip metadata
lexnv 8378086
subxt: Remove heck dependency
lexnv 5fa5df8
codegen: Add pallet name as an identifier for pallet hashing
lexnv 8da1497
metadata: Implement MetadataHashDetails
lexnv 62a7eec
metadata: Adjust testing to `MetadataHashDetails` interface
lexnv f026078
codegen: Remove extra `pallet_name`
lexnv 31c016c
subxt: Update polkadot.rs
lexnv 8fcf824
subxt: Fix clippy issue
lexnv 816e916
codegen: Change StorageApi to support `_unchecked` methods
lexnv 58fd3e3
codegen: Change TransactionApi to support `_unchecked` methods
lexnv e0ef532
subxt: Switch back from `TryFrom` to `From` for `subxt::Client`
lexnv b0061e4
codegen, subxt: Remove `ClientUnchecked`
lexnv 5ecf7b7
codegen: Expose `validate_metadata` as validation of compatibility me…
lexnv 392da6d
examples: Update to the new interface
lexnv f2e8dfc
subxt: Update test integration to latest interface
lexnv f8df12a
subxt: Update polkadot.rs
lexnv 467d707
metadata/tests: Check different pallet index order
lexnv d20bcd0
metadata/tests: Check recursive type hashing
lexnv e042c05
metadata/tests: Check recursive types registered in different order
lexnv b3c4029
metadata/tests: Fix recursive types warning
lexnv 1900091
metadata/tests: Remove test assets
lexnv 252c4d1
metadata/tests: Extend tests to verify cached pallet values
lexnv 964f92c
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv 9278811
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv 551c595
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv db8fdfb
examples: Add metadata compatiblity example
lexnv d25c912
examples: Revert balance_transfer to initial form
lexnv 5fb0cdd
codegen: Add ConstantsApi metadata check
lexnv 7293531
tests: Modify tests to accomodate ConstantsApi changes
lexnv a5415b8
examples: Modify verified version
lexnv ce05435
subxt: Generate polkadot.rs from `0.9.18-4542a603cc-aarch64-macos`
lexnv ba95ac8
examples: Update polkadot_metadata.scale from `0.9.18-4542a603cc-aarc…
lexnv 1efd776
metadata: Update documentation
lexnv dc3b106
tests: Modify default pallet usage
lexnv bb0b838
metadata/tests: Remove hex dependency
lexnv 3a2d695
metadata/tests: Add MetadataTestType to capture complex types
lexnv 853ba6f
metadata/tests: Update tests to use complex types
lexnv f0a2b75
metadata/tests: Check metadata correctness via extending pallets
lexnv 545dfd0
metadata/tests: Extend pallet hash with Events
lexnv 63b32bc
metadata/tests: Extend pallet hash with constants
lexnv 7d55527
metadata/tests: Extend pallet hash with error
lexnv 4e30b52
examples: Extend metadata compatibiliy with StorageApi and ConstantsApi
lexnv dab2beb
Modify comments and documentation
lexnv a152c6b
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv 4af15fe
metadata: Benchmarks for full validation and pallet validation
lexnv 74d3bc8
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv 58cbc3f
metadata/benches: Fix clippy
lexnv efa5df1
metadata: Hash metadata just by inspecting the provided pallets
lexnv 414dee6
metadata: Make pallets generic over T for `AsRef<str>`
lexnv 7396595
codegen: Expose the name of the pallets composing the metadata
lexnv 7d1adc3
subxt/tests: Update polkadot.rs with pallets name
lexnv 1f5c5f4
codegen: Obtain metadata hash only by inspecting pallets
lexnv cbc3727
codegen,subxt: Extend the metadata hash to utilize just pallets
lexnv a82fad2
subxt/tests: Update polkadot.rs with client metadata has per pallet
lexnv c96606a
metadata/tests: Test `get_metadata_per_pallet_hash` correctness
lexnv c558df4
metadata: Fix clippy
lexnv a6f5650
metadata/benches: Fix decode of metadata
lexnv 8f57e20
metadata: Fix clippy
lexnv 23b382f
[static metadata] validate storage, calls and constants per call (#507)
jsdw 9a08a84
Fix clippy
lexnv 6cbebc9
bench the per-call metadata functions
jsdw fd4d425
metadata: Add test for `node_template_runtime_variant`
lexnv 640b052
ensure criteron cli opts work
jsdw 1323696
group benchmarks and avoid unwrap issues
jsdw 9a7bbbc
metadata: Check template runtime for handling the pallet swap order case
lexnv 5548b5a
Merge remote-tracking branch 'origin/398_static_md_check' into 398_st…
lexnv fd00319
metadata: Remove debug logs
lexnv a0e3b6c
metadata: Optimise by removing field's name and type_name and type's …
lexnv 0ce19bb
metadata: Refactor `get_type_hash` to break recursion earlier
lexnv eb051a9
subxt: Add tests for `hash_cache`
lexnv cb2de4a
subxt: Add tests for checking Metadata Inner cache
lexnv e4e2866
metadata: Check semantic changes inside enum and struct fields
lexnv f65dabc
metadata: Add enums named differently with compatible semantic meaning
lexnv 6c776eb
metadata: Guard testing of release versions for `node_template_runtime`
lexnv 9c46526
Improve documentation
lexnv b713993
Update polkadot.rs
lexnv 2d55752
metadata/tests: Manually construct type of `node_template_runtimeL::C…
lexnv 79ea3fe
no more special Call handling, avoid a little cloning, and actually s…
jsdw a1f9038
remove unused deps and fmt
jsdw 9691f51
RuntimeMetadataLastVersion => RuntimeMetadataV14
jsdw fd7aa62
remove a bunch of allocations in the metadata hashing, speed up from …
jsdw a658d6f
update release docs to release metadata crate too
jsdw 06241f4
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv 8c4aca6
subxt: Remove codegen dependency
lexnv 0f7a770
subxt: Replace std RwLock with parking_lot
lexnv e355ec6
subxt/tests: Add ws address to `TestNodeProcess`
lexnv 73be46c
subxt/tests: Add metadata validation integration test
lexnv b4c0185
subxt: Allow setting metadata on the ClientBuilder
lexnv 2a625a9
subxt/tests: Check incompatible metadatas
lexnv 633c3f0
metadata: Fix constant hashing for deterministic output
lexnv 542dec6
subxt/tests: Check metadata validation for constants
lexnv 74be005
subxt/tests: Test validation for calls
lexnv 83f0ba4
subxt/tests: Test validation for storage
lexnv 1abd3ca
Merge remote-tracking branch 'origin/master' into 398_static_md_check
lexnv b0e8b4c
subxt: Expose `set_metadata` for testing only
lexnv 460c18c
subxt: Guard metadata tests under integration-tests
lexnv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||||
---|---|---|---|---|---|---|---|---|
|
@@ -52,6 +52,7 @@ use derivative::Derivative; | |||||||
pub struct ClientBuilder { | ||||||||
url: Option<String>, | ||||||||
client: Option<RpcClient>, | ||||||||
metadata: Option<Metadata>, | ||||||||
page_size: Option<u32>, | ||||||||
} | ||||||||
|
||||||||
|
@@ -61,6 +62,7 @@ impl ClientBuilder { | |||||||
Self { | ||||||||
url: None, | ||||||||
client: None, | ||||||||
metadata: None, | ||||||||
page_size: None, | ||||||||
} | ||||||||
} | ||||||||
|
@@ -83,6 +85,14 @@ impl ClientBuilder { | |||||||
self | ||||||||
} | ||||||||
|
||||||||
/// Set the metadata. | ||||||||
/// | ||||||||
/// *Note:* Metadata will no longer be downloaded from the runtime node. | ||||||||
pub fn set_metadata(mut self, metadata: Metadata) -> Self { | ||||||||
self.metadata = Some(metadata); | ||||||||
self | ||||||||
} | ||||||||
|
||||||||
/// Creates a new Client. | ||||||||
pub async fn build<T: Config>(self) -> Result<Client<T>, BasicError> { | ||||||||
let client = if let Some(client) = self.client { | ||||||||
|
@@ -92,14 +102,19 @@ impl ClientBuilder { | |||||||
crate::rpc::ws_client(url).await? | ||||||||
}; | ||||||||
let rpc = Rpc::new(client); | ||||||||
let (metadata, genesis_hash, runtime_version, properties) = future::join4( | ||||||||
rpc.metadata(), | ||||||||
let (genesis_hash, runtime_version, properties) = future::join3( | ||||||||
rpc.genesis_hash(), | ||||||||
rpc.runtime_version(None), | ||||||||
rpc.system_properties(), | ||||||||
) | ||||||||
.await; | ||||||||
let metadata = metadata?; | ||||||||
|
||||||||
let metadata = if let Some(metadata) = self.metadata { | ||||||||
metadata | ||||||||
} else { | ||||||||
let metadata = rpc.metadata().await; | ||||||||
metadata? | ||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||
}; | ||||||||
|
||||||||
Ok(Client { | ||||||||
rpc, | ||||||||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately,
cfg(test)
is not active for integration tests. I would place this underintegration_tests
feature and follow up with #515 😄 .