Documentation for subxt-cli and windows.

[twhiteheadzm]

I had followed the instructions for using subxt-cli to export the metadata but when I imported the metadata into my project it always showed errors saying it could not understand it. I tried this even with the substrate-node-template with the same results. I eventually tried running subxt-cli in WSL and that seems to have fixed it. So it seems there is some incompatibility with windows. At least temporarily it might be wise to add this to the documentation.

[tadeohepperle]

That is interesting, where exactly did you encounter the error? While downloading the metadata from a node via the CLI or when using it with the subxt macro? What exactly do you mean by imported the metadata into my project?

[niklasad1]

For substrate in general Windows is a tricky to use and please have look at https://docs.substrate.io/install/windows/

To, echo back what @tadeohepperle said please specify which Windows version you are using, command, subxt CLI version and attach the entire backtrack/error you got.

[twhiteheadzm]

Version on both OS's is:
subxt-cli 0.28.0-unknown

I created the files as per instructions:
subxt metadata -f bytes > metadata.scale
Attached are the resulting files from Windows 11 and ubuntu 20.08 running in WSL
metadata.zip

In my project I have:
#[subxt::subxt(runtime_metadata_path = "artifacts/windows_metadata.scale")] pub mod borlaug_subxt {}
and I get:

error: Could not decode metadata, only V14 metadata is supported: Could not decode `RuntimeMetadataPrefixed.1`:
          Could not decode `RuntimeMetadata::V14.0`:
              Could not decode `RuntimeMetadataV14::types`:
                  Could not decode `PortableRegistry::types`:
                      Could not decode `PortableType::ty`:
                          Could not decode `Type::type_def`:
                              Could not decode `TypeDef::Variant.0`:
                                  Could not decode `TypeDefVariant::variants`:
                                      Could not decode `Variant::fields`:
                                          Could not decode `Field::ty`:
                                              Could not decode `UntrackedSymbol::id`:
                                                  out of range decoding Compact<u32>
 --> api\src\lib.rs:42:1
  |
42 | #[subxt::subxt(runtime_metadata_path = "artifacts/windows_metadata.scale")]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: this error originates in the attribute macro `subxt::subxt` (in Nightly builds, run with -Z macro-backtrace for more info)

I tried doing only a single system pallet with similar results (slightly different place where it fails to decode)

[jsdw]

I'd be interested to know; does running subxt metadata -f bytes > metadata.scale on windows produce identical output to running it on linux? If you have WSL, I wonder whether you could md5 metadata.scale and see whether the length in bytes matches for each. (I'm wondering if doing > metadata.scale on windows is screwing with the output in some way)

I also wonder whether you need to reverse the slash in #[subxt::subxt(runtime_metadata_path = "artifacts/windows_metadata.scale")] to play nice with windows, or perhaps try an absolute path or something?

[twhiteheadzm]

I have provided the two outputs attached in my previous comment. They are very close to identical and all the visible text is the same but there are differences in some of the binary characters.

[jsdw]

Ah so you have, apologies for missing that! Let me have a look...

[jsdw]

I did look into this a bit, and can see various small differences between the metadatas generated. Since the CLI tool in each case is just decoding the bytes via parity-scale-codec and then writing those bytes to stdout, I have no good idea as to why this difference would exist; none of these libraries to my knowledge would have any windows-specific behaviours.

It'd be interesting to work out whether the error is in the actual decoding or the writing of bytes to disk (I've been assuming windows is mangling something in the latter but I have no easy way to test this).

Can I also confirm that both metadatas were generated from the exact same running node (running on the same OS etc too)? I assume so but just want to rule out any possibility of the node sending something different.

[twhiteheadzm]

Can I also confirm that both metadatas were generated from the exact same running node (running on the same OS etc too)?

Yes, the node was running in WSL on the same machine and I ran the command on both OS's one after the other without restarting the node.

[jsdw]

@lexnv / @niklasad1 / @tadeohepperle I don't suppose one of you has a copy of windows somewhere that you could try this on? It'd be good to figure out where the issue is coming from at least :)

[lexnv]

Hi,

I did manage to run this experiment on a Windows box.

I entered the CLI directory and run the following command:

cargo run metadata --url wss://rpc.polkadot.io:443 > ../artifacts/windows_metadata.scale

The only difference here from your input is that I am targeting the official "wss://rpc.polkadot.io:443" endpoint, instead of a local running substrate chain on Windows.

I did modify the constants_static.rs example to point out to my newly download metadata and to target the chain:

use subxt::{OnlineClient, PolkadotConfig};

#[subxt::subxt(runtime_metadata_path = "../artifacts/windows_metadata.scale")]
pub mod polkadot {}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a client to use:
    // Note: this is targeting "wss://rpc.polkadot.io:443", not the local node.
    let api = OnlineClient::<PolkadotConfig>::from_url("wss://rpc.polkadot.io:443").await?;

    // A query to obtain some contant:
    let constant_query = polkadot::constants().system().block_length();

    // Obtain the value:
    let value = api.constants().at(&constant_query)?;

    println!("Block length: {value:?}");
    Ok(())
}

The result is as expected

Block length: BlockLength { max: PerDispatchClass { normal: 3932160, operational: 5242880, mandatory: 5242880 } }

I did this both in WSL and plain windows and I used the following instructions to install Rust:
https://learn.microsoft.com/en-us/windows/dev-environment/rust/setup

This might be an issue related with substrate itself, and I believe we recommend running the node on linux based machines. As Niklas mentioned, there might be some issues related to compatibility in Windows platform.

[twhiteheadzm]

Just tried the polkadot url as suggested above. The only difference is I am using the installed subxt cli version subxt-cli 0.28.0-unknown

subxt metadata --url wss://rpc.polkadot.io:443 > windows-polkadot-metadata.scale

The resulting files on Windows and WSL have some byte differences

scale files polkadot.zip

I have not tried to use them but for my own chain I am using the .scale file generated in WSL and using it in my subxt based project in windows without issues, but the scale file generated in windows does not work in either windows or WSL.

[jsdw]

@lexnv did you spot any byte differences when you tried to fetch the metadata via windows versus linux (while running the actual node on linux)?

[lexnv]

There is a difference in the metadata format.

The SHA256 differs from the metadata fetched from the polkadot chain:

• Windows: 541EC9B99C0514D1028BDC477CF1DDCEC7E71D7B631EDC4C1ABCBC43FBA4B1EC

From the commit fdc0d09 of subxt:

• Linux: dff8a26ba4838d625377262e2572075aa51bf23744e43efd7d7b1e0098686c25
• Macos: dff8a26ba4838d625377262e2572075aa51bf23744e43efd7d7b1e0098686c25

[tadeohepperle]

@lexnv I installed subxt on both windows and linux in the same way to make sure the same version of subxt is running, with cargo install subxt-cli.
I ran these commands on windows and linux respectively and compared the files:

# run on windows:
subxt metadata --url wss://rpc.polkadot.io:443 -f hex > polkadot_remote_windows.hex
subxt metadata --url wss://rpc.polkadot.io:443 -f bytes > polkadot_remote_windows.scale
subxt metadata --url wss://rpc.polkadot.io:443 -f json > polkadot_remote_windows.json


# run on linux:
subxt metadata --url wss://rpc.polkadot.io:443 -f hex > polkadot_remote_linux.hex
subxt metadata --url wss://rpc.polkadot.io:443 -f bytes > polkadot_remote_linux.scale
subxt metadata --url wss://rpc.polkadot.io:443 -f json > polkadot_remote_linux.json

# compare them:
md5sum polkadot_remote_windows.hex polkadot_remote_linux.hex
md5sum polkadot_remote_windows.scale polkadot_remote_linux.scale
md5sum polkadot_remote_windows.json polkadot_remote_linux.json

# run on windows:
subxt metadata --file ../subxt/artifacts/polkadot_metadata_tiny.scale -f hex > polkadot_tiny_windows.hex
subxt metadata --file ../subxt/artifacts/polkadot_metadata_tiny.scale -f bytes > polkadot_tiny_windows.scale
subxt metadata --file ../subxt/artifacts/polkadot_metadata_tiny.scale -f json > polkadot_tiny_windows.json

# run on linux:
subxt metadata --file ../subxt/artifacts/polkadot_metadata_tiny.scale -f hex > polkadot_tiny_linux.hex
subxt metadata --file ../subxt/artifacts/polkadot_metadata_tiny.scale -f bytes > polkadot_tiny_linux.scale
subxt metadata --file ../subxt/artifacts/polkadot_metadata_tiny.scale -f json > polkadot_tiny_linux.json

# compare them:
md5sum polkadot_tiny_windows.hex polkadot_tiny_linux.hex
md5sum polkadot_tiny_windows.scale polkadot_tiny_linux.scale
md5sum polkadot_tiny_windows.json polkadot_tiny_linux.json

Both local and remote metadata show the same resulting files. I think the difference encountered before might have been subxt version related?

[lexnv]

I've targeted a local running polkadot node on my Mac machine and fetched the metadata.

Indeed the metadata presents differences when using PowerShell on windows, although the linux subsystem outputs the information right.

This is related to how Windows redirects output from stdout to a file.

To mitigate the differences between Linux and Windows, I've created a path: #1018.

To ensure consistency between platforms users can point the metadata command to an output-file and let the subxt do the writing.

[jsdw]

Thankyou @tadeohepperle and @lexnv for digging into this :)