Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XML deserialization error during simple list blob #1092

Closed
mdrach opened this issue Sep 13, 2022 · 1 comment · Fixed by #1097
Closed

XML deserialization error during simple list blob #1092

mdrach opened this issue Sep 13, 2022 · 1 comment · Fixed by #1097

Comments

@mdrach
Copy link

mdrach commented Sep 13, 2022

Hello, I'm writing a basic Azure client and can't seem to list blobs from a container without running into an xml parsing issue:

Here's the error:

RUST_BACKTRACE=1 cargo run
   Compiling repro-storage-blobs-issue v0.1.0 (/Users/max/sisu/projects/azure_storage_blobs_issue/repro-storage-blobs-issue)
    Finished dev [unoptimized + debuginfo] target(s) in 2.41s
     Running `target/debug/repro-storage-blobs-issue`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { context: Full(Custom { kind: DataConversion, error: Custom { field: "duplicate field `Blob`" } }, "failed to deserialize the following xml into a azure_storage_blobs::container::operations::list_blobs::ListBlobsResponseInternal\n\u{feff}<?xml version=\"1.0\" encoding=\"utf-8\"?><EnumerationResults ServiceEndpoint=\"https://sisuautomatedtest.blob.core.windows.net/\" ContainerName=\"lowlatencyrequests\"><Prefix>get-most-recent-key-5/</Prefix><Delimiter>/</Delimiter><Blobs><Blob><Name>get-most-recent-key-5/2021-08-04-testfile1</Name><Properties><Creation-Time>Tue, 13 Sep 2022 08:20:48 GMT</Creation-Time><Last-Modified>Tue, 13 Sep 2022 08:20:48 GMT</Last-Modified><Etag>0x8DA9560DD170CFD</Etag><Content-Length>19</Content-Length><Content-Type>text/plain</Content-Type><Content-Encoding /><Content-Language /><Content-CRC64 /><Content-MD5>3X/+gWTy92gIJFXx57gLYA==</Content-MD5><Cache-Control /><Content-Disposition /><BlobType>BlockBlob</BlobType><AccessTier>Hot</AccessTier><AccessTierInferred>true</AccessTierInferred><LeaseStatus>unlocked</LeaseStatus><LeaseState>available</LeaseState><ServerEncrypted>true</ServerEncrypted></Properties><OrMetadata /></Blob><BlobPrefix><Name>get-most-recent-key-5/2021-08-04T21:48:48.592953Z-15839722113750148182/</Name></BlobPrefix><Blob><Name>get-most-recent-key-5/2021-09-04-testfile2</Name><Properties><Creation-Time>Tue, 13 Sep 2022 08:07:01 GMT</Creation-Time><Last-Modified>Tue, 13 Sep 2022 08:19:21 GMT</Last-Modified><Etag>0x8DA9560A916932D</Etag><Content-Length>19</Content-Length><Content-Type>text/plain</Content-Type><Content-Encoding /><Content-Language /><Content-CRC64 /><Content-MD5>b0CPJB6eDfKUzzW7dlboKQ==</Content-MD5><Cache-Control /><Content-Disposition /><BlobType>BlockBlob</BlobType><AccessTier>Hot</AccessTier><AccessTierInferred>true</AccessTierInferred><LeaseStatus>unlocked</LeaseStatus><LeaseState>available</LeaseState><ServerEncrypted>true</ServerEncrypted></Properties><OrMetadata /></Blob><Blob><Name>get-most-recent-key-5/2022-08-04-testfile3</Name><Properties><Creation-Time>Tue, 13 Sep 2022 08:07:01 GMT</Creation-Time><Last-Modified>Tue, 13 Sep 2022 08:19:21 GMT</Last-Modified><Etag>0x8DA9560A91F9296</Etag><Content-Length>34</Content-Length><Content-Type>text/plain</Content-Type><Content-Encoding /><Content-Language /><Content-CRC64 /><Content-MD5>1F1MssyZOvhY4OZevHWEsw==</Content-MD5><Cache-Control /><Content-Disposition /><BlobType>BlockBlob</BlobType><AccessTier>Hot</AccessTier><AccessTierInferred>true</AccessTierInferred><LeaseStatus>unlocked</LeaseStatus><LeaseState>available</LeaseState><ServerEncrypted>true</ServerEncrypted></Properties><OrMetadata /></Blob></Blobs><NextMarker /></EnumerationResults>") }', src/main.rs:26:21
stack backtrace:
   0: rust_begin_unwind
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
   2: core::result::unwrap_failed
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/result.rs:1805:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/result.rs:1098:23
   4: repro_storage_blobs_issue::main::{{closure}}
             at ./src/main.rs:26:17
   5: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/future/mod.rs:91:19
   6: tokio::park::thread::CachedParkThread::block_on::{{closure}}
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/park/thread.rs:267:54
   7: tokio::coop::with_budget::{{closure}}
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/coop.rs:102:9
   8: std::thread::local::LocalKey<T>::try_with
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/local.rs:445:16
   9: std::thread::local::LocalKey<T>::with
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/local.rs:421:9
  10: tokio::coop::with_budget
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/coop.rs:95:5
  11: tokio::coop::budget
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/coop.rs:72:5
  12: tokio::park::thread::CachedParkThread::block_on
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/park/thread.rs:267:31
  13: tokio::runtime::enter::Enter::block_on
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/runtime/enter.rs:152:13
  14: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/runtime/scheduler/multi_thread/mod.rs:79:9
  15: tokio::runtime::Runtime::block_on
             at /Users/max/.cargo/registry/src/github.jparrowsec.cn-1ecc6299db9ec823/tokio-1.21.0/src/runtime/mod.rs:492:44
  16: repro_storage_blobs_issue::main
             at ./src/main.rs:25:5
  17: core::ops::function::FnOnce::call_once
             at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Here's the code I'm running:

use azure_storage::prelude::*;
use azure_storage_blobs::prelude::*;

use futures::stream::StreamExt;

const ACCOUNT: &str = "<redacted>";
const KEY: &str = "<redacted>";
const CONTAINER: &str = "<redacted>";

#[tokio::main]
async fn main() {
    let container_client = ClientBuilder::new(
        ACCOUNT,
        StorageCredentials::access_key(ACCOUNT.to_string(), KEY.to_string()),
    )
    .container_client(CONTAINER);

    let mut stream = container_client
        .list_blobs()
        .prefix("get-most-recent-key-5/")
        .delimiter("/")
        .into_stream();

    while let Some(res) = stream.next().await {
        let _ = res.unwrap(); // xml panic happens here

        panic!("we never get here!");
    }
}

I'm using azure_storage and azure_storage_blobs 0.6 (commit f9ec73563f615553ca097b0457bed8347ad2227d). The same bug occurs in 0.5 as well.

The file structure of my blob store looks like this:

❯ tree get-most-recent-key-5
get-most-recent-key-5
├── 2021-08-04-testfile1
├── 2021-08-04T21:48:48.592953Z-15839722113750148182
│   └── 2021-08-04T21:48:48.592953Z-15839722113750148182
├── 2021-09-04-testfile2
└── 2022-08-04-testfile3

I just used the following command to upload these files:

az storage blob sync --account-key <redacted>  --account-name <redacted> --container <redacted> --source ./get-most-recent-key-5  --dest get-most-recent-key-5/

Below are a zipfile with the code used to repro and a zipfile with the contents of my Azure blob store container.

repro-storage-blobs-issue.zip
get-most-recent-key-5.zip
@cataggar
Copy link
Member

cataggar commented Sep 13, 2022 •
edited
Loading

Thanks for reporting @mdrach,

It looks like the error is good and includes everything needed to reproduce and unit test. I see a utf8 bom followed by xml.

Error { context: Full(Custom { kind: DataConversion, error: Custom { field: "duplicate field Blob" } }, "failed to deserialize the following xml into a azure_storage_blobs::container::operations::list_blobs::ListBlobsResponseInternal\n\u{feff}<EnumerationResults ServiceEndpoint="https://sisuautomatedtest.blob.core.windows.net/\" ContainerName="lowlatencyrequests">get-most-recent-key-5//get-most-recent-key-5/2021-08-04-testfile1Tue, 13 Sep 2022 08:20:48 GMTTue, 13 Sep 2022 08:20:48 GMT0x8DA9560DD170CFD19text/plain3X/+gWTy92gIJFXx57gLYA==BlockBlobHottrueunlockedavailabletrueget-most-recent-key-5/2021-08-04T21:48:48.592953Z-15839722113750148182/get-most-recent-key-5/2021-09-04-testfile2Tue, 13 Sep 2022 08:07:01 GMTTue, 13 Sep 2022 08:19:21 GMT0x8DA9560A916932D19text/plainb0CPJB6eDfKUzzW7dlboKQ==BlockBlobHottrueunlockedavailabletrueget-most-recent-key-5/2022-08-04-testfile3Tue, 13 Sep 2022 08:07:01 GMTTue, 13 Sep 2022 08:19:21 GMT0x8DA9560A91F929634text/plain1F1MssyZOvhY4OZevHWEsw==BlockBlobHottrueunlockedavailabletrue") }', src/main.rs:26:21

Just doing a quick unescape & format in VS Code, I get:

<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://sisuautomatedtest.blob.core.windows.net/" ContainerName="lowlatencyrequests">
  <Prefix>get-most-recent-key-5/</Prefix>
  <Delimiter>/</Delimiter>
  <Blobs>
    <Blob>
      <Name>get-most-recent-key-5/2021-08-04-testfile1</Name>
      <Properties>
        <Creation-Time>Tue, 13 Sep 2022 08:20:48 GMT</Creation-Time>
        <Last-Modified>Tue, 13 Sep 2022 08:20:48 GMT</Last-Modified>
        <Etag>0x8DA9560DD170CFD</Etag>
        <Content-Length>19</Content-Length>
        <Content-Type>text/plain</Content-Type>
        <Content-Encoding />
        <Content-Language />
        <Content-CRC64 />
        <Content-MD5>3X/+gWTy92gIJFXx57gLYA==</Content-MD5>
        <Cache-Control />
        <Content-Disposition />
        <BlobType>BlockBlob</BlobType>
        <AccessTier>Hot</AccessTier>
        <AccessTierInferred>true</AccessTierInferred>
        <LeaseStatus>unlocked</LeaseStatus>
        <LeaseState>available</LeaseState>
        <ServerEncrypted>true</ServerEncrypted>
      </Properties>
      <OrMetadata />
    </Blob>
    <BlobPrefix>
      <Name>get-most-recent-key-5/2021-08-04T21:48:48.592953Z-15839722113750148182/</Name>
    </BlobPrefix>
    <Blob>
      <Name>get-most-recent-key-5/2021-09-04-testfile2</Name>
      <Properties>
        <Creation-Time>Tue, 13 Sep 2022 08:07:01 GMT</Creation-Time>
        <Last-Modified>Tue, 13 Sep 2022 08:19:21 GMT</Last-Modified>
        <Etag>0x8DA9560A916932D</Etag>
        <Content-Length>19</Content-Length>
        <Content-Type>text/plain</Content-Type>
        <Content-Encoding />
        <Content-Language />
        <Content-CRC64 />
        <Content-MD5>b0CPJB6eDfKUzzW7dlboKQ==</Content-MD5>
        <Cache-Control />
        <Content-Disposition />
        <BlobType>BlockBlob</BlobType>
        <AccessTier>Hot</AccessTier>
        <AccessTierInferred>true</AccessTierInferred>
        <LeaseStatus>unlocked</LeaseStatus>
        <LeaseState>available</LeaseState>
        <ServerEncrypted>true</ServerEncrypted>
      </Properties>
      <OrMetadata />
    </Blob>
    <Blob>
      <Name>get-most-recent-key-5/2022-08-04-testfile3</Name>
      <Properties>
        <Creation-Time>Tue, 13 Sep 2022 08:07:01 GMT</Creation-Time>
        <Last-Modified>Tue, 13 Sep 2022 08:19:21 GMT</Last-Modified>
        <Etag>0x8DA9560A91F9296</Etag>
        <Content-Length>34</Content-Length>
        <Content-Type>text/plain</Content-Type>
        <Content-Encoding />
        <Content-Language />
        <Content-CRC64 />
        <Content-MD5>1F1MssyZOvhY4OZevHWEsw==</Content-MD5>
        <Cache-Control />
        <Content-Disposition />
        <BlobType>BlockBlob</BlobType>
        <AccessTier>Hot</AccessTier>
        <AccessTierInferred>true</AccessTierInferred>
        <LeaseStatus>unlocked</LeaseStatus>
        <LeaseState>available</LeaseState>
        <ServerEncrypted>true</ServerEncrypted>
      </Properties>
      <OrMetadata />
    </Blob>
  </Blobs>
  <NextMarker />
</EnumerationResults>

Then it is just a matter of creating a test case that tries to desialize to a ListBlobsResponseInternal to reproduce & troubleshoot.

@ctaggart ctaggart mentioned this issue Sep 16, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

add support for <BlobPrefix> when listing blobs ctaggart/azure-sdk-for-rust
2 participants
@mdrach @cataggar