ZLibStream does not seem to decompress correctly.

[mburbea]

Description

I'm working on a save editor for the game Kingdoms of Amalur. Parts of the save file can be stored in compressed chunks using zlib compression.
Before .net6, I was using the nuget package DotNetZip and that seems to work correctly. However, I tried to switch from the library to the new System.IO.Compression.ZLibStream and it was no longer being read correctly. It seems to be cutting the data short.

I have created a repro here:
https://github.com/mburbea/ZlibStreamRepro/tree/master

Reproduction Steps

Clone the repro repository and run the program.

Expected behavior

Output should be:
Both read the same amount

Actual behavior

CoreLib read:51869
DotNetZip Read:133285
First discrepancy found at byte 51869 of output

Regression?

No.

Known Workarounds

Use a third party library.

Configuration

.net 6.10
Windows 11 x64

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I'm working on a save editor for the game Kingdoms of Amalur. Parts of the save file can be stored in compressed chunks using zlib compression.
Before .net6, I was using the nuget package DotNetZip and that seems to work correctly. However, I tried to switch from the library to the new System.IO.Compression.ZLibStream and it was no longer being read correctly. It seems to be cutting the data short.

I have created a repro here:
https://github.com/mburbea/ZlibStreamRepro/tree/master

Reproduction Steps

Clone the repro repository and run the program.

Expected behavior

Output should be:
Both read the same amount

Actual behavior

CoreLib read:51869
DotNetZip Read:133285
First discrepancy found at byte 51869 of output

Regression?

No.

Known Workarounds

Use a third party library.

Configuration

.net 6.10
Windows 11 x64

Other information

No response

Author:	mburbea
Assignees:	-
Labels:	area-System.IO.Compression, untriaged
Milestone:	-

[vcsjones]

var coreLibAmount = coreLibStream.Read(coreLibBuffer, 0, coreLibBuffer.Length);
var dotnetZipAmount = dotnetZipStream.Read(dotnetZipBuffer, 0, dotnetZipBuffer.Length);

This is a known change in .NET 6. See https://docs.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/6.0/partial-byte-reads-in-streams.

You need to call Read in a loop until it returns 0. Read is not guaranteed to return all of the bytes requested, only guarantee is when there are no more bytes to return.

[mburbea]

Ah interesting. I didn't realize that the new ZLibStream is also impacted by this change. It's not mentioned on that page.

[danmoseley]

I presume that ZLibStream was affected by the change; it has been following the standard stream contract from the start, so it isn't listed there. In your case I'm guessing you were assuming DotNetZip's behavior, and then you switched to ZLibStream, it would have appeared to be a break.

@mburbea if you verify that this works for you, we can close the issue. Thanks for reporting it though.

[vcsjones]

Ah interesting. I didn't realize that the new ZLibStream is also impacted by this change. It's not mentioned on that page.

ZLibStream is new in .NET 6 so that is probably why it was not documented as a behavior change. However, from the Stream.Read docs:

An implementation is free to return fewer bytes than requested even if the end of the stream has not been reached

It's valid for any stream implementation to return fewer bytes than requested.

[mburbea]

Good to know, I know I have some .net 5 code on at work that assumes that gzipped data can just be read into a MemoryStream as a one shot.
After testing and basically adding the following method I can confirm this corrects the behavior.

    public static int ReadAll(this ZLibStream stream, Span<byte> buffer)
    {
        var totalAmountRead = 0;
        while (stream.Read(buffer) is int read and > 0)
        {
            totalAmountRead += read;
            buffer = buffer[read..];
        }
        return totalAmountRead;
    }

@danmoseley, I'm closing the issue thanks for the quick triage!

[kasperk81]

this may get wrapped into a helper/extension method in future version #58216