Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Creating a new Image<TPixel> by Wrapping Memory<byte> #1097

Closed
vpenades opened this issue Jan 29, 2020 · 2 comments · Fixed by #1314
Closed

Creating a new Image<TPixel> by Wrapping Memory<byte> #1097

vpenades opened this issue Jan 29, 2020 · 2 comments · Fixed by #1314

Comments

@vpenades
Copy link
Contributor

As discussed in gitter:

With the current API, it is possible to create a new image in two ways:

  • LoadPixelData: which accepts a Span , but it makes an internal copy of the source data.
  • WrapMemory: which requires a Memory to wrap user memory.

The problem is that in many cases, the source data is provided as a plain byte array. this is specially true when dealing with devices at low level, like cameras and sensors.

In these cases, it would be desirable to do have Image,WrapPixelData(Byte[]) or something like that:

var bits = new Byte[256*256*4];
var image = Image<Argb32>.WrapPixelData(bits, 256, 256);

@antonfirsov unfortunately, MemoryMarshal can't cast Memory-s only Spans. There is no API for what you want, but you can implement a MemoryManager which does the span-casting on it's API surface. Feel free to raise a feature request so we remember to add this later.

I am aware that with the current ImageSharp architecture this might be difficult, but I believe it's a case scenario that might show quite often, so if you guys can find a solution for this...

@antonfirsov antonfirsov changed the title Creating a new Image<TPixel> by Wrapping a Byte[] array Creating a new Image<TPixel> by Wrapping Memory<byte> Jan 31, 2020
@antonfirsov
Copy link
Member

antonfirsov commented Jan 31, 2020

I edited the title, since byte[] could be implicitly converted to Memory<byte> (same element type). We need to add the following method:

public class Image
{
+    public static Image<TPixel> WrapMemory(Memory<byte> memory, int width, int height);
}

@vpenades
Copy link
Contributor Author

vpenades commented Jan 31, 2020

public class Image
{
+    public static Image<TPixel> WrapMemory(Memory<byte> memory, int width, int height);
}

Yes, that's exactly what I would need. But I'm wondering how would you implement it.

It's certainly non trivial to cast a Memory<byte> to a Memory<TPixel> and if there's an API for that I would gladly want to know...

So I pressume ImageSharp would need to do some work under the hood.

@JimBobSquarePants JimBobSquarePants modified the milestones: 1.0.0, Future Jul 16, 2020
@Sergio0694 Sergio0694 self-assigned this Aug 13, 2020
ghost pushed a commit to CommunityToolkit/WindowsCommunityToolkit that referenced this issue Nov 12, 2020
## PR Type
What kind of change does this PR introduce?
<!-- Please uncomment one or more that apply to this PR. -->

 - Feature

## What is the current behavior?
<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
Right now there is no (easy) way to cast a `Memory<TFrom>` instance to a `Memory<TTo>` instance. There are APIs to to do that for `Span<T>` instances, but not for `Memory<T>`. The reason for that is that with a `Span<T>` it's just a matter of retrieving the wrapped reference, reinterpreting it and then adjusting the size, then creating a new `Span<T>` instance. But a `Memory<T>` instance is completely different: it wraps an object which could be either a `T[]` array, a `MemoryManager<T>` instance, etc. The result is that currently there are no APIs in the BCL nor in the toolkit to just "cast" a `Memory<T>`.

This feature has been requested by a number of developers, including in a well known library such as `ImageSharp`:

> Yes, that's exactly what I would need. But I'm wondering how would you implement it.
> It's certainly non trivial to cast a `Memory<byte>` to a `Memory<TPixel>` and if there's an API for that I would gladly want to know...
> So I pressume `ImageSharp` would need to do some work under the hood.

(_`ImageSharp` issue, [here](SixLabors/ImageSharp#1097 (comment))
To solve that, I created a very simplified version of the code included in this PR, into a PR [here](SixLabors/ImageSharp#1314).

Having this available right out of the box in the `HighPerformance` package would be helpful in a number of similar situations, especially with `Memory<T>` APIs becoming more and more common across libraries now (as they've been out for a while).

## What is the new behavior?
<!-- Describe how was this issue resolved or changed? -->
This PR includes 4 new extensions for the `Memory<T>` and `ReadOnlyMemory<T>` types that enable the following:

```csharp
// Cast between two Memory<T> instances...
Memory<byte> memoryOfBytes = new byte[128].AsMemory();
Memory<float> memoryOfFloats = memoryOfBytes.Cast<byte, float>();

// ...any number of times is needed
Memory<int> memoryOfInts = memoryOfFloats.Cast<float, int>();
Memory<byte> backToBytesMemory = memoryOfInts.Cast<int, byte>();

// Or just convert into bytes directly
Memory<int> sourceAsInts = new int[128].AsMemory();
Memory<byte> sourceAsBytes = sourceAsInts.AsBytes();

// Want to get a stream from a string? Why not! 😄
using (Stream stream = "Hello world".AsMemory().AsBytes().AsStream())
{
    // Use the stream here, which reads *directly* from the string data!
}
```

Here is the full list of the new APIs introduced in this PR:

```csharp
namespace Microsoft.Toolkit.HighPerformance.Extensions
{
    public static class MemoryExtensions
    {
        public static Memory<byte> AsBytes<T>(this Memory<T> memory)
            where T : unmanaged;

        public static Memory<TTo> Cast<TFrom, TTo>(this Memory<TFrom> memory)
            where TFrom : unmanaged
            where TTo : unmanaged;
    }

    public static class ReadOnlyMemoryExtensions
    {
        public static ReadOnlyMemory<byte> AsBytes<T>(this ReadOnlyMemory<T> memory)
            where T : unmanaged;

        public static ReadOnlyMemory<TTo> Cast<TFrom, TTo>(this ReadOnlyMemory<TFrom> memory)
            where TFrom : unmanaged
            where TTo : unmanaged;
    }
}
```

## Notes

Marking as draft as this is still being worked on, but feedbacks and reviews are welcome! 😄

## PR Checklist

Please check if your PR fulfills the following requirements:

- [X] Tested code with current [supported SDKs](../readme.md#supported)
- [ ] ~~Pull Request has been submitted to the documentation repository [instructions](..\contributing.md#docs). Link: <!-- docs PR link -->~~
- [ ] ~~Sample in sample app has been added / updated (for bug fixes / features)~~
    - [ ] ~~Icon has been created (if new sample) following the [Thumbnail Style Guide and templates](https://github.com/windows-toolkit/WindowsCommunityToolkit-design-assets)~~
- [X] Tests for the changes have been added (for bug fixes / features) (if applicable)
- [X] Header has been added to all new source files (run *build/UpdateHeaders.bat*)
- [X] Contains **NO** breaking changes
Sergio0694 pushed a commit to CommunityToolkit/dotnet that referenced this issue Oct 29, 2021
## PR Type
What kind of change does this PR introduce?
<!-- Please uncomment one or more that apply to this PR. -->

 - Feature

## What is the current behavior?
<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
Right now there is no (easy) way to cast a `Memory<TFrom>` instance to a `Memory<TTo>` instance. There are APIs to to do that for `Span<T>` instances, but not for `Memory<T>`. The reason for that is that with a `Span<T>` it's just a matter of retrieving the wrapped reference, reinterpreting it and then adjusting the size, then creating a new `Span<T>` instance. But a `Memory<T>` instance is completely different: it wraps an object which could be either a `T[]` array, a `MemoryManager<T>` instance, etc. The result is that currently there are no APIs in the BCL nor in the toolkit to just "cast" a `Memory<T>`.

This feature has been requested by a number of developers, including in a well known library such as `ImageSharp`:

> Yes, that's exactly what I would need. But I'm wondering how would you implement it.
> It's certainly non trivial to cast a `Memory<byte>` to a `Memory<TPixel>` and if there's an API for that I would gladly want to know...
> So I pressume `ImageSharp` would need to do some work under the hood.

(_`ImageSharp` issue, [here](SixLabors/ImageSharp#1097 (comment))
To solve that, I created a very simplified version of the code included in this PR, into a PR [here](SixLabors/ImageSharp#1314).

Having this available right out of the box in the `HighPerformance` package would be helpful in a number of similar situations, especially with `Memory<T>` APIs becoming more and more common across libraries now (as they've been out for a while).

## What is the new behavior?
<!-- Describe how was this issue resolved or changed? -->
This PR includes 4 new extensions for the `Memory<T>` and `ReadOnlyMemory<T>` types that enable the following:

```csharp
// Cast between two Memory<T> instances...
Memory<byte> memoryOfBytes = new byte[128].AsMemory();
Memory<float> memoryOfFloats = memoryOfBytes.Cast<byte, float>();

// ...any number of times is needed
Memory<int> memoryOfInts = memoryOfFloats.Cast<float, int>();
Memory<byte> backToBytesMemory = memoryOfInts.Cast<int, byte>();

// Or just convert into bytes directly
Memory<int> sourceAsInts = new int[128].AsMemory();
Memory<byte> sourceAsBytes = sourceAsInts.AsBytes();

// Want to get a stream from a string? Why not! 😄
using (Stream stream = "Hello world".AsMemory().AsBytes().AsStream())
{
    // Use the stream here, which reads *directly* from the string data!
}
```

Here is the full list of the new APIs introduced in this PR:

```csharp
namespace Microsoft.Toolkit.HighPerformance.Extensions
{
    public static class MemoryExtensions
    {
        public static Memory<byte> AsBytes<T>(this Memory<T> memory)
            where T : unmanaged;

        public static Memory<TTo> Cast<TFrom, TTo>(this Memory<TFrom> memory)
            where TFrom : unmanaged
            where TTo : unmanaged;
    }

    public static class ReadOnlyMemoryExtensions
    {
        public static ReadOnlyMemory<byte> AsBytes<T>(this ReadOnlyMemory<T> memory)
            where T : unmanaged;

        public static ReadOnlyMemory<TTo> Cast<TFrom, TTo>(this ReadOnlyMemory<TFrom> memory)
            where TFrom : unmanaged
            where TTo : unmanaged;
    }
}
```

## Notes

Marking as draft as this is still being worked on, but feedbacks and reviews are welcome! 😄

## PR Checklist

Please check if your PR fulfills the following requirements:

- [X] Tested code with current [supported SDKs](../readme.md#supported)
- [ ] ~~Pull Request has been submitted to the documentation repository [instructions](..\contributing.md#docs). Link: <!-- docs PR link -->~~
- [ ] ~~Sample in sample app has been added / updated (for bug fixes / features)~~
    - [ ] ~~Icon has been created (if new sample) following the [Thumbnail Style Guide and templates](https://github.com/windows-toolkit/WindowsCommunityToolkit-design-assets)~~
- [X] Tests for the changes have been added (for bug fixes / features) (if applicable)
- [X] Header has been added to all new source files (run *build/UpdateHeaders.bat*)
- [X] Contains **NO** breaking changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants