Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Azure OpenAI] Image generation support from 2023-06-01-preview service API #37317

Conversation

trrwilson
Copy link
Member

@trrwilson trrwilson commented Jun 29, 2023

This change introduces DALL-E image generation support to the .NET SDK based on the latest TypeSpec code generation updates.

The code is a bit "thick" with custom code relative to what we'd like, but that's necessary at the moment given that Azure OpenAI performs image generation as a long-running operation (LRO) while standard OpenAI performs it as a normal, synchronous POST operation.

To represent this,

  • Response<ImageGenerations> GetImageGenerationsAsync(ImageGenerationOptions, ...) is introduced, parallel in structure to e.g. Response<ChatCompletions> GetChatCompletionsAsync(..., ChatCompletionsOptions, ...)
  • ImageGenerations contains a collection of ImageResponseItem elements, each of them a concrete instance of one of the two derived classes: ImageLocation and ImagePayload
  • Which of the two concrete types populates the abstract collection of IReadOnlyList<ImageResponseItem> Data is determined by the request value of ResponseFormat in the ImageGenerationOptions
  • For consuming format-specific data (URL or base64 string), consumers need to cast the item to the appropriate derived type, e.g. (imageGenerationsResponse.Value.Data[0] as ImageLocation).Url
  • The abstract class also provides a generalized Stream GetStream() method that will generate a stream for the image data independently of the response format (using the client's pipeline to facilitate downloads in the case of URL format and simply wrapping the base64 data in the case of base64 format)

Internally, the implementation of GetImageGenerations wraps a deferred LRO call in the case of Azure OpenAI requests and constructs a simpler, straightforward request in the case of standard OpenAI service use.

These capabilities are based on the 2023-06-01-preview service API and we update that dependency (including test collateral) as part of this.

@azure-sdk
Copy link
Collaborator

API change check

APIView has identified API level changes in this PR and created following API reviews.

Azure.AI.OpenAI

@jsquire jsquire removed the Open AI label Jun 30, 2023
@trrwilson
Copy link
Member Author

Subsumed by the converged PR: #37539

@trrwilson trrwilson closed this Jul 19, 2023
@trrwilson trrwilson deleted the user/travisw/aoai-2023-06-01-initial-update branch July 19, 2023 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants