Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(embeddings): use stdlib array type for improved performance #2060

Merged
merged 4 commits into from
Feb 6, 2025

Conversation

tonybaloney
Copy link
Contributor

@tonybaloney tonybaloney commented Jan 28, 2025

Implements #2059

Changes the default embedding request to use base64 responses (unless overriden).
Changes the default response parser to use the builtin array type to convert from the base64 to a float32 compact array then directly into a list.

4x faster than the current default (see issue with benchmark)
20% faster than with numpy, although I've left the numpy code in. The numpy code could be removed completely.

Benchmark Min Max Mean Min (+) Max (+) Mean (+)
Standard vs array 0.988 1.523 1.233 0.292 (3.4x) 0.313 (4.9x) 0.297 (4.1x)
Standard vs numpy 0.928 1.046 0.991 0.341 (2.7x) 0.370 (2.8x) 0.361 (2.7x)
numpy vs array 0.333 0.383 0.347 0.284 (1.2x) 0.316 (1.2x) 0.298 (1.2x)

@tonybaloney tonybaloney marked this pull request as ready for review January 29, 2025 01:53
@tonybaloney tonybaloney requested a review from a team as a code owner January 29, 2025 01:53
@tonybaloney tonybaloney changed the title Use builtin array type to convert f32 bytes to float list Use base64 embedding responses by default and use stdlib array type to decode results Jan 29, 2025
@RobertCraigie RobertCraigie changed the base branch from main to next January 31, 2025 11:30
Copy link
Collaborator

@RobertCraigie RobertCraigie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this!

I think you'll need to rebase against next to fix the test errors in this PR fyi

src/openai/resources/embeddings.py Show resolved Hide resolved
@tonybaloney
Copy link
Contributor Author

I've rebased to next.

Regarding the defaults, it still has the same fallback to inspect the result. Adding the parameter is currently a side-effect of having numpy installed in the same venv, no?

@RobertCraigie RobertCraigie changed the title Use base64 embedding responses by default and use stdlib array type to decode results feat(embeddings): use stdlib array type for improved performance Feb 6, 2025
Copy link
Collaborator

@RobertCraigie RobertCraigie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants