New streaming api #91
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meatballhat
reviewed
Oct 3, 2024
stream.go
Outdated
Comment on lines
215
to
214
| case SSETypeOutput: | ||
| io.WriteString(writer, event.Data()) | ||
| case SSETypeDone: | ||
| writer.Close() | ||
| return |
There was a problem hiding this comment.
Would error handling be useful here?
There was a problem hiding this comment.
SSETypeDone isn't an error case, so I'm not sure what you mean?
philandstuff
force-pushed
the
new-streaming
branch
from
5656e2c to
965e0ef
Compare
This is a substantial refactoring: - I've changed the retry logic from making a recursive call to using a loop - I've started using the existing r.options.retryPolicy to define retry behavior - I've extracted a common streamEventsTo function, used by both streamTextTo and streamFilesTo, so that retry behavior is defined in one place
|
Oof I think this is leaking goroutines https://github.com/replicate/synthetics/pull/108#issuecomment-2405810904 I'll dig in to this tomorrow. |
philandstuff
force-pushed
the
new-streaming
branch
from
b615aa9 to
84460b1
Compare
The eventsource one leaks goroutines if you don't hold it just right.
philandstuff
force-pushed
the
new-streaming
branch
from
84460b1 to
32241c2
Compare
|
This no longer leaks 🎉 |
No goroutines! Only Readers!
Again, no goroutines! Only Readers!
philandstuff
force-pushed
the
new-streaming
branch
from
27e45f7 to
abaf8f3
Compare
nickstenning
approved these changes
Oct 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds functions StreamPredictionText() and StreamPredictionFiles() that stream output from models that return iterators of strings or files, respectively.
This is scrappy but functional. I'd like to discuss the approach and see if people think this is sensible.