Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Support ingesting compressed documents #1178

Closed
dejanb opened this issue Aug 23, 2023 · 1 comment
Closed

[feature] Support ingesting compressed documents #1178

dejanb opened this issue Aug 23, 2023 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@dejanb
Copy link
Contributor

dejanb commented Aug 23, 2023

This is an another things that could help with #731

Currently format of Documents sent for ingestion is

type Document struct {
	Blob              []byte
	Type              DocumentType
	Format            FormatType
	SourceInformation SourceInformation
}

where Blob is the uncompressed data. For large files consumers will not be able to be ingested as NATS messages will be dropped (nats: slow consumer, messages dropped on connection).

As documents are highly compressible, it would be good to support ingesting them supporting popular compression algorithms. I think this feature can also have a wider usability, for example with proposed upload API (#1177) or downloading compressed files from S3 (also work in progress). In general we should allow people to send compressed files around if they want.

I'm currently working on draft PR for this.

@dejanb dejanb added the enhancement New feature or request label Aug 23, 2023
@dejanb dejanb changed the title [feature] Supported ingesting compressed documents [feature] Support ingesting compressed documents Aug 23, 2023
@dejanb
Copy link
Contributor Author

dejanb commented Sep 12, 2023

Implemented with #1222

@dejanb dejanb closed this as completed Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant