You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an another things that could help with #731
Currently format of Documents sent for ingestion is
type Document struct {
Blob []byte
Type DocumentType
Format FormatType
SourceInformation SourceInformation
}
where Blob is the uncompressed data. For large files consumers will not be able to be ingested as NATS messages will be dropped (nats: slow consumer, messages dropped on connection).
As documents are highly compressible, it would be good to support ingesting them supporting popular compression algorithms. I think this feature can also have a wider usability, for example with proposed upload API (#1177) or downloading compressed files from S3 (also work in progress). In general we should allow people to send compressed files around if they want.
I'm currently working on draft PR for this.
The text was updated successfully, but these errors were encountered:
This is an another things that could help with #731
Currently format of Documents sent for ingestion is
where Blob is the uncompressed data. For large files consumers will not be able to be ingested as NATS messages will be dropped (
nats: slow consumer, messages dropped on connection
).As documents are highly compressible, it would be good to support ingesting them supporting popular compression algorithms. I think this feature can also have a wider usability, for example with proposed upload API (#1177) or downloading compressed files from S3 (also work in progress). In general we should allow people to send compressed files around if they want.
I'm currently working on draft PR for this.
The text was updated successfully, but these errors were encountered: