Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split requests for max-body-size #392

Closed
max-soe opened this issue Jan 11, 2022 · 2 comments · Fixed by #397
Closed

Split requests for max-body-size #392

max-soe opened this issue Jan 11, 2022 · 2 comments · Fixed by #397
Labels
enhancement New feature or request
Milestone
1.32.0

Comments

@max-soe
Copy link

max-soe commented Jan 11, 2022

Proposal:
You can specify the max body size of the influxdb and the client split the data points into correct batches.

Current behavior:
If you try to sent a bigger batch of datapoints with
client.write_points(points).

You get the error message:

<head><title>413 Request Entity Too Large</title></head>
<body>
<center><h1>413 Request Entity Too Large</h1></center>
<hr><center>nginx/1.17.4</center>
</body>
</html>

This already happen if the plain data points only have 1.500.000 bytes.

Desired behavior:
The client check the size of the body and split the data into different requests if necessary.

Use case:
We are not the owner of the influxdb instance and can not change the max-body-size.
But it's also hard to calculate the bytes of the body, because the body is created in this library.
Further the byte size of our data points are very different.
So it's very hard to find a good batch size for all our datapoints.
@bednar
Copy link
Contributor

bednar commented Jan 12, 2022

Hi @max-soe,

thanks for your proposal.

Is this something you might be willing to help with?

Regards

@bednar bednar added the enhancement New feature or request label Jan 12, 2022
@bednar bednar mentioned this issue Jan 24, 2022
Merged
6 tasks
@bednar
Copy link
Contributor

bednar commented Jan 24, 2022

Hi @max-soe,

Currently We don't want to include this in our library because it is not that common. I've prepared the following example to show you how to achieve the required functionality by RxPy:

influxdb-client-python/examples/write_batching_by_bytes_count.py

Line 83 in 8c8ab27

with InfluxDBClient(url='http://localhost:8086', token='my-token', org='my-org', retries=retries) as client:

Regards

@bednar bednar added this to the 1.32.0 milestone Aug 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
1.32.0
2 participants
@bednar @max-soe