Additional metrics for bulk requests

[cdahlqvist]

When comparing bulk indexing throughput for different types of data, pure EPS can be very misleading if events have different sizes. I have found the total volume of JSON records indexed per time period to sometimes be a more useful measure, as this takes event size into consideration.

This pull request adds the total size of the bulk request as well as the total size of the documents indexed to the metrics reported, allowing the amount of JSON indexed per timeframe and total volume sent over the wire to be calculated.

[danielmitterdorfer]

Thanks for your PR @cdahlqvist! I left a couple of comments.

[danielmitterdorfer]

nit: typo "requset" -> "request"

[danielmitterdorfer]

This adds some overhead which is - depending on the track - at worst in the single-digit percentage range (I've measured up to 95ms). I think it would make sense to move this loop to the method detailed_stats. Users can then enable it for their tracks by setting detailed-results to true. Complete example:

{
  "name": "bulk-index",
  "operation-type": "index",
  "bulk-size": 5000,
  "detailed-results": true
}

Btw, you could also use enumerate for the loop instead of range to simplify it a bit:

for line_number, data in enumerate(params["body"]):
    line_size = len(data)
    if params["action_metadata_present"]:
        total_document_size_bytes += line_size
    #....

Another fine detail: the len function in Python returns the number of characters (although it's not really clearly documented) but combining the following two facts show that this is the case:

1. "[len] [r]eturn[s] the length (the number of items) of an object. The argument may be a sequence (such as a string [...])" (source)
2. "Strings are immutable sequences of Unicode code points." (source)

Therefore, we can reason that len returns the number of Unicode code points (i.e. characters) of a string. For characters that can be represented with 1 byte, the number of characters is identical to the number of bytes but strictly speaking it's the number of characters. To determine the number of bytes you need to encode the string as UTF-8, e.g. len(s.encode('utf-8')).

[cdahlqvist]

@danielmitterdorfer Have updated the PR according to given suggestions.

[danielmitterdorfer]

LGTM. I think it makes sense to add the following two asserations to BulkIndexRunnerTests#test_mixed_bulk_with_detailed_stats(self, es):

        self.assertEqual(158, result["bulk-request-size-bytes"])
        self.assertEqual(62, result["total-document-size-bytes"])

[cdahlqvist]

Added the assertions to the test.

[danielmitterdorfer]

LGTM. You can merge at any time.