Mechanism needed for benchmarking update-heavy workflows against create-only or index-only

[redneckbeard]

Currently, rally can be configured to generate metadata that contains conflicting ids for 25% of the documents in a corpus. However, the action specified in the metadata is always index. Since the performance characteristics of update changed significantly in 5.0+, this seems like a blind spot in the current suite.

While I understand that metadata can be interleaved with the documents in a corpus instead of being generated by rally, side-by-side comparison of performance of the same corpus with conflicting ids would be considerably simpler if the create/update workflow could use generated metadata.

[danielmitterdorfer]

Thanks for the suggestion @redneckbeard. I would:

• make the amount of conflicts configurable (0 - 100%)
• use update instead of index in the action and meta-data line for conflicting ids

As you write, for more complex cases there is always the possibility to provide the action and meta-data line in the original source file.

[redneckbeard]

@danielmitterdorfer that sounds great. Where things got messy for me was having to parse the JSON for the metadata here https://github.com/elastic/rally/blob/master/esrally/track/params.py#L817, and if you have an update, wrap the document json in {"doc": %(document)s}. I figured maintainers would be able to see a cleaner way.

[danielmitterdorfer]

My initial thought was to implement this in GenerateActionMetaData. However, I need to look whether this is really feasible (it probably is) but it will take a little while until I get to it as I'm busy at ElasticON this week.