WIP: Plugable Query Result Storage-Cache

[ismailsimsek]

This pull request is adding plugable QueryResultData which is responsible managing query_result.data (big python object).
This model is flexible and can easily be extended, like using s3. current implementations are 'db' default and 'file' as using local disk to store query_result.data

Why?
masking sure application can handle big Queryresults and hoping to improve application performance with big resultsets

Implementation notes

• by default it is using ‘db’ and code is backward compatible
• QueryResultData is used only when its necessary, rest of QueryResult operations are done without ‘data’ (large python object)
  • since query_result_data rows are much smaller now any query hitting this table going to be fast. since it dont need to read “data” field from disk.
• QueryResultData using QueryResult.id as a key value of result object. in case of ‘db’ its PK
• QueryResultData only used from QueryResult so this way we get minimum refactoring impact.
• QueryResultData is only responsible storing, retrieving or deleting data object. it is independent of the type of ‘data’ object.

What's next
changing query result object to more performant object. avro, MessagePack ...

[arikfr]

Thank you, @ismailsimsek ! There are some implementation specific comments about this implementation.

But I think it's worth to discuss the implementation direction in more general lines first --

1. The API should allow for "streaming" load and save of results, this to really allow handling large results while minimizing the memory usage during both operations.

2. I'm not sure there is a point in splitting the results into two tables when using DB persistence. Postgres has its own mechanism to make sure rows stay within reasonable size (TOAST).

Also running a migration on query_results might not be feasible on some deployments (because of size), so might worth to consider an implementation that doesn't require it. I was thinking about repurposing the data column to store a "pointer" to the actual results location. Then it can include some more metadata (like rows count, columns metadata, and maybe even preview of the data).

[ismailsimsek]

1. The API should allow for "streaming" load and save of results, this to really allow handling large results while minimizing the memory usage during both operations.

agree, one way i can think of is passing QueryResultData object in the application and streaming data only when it needed. using save, load methods.
do you have any serialization format in mind? with json only way doing it is appending multiple json objects(each one is a row) to a file which is problematic when reading it
avro has DataFileWriter.appendTo option and you can iterate over records when reading. but it might be problematic defining avro schema, data types from columns might not be compatible

[arikfr]

agree, one way i can think of is passing QueryResultData object in the application and streaming data only when it needed. using save, load methods.

Yep, something along these lines.

do you have any serialization format in mind? with json only way doing it is appending multiple json objects(each one is a row) to a file which is problematic when reading it

Yes, I was thinking about JSON Lines. It allows for streaming and more efficient storage wise, as there are no column names in each line. It might be a bit harder to read on the client side, but shouldn't be that much harder.

[codeclimate]

Function __init__ has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

[codeclimate]

Function get_handler has a Cognitive Complexity of 7 (exceeds 5 allowed). Consider refactoring.

[codeclimate]

Too many blank lines (2)

[codeclimate]

Missing whitespace around operator

[codeclimate]

Missing whitespace after ','

[ismailsimsek]

@arikfr high level design is ready do you have time to have look at it ?

[arikfr]

Thank you for your effort, @ismailsimsek. We decided to take a bit of a different approach in: #4147.