Slow libecl reading with large ensembles

[anders-kiaer]

In use cases like e.g. FlowNet, it takes a veeeeeeeeeeeeeeeery long time to extract libecl data (through the fmu-ensemblescalls?). It also takes some time in large classical ensembles.

The parquet format supports metadata at multiple levels, and there are ways to also use that with pandas:
pandas-dev/pandas#20534 (comment)

Add (or extend) some forward model in semeio to read+store time series + necessary metadata to parquet, and read that by default in Webviz if the file exists?

Opinions @asnyv @berland?

[berland]

I assume you are talking about pre-processing on realization level to some data format that is easier to parse later on. equinor/fmu-tools#47 is an unfinished attempt at this, to make a standardized forward model that is being run on each realization, and where dumping to parquet or uploading anywhere would be possible.

The ERT api could be another possibility.

[anders-kiaer]

Related: equinor/webviz-config#382

[anders-kiaer]

After the work with @sigurdp on reducing memory footprint, the next major framework improvement would be to reduce Webviz build time (it is very slow for large ensembles/models). A brief offline chat with @jcrivenaes and @perolavsvendsen confirms that something like what is sketched below would be in line with SUMO and FMU metadata/Drogon.

Proposal: Basically a very simple job that reads .UNSMRY using libecl on realization level (while still on the cluster) and dumps it raw to a open standardized, efficient binary format, e.g. compressed .arrow files or HDF5. Metadata (units, is_rate/is_total...) could be added as metadata to the same file. Related issue wrt. metadata speed: #539

Another use case of such a job would be to optionally, for certain selected vectors which are sparse by nature (WSTAT and grid cell completion status, e.g. OPEN, SHUT, STOP) - to be stored "sparse" only (.UNSMRY would store WSTAT and grid cell completion status for every time step and COMPDAT grid cell, even though it easily could be ~99.9% redundant data). @lindjoha

[anders-kiaer]

After #673 there is now a forward model dumping .arrow files (https://arrow.apache.org/overview/), which have shown crazy good read performance on RHEL when testing on real models.