Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added ERT forward model to convert from .UNSMRY to Arrow file format #685

Merged
merged 8 commits into from
Jun 25, 2021

Conversation

sigurdp
Copy link
Collaborator

@sigurdp sigurdp commented Jun 25, 2021

This PR adds an ERT forward model that can converts .UNSMRY files to Apache Arrow IPC file format.
The Arrow IPC file format is a columnar data format optimized for table-like datasets, see https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format
This file format is also known as Feather version 2, see https://arrow.apache.org/docs/python/feather.html

The ERT job imports data from an .UNSMRY file and writes output to an .arrow file in the above format.

The following data types are used in the exported data

  • DATE column is stored as an Arrow timetamp with millisecond resolution timestamp[ms]
  • All numeric columns will be stored as 32 bit floats
  • Summary meta data will be attached to each field/column in the exported table's schema under the
    'smry_meta' key

Currently CPI columns in the summary data wil be excluded from the exported .arrow file.


Contributor checklist

  • 🎉 This PR closes ERT forward model .UNSMRY → (.parquet or .arrow) #673
  • 🤖 I have added tests, or extended existing tests, to cover any new features or bugs fixed in this PR.
  • 📖 I have considered adding a new entry in CHANGELOG.md, and added it if should be communicated there.

@sigurdp sigurdp self-assigned this Jun 25, 2021
Copy link
Collaborator

@anders-kiaer anders-kiaer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🚀 Minor comments only. This will be a great improvement when in production 🏎️ ...and the plugins start using this data source through the data factory.

webviz_subsurface/ert_jobs/smry2arrow.py Outdated Show resolved Hide resolved
webviz_subsurface/ert_jobs/smry2arrow.py Outdated Show resolved Hide resolved
webviz_subsurface/ert_jobs/smry2arrow.py Outdated Show resolved Hide resolved
webviz_subsurface/ert_jobs/smry2arrow.py Outdated Show resolved Hide resolved
webviz_subsurface/ert_jobs/smry2arrow.py Show resolved Hide resolved
@anders-kiaer anders-kiaer added the next release 🚢 To be included in next release label Jun 25, 2021
@sigurdp sigurdp marked this pull request as ready for review June 25, 2021 13:13
@anders-kiaer anders-kiaer merged commit 4531fcd into equinor:master Jun 25, 2021
@anders-kiaer anders-kiaer deleted the sigurdp/ert-smry2arrow branch June 25, 2021 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
next release 🚢 To be included in next release
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

ERT forward model .UNSMRY → (.parquet or .arrow)
2 participants