Python API to get dataframes out of an RRD

[jleibs]

The user should be able to open an existing RRD.

Using the open RRD object, we should expose 4 new APIs:

• Get “Schema”
  • Probably takes an EntityExpression as an input
  • Returns Columns w/ information about entities, components, and types, static/dynamic
• List timelines
  • Timeline + TimeRange
• Latest-at (list of columns (multi entity), timestamp)
  • Returns ArrowRecordBatch (single row)
• RangeQuery (list of columns (multi entity), single point of view column, time range, timeline)
  • Returns iterator of ArrowRecordBatch (multiple rows)
  • This should be a "streaming" API so we don't need to allocate memory for the entire frame

The inputs to the APIs that take a list of columns likely either use ColumnDescriptors returned from the Schema API, or an EntityPath expression that can be indirectly evaluated to a list of columns.

When this is done we should have an easy piece of example code that opens and rrd and constructrs a dataframe that is fed into pandas or polars.

Observed API issues:

Querying for an image is weird:

pov = schema.column_for("world/cameras/image/rgb", rr.components.Blob)

[teh-cmc]

Sounds like part of this should be an alternative crate to re_query that implements uncached multi-component queries, performs the join/clamping and finally packs everything into a single final Chunk.

This Chunk can then be passed to the SDKs for further native integration with the host language.

[nikolausWest]

@teh-cmc: does this issue have enough specificity to implement these queries or do you need further details on the exact semantics of the queries?

[jleibs]

• Depends on: New ChunkStore APIs to facilitate data access #7284