Thoughts on a data schema for MUSE 2.0 input data

[ahawkes]

What do people think of this? Basically a relational database format.

https://lucid.app/lucidchart/6e05ea99-97d3-45f0-b1cf-2a300eff21e7/edit?viewport_loc=-12%2C-226%2C3518%2C2384%2C0_0&invitationId=inv_ad88b9f5-799d-4bf6-9e60-1559aa1688fc

Questions:

1. There are a few parameters that have a "one to many" relationship that don't have their own tables. Should they have their own tables?
2. At the moment each box, except the grey ones at the top, have their own input csv file. Are there better ways to do that?
3. What is the best way to structure the data after input in Rust? Carry around a pointer to a DB-like structure with all the data? Something like xarray? Something like C structs?

[ahawkes]

@tsmbland @dalonsoa @alexdewar

[dalonsoa]

I've made a few questions in the chart - pretty neat tool, by the way!

About your questions,
1- I think that if we are going for a DB structure, everything would need to have its table.
2- I think CSV files are a good choice, as long as we have a sensible file structure and robust validation in place.
3- Not sure, really. @alexdewar might have some suggestions, but might also require investigating the options a bit more.

I'm a bit unsure of how timeslices work in the new version, though. Clearly, they are defined differently than in old MUSE, where you always defined 3 levels (day/week/month, typically). It might be worth discussing what you have in mind in more detail.

[alexdewar]

I've had a look through the chart and it seems broadly sensible to me, though I don't understand the minutiae of the current model well enough to offer much in the way of comment. It'll be good as a starting point though and we can always revise as we go along.

Re the questions:

1. I think so!
2. Agree with @dalonsoa. Rust seems to be pretty good for validating input files too
3. I think we'll have to figure it out as we go along to an extent, but I suspect that it will make sense to have one big struct containing the input data that we can pass around a reference to. We might want some of the tabular data in polars dataframes, depending on how we want to use it, but vectors of structs are another option.

[ahawkes]

Just linking the final version @tsmbland arrived at here:

https://lucid.app/lucidchart/6d02ff7b-35ba-4e90-97ec-b695ff68b6e1/edit?page=0_0&invitationId=inv_ab477d33-772d-476f-b2e8-7d4cac7c90cf#