Feedback on the new parameter_value schema

[suvayu]

The new schema flattens all data to a tabular form: a list of index/value arrays. Briefly, it can be expressed as the following:

@dataclass(frozen=True)
class ArrayIndex:
	...


@dataclass(frozen=True)
class Array:
	...


Table: TypeAlias = list[ArrayIndex | Array | ...]

For the full description, see the Python models, and the corresponding JSON schema¹. It also includes couple of compressed encodings like run-end and dictionary encoding for arrays with repeated values. Missing values are natively supported. It supports all the original underlying types currently supported by Spine, with a few restrictions discussed below.

Questions

Duration

"Year" and "month" are ambiguous and not part of numpy/pandas/etc., "weeks" is the longest regular interval; e.g. month can mean anything between 28-31 days. We would like to avoid this ambiguity in the new format.

• When is the "duration" format normally used?
• Can it stand on its own, or is it only present along with a time-stamp, e.g. index of a time-series to denote frequency? Example from the docs:

  {
    "type": "map",
    "index_type": "date_time",
    "data": {
      "2010-01-01T00:00": {
        "type": "map",
        "index_type": "duration",
        "data": [["1D", -1.0], ["1D", -1.5]]
      },
      "2010-02-01-T00:00": {
        "type": "map",
        "index_type": "duration",
        "data": [["1 month", 2.3], ["2 months", 2.5]]
      }
    }
  }

  • Example "months": is it important to target a specific date of the month, or do is it sufficient to have 12 datapoints per year? In the latter case, using an integer to encode would be a better choice.
  • We need some real-world examples on how this is used
• Weeks are not supported in the old format. Should we add it in the new?

Time Patterns

• Is "intersection" actually used?
• Is it allowed to mix time-units when specifying unions, or is it uniform?
• Is the sole use-case to have indices in a time-series?
• We need some real-world examples on how this is used

Map

Consider this stochastic data from Antti (in the current map format):

stochastic_scnenario	analysis_time	time	value
scenario1	00:00	00:00	2
scenario1	00:00	18:00	3
scenario1	12:00	12:00	2.4
scenario1	12:00	06:00	4
scenario2	00:00		2.3
scenario2	12:00		3

• What is the difference between "analysis_time" and "time"?
• Is it always the case that the first "time" value coincides with the first "analysis_time" (marked bold), or is it coincidence in Antti's example?

Pydantic

So far we have avoided pydantic as a dependency. We plan to keep it in a separate repository for schema generation. This repo could also include tools for debugging user issues. If we need runtime validation, we need pydantic as a dependency within spinedb_api. What are your thoughts on this?

Implementation Choices

Old format uses the default name 'x' when the user doesn't provide anything. This can be ambiguous for the flattened datastructure. This is why we chose the following:

• If users supply the same column name twice, raise an error message:

  "Columns 5 and 3 conflict, having the same name '{name}'"

• If column names are missing, we assign one based on the following template:

  "col_{sequence}" (nesting depth in the old format)

---

@soininen @jkiviluo @manuelma I (and @OleMussmann) are posting this so that you can think ahead a bit before next week's Toolbox dev meeting, and anyone else can also share their feedback.

Footnotes

1. https://json-schema.org/learn/getting-started-step-by-step ↩

[jkiviluo]

Duration

I believe ambiguous durations are used only by SpineOpt. Duration is used to define e.g. for how long the model should run. It can also be used to define the length of time steps. For example, in an investment model, the duration of the investment time step can be one year (annual investments). Duration works as long as there is always a real timestamp to which the duration relates to. I think that's the only way SpineOpt uses Durations (and hence SpineOpt can deal with it).

Other models tend to use less rigid time system. This is also the case in ines-spec where timestamps (time series using datetime) is used for historical data. This makes sense because one can then correctly consider leap years and such things actually present in real data. But when modelling future years using those historical profiles, it would be problematic to use datetimes of those future years, since the leap year (or weekday) will not often match (with the profile data from history). So, when dealing with future data, ines-spec uses 'periods' where each period has a start_time (using datetime) and 'years_presented', which is basically the same as Duration in years. I'm currently using float for that, since, as you say, Duration is ambiguous. The model that gets the data from ines-spec has to deal with time stamps that are from the past and model years that are from the future (but that's what the data really is about). I think the best solution on the model side is to drop formal datetime, replace the year, and use these forged datetime strings as indexes. Your results can then show a leap year in 2029, but it doesn't really affect the quality of the results. But that shouldn't be the problem of Toolbox (it's model internals) - other than letting users use string indexed data efficiently (as I believe will happen with this new schema).

Also, sometimes we model multiple years at once, so the seams between years need to work without jumps (i.e. cutting leap years to 8760 is not a good solution).

All that said, Duration can be quite handy. E.g. when you want to test a model and run only two weeks, it's nice to use 2w as model Duration. Using time steps would be more involved (especially when going over month lines - that is actually quite error prone). So, having Duration that would not accept months, but would know how to deal with a year in the complex way I outlined above would be ideal in my view.

I don't know if there is some data that is given at monthly basis in SpineOpt - if there is, then it's a bit of problem, but I think your solution to use integer months would be suitable in those cases (as long as it's about specific historical months). Just prevent using month in Duration.

Ps. I used 'historical' and 'past' to distinguish from future. However, sometimes time series data can be generated and as such have future datetimes. Doesn't change anything though.

[jkiviluo]

Pydantic

Claude tells me that Pydantic is light-weight and quite stable (in version 2), so I don't see it as a problem to have it as a dependency. I suppose lots of applications use it.

[jkiviluo]

Map

The analysis time is the timestep from which the stochastic data originates (when the data about the future was created). Time is the actual timestep for which the data is given (in a particular branch/scenario.

In modelling practice the analysis time and the first time step are the same, but in principle one could have analysis_time earlier than the first time. Data is usually created earlier, but model structures tend to ignore any lags there may be (they don't affect the workings of the model).

From the example, I don't know why Scenario 2 does not have 'time' stamp, just analysis_time. It doesn't really make sense. Deterministic data would not have analysis_time (nor scenario), but it would have time.

[jkiviluo]

As Manuel is on leave, maybe @DillonJ can provide examples of time pattern.

[DillonJ]

It's all documented toolbox side I believe

[DillonJ]

https://spine-toolbox.readthedocs.io/en/latest/parameter_value_editor.html#time-patterns

[DillonJ]

I feel we should assume all the functionality is used. We use it and we should assume others use it too. It's incredibly useful for defining data that varies by time of day... e,g, reserve requirements.

[suvayu]

Jody, could you give some actual examples of the data? It's very difficult to test a new implementation otherwise. That's why we included the specific questions about intersections, and unions. I'm particularly curious about the intersection feature, I can't imagine what that looks like.

[DillonJ]

Regarding Maps - they are used for other things in SpineOpt other than for stochastic data:

• Monte-carlo scenario data
• Solver options
• Representative period mappings

[suvayu]

Same here, some actual examples would be really helpful. We are trying to add more structure to an something that is otherwise extremely flexible. Having more examples helps us reach a common denominator faster.

While I can imagine MC scenario data & representative period mappings can be represented with the current proposal, solver options probably cannot. Having clear examples would help us modify it to include that.

[soininen]

Durations

Related issue spine-tools/Spine-Database-API#321 and to some extend, spine-tools/Spine-Database-API#319

I have no strong opinion on whatever we choose to do with years and months as I think it is better that the model developers who actually need to deal with them have a say here. The only place where we do any calculations with durations in spinedb_api is when we build time stamps for fixed resolution time series and there we have the start time and use the datetutil library which means we can deal with months and leap years.

Can it stand on its own,[...]?

Yes.

Weeks are not supported in the old format. Should we add it in the new?

I would not mind, if the model developers feel it could be useful.

Time Patterns

Is the sole use-case to have indices in a time-series?

Not sure if I understand the question. Time-pattern is a "time series" of sorts on its own, is that what you mean?

Map

I do not think we want to specialize our data structures too heavily for model-specific cases.

Pydantic

We could maybe utilize Pydantic in Spine DB Server some day. Also, it is possible to paste JSON directly to DB editor, which could be validated with Pydantic. But why add it as a dependency before we actually use it?

Implementation choices

I approve the index name ambiguity guards [official approval stamp here].

[suvayu]

Time Pattern

Not sure if I understand the question. Time-pattern is a "time series" of sorts on its own, is that what you mean?

Yes, that was it. And the other questions are about the intersection and union features in the time pattern syntax.

Map

Fair. We were trying to understand the usage pattern, and maybe got too drawn into the details.

Pydantic

Also, it is possible to paste JSON directly to DB editor, which could be validated with Pydantic

I had exactly that use case in mind! We can take it one step at a time.

Implementation choices

😃

[suvayu]

@jkiviluo @soininen I'll consolidate my Duration related response here for brevity.

Duration

The reasons for our questions about duration is because there are a couple of ways to represent this in pyarrow. I'll try to summarise the choices here.

1. Duration without ambiguity (W, D, M, s, ms, us, ns): these can be represented in Python by datetime.timedelta, or pandas.Timedelta. PyArrow uses a DurationType, which reduces everything to s. us, or ns. This trivially convert to pandas.Timedelta.
2. Duration with ambiguity (Y, m, and all of the above): these can be represented in Python by dateutil.relativedelta.relativedelta, or pandas.DateOffset. PyArrow represents this as MonthDayNano (with some reasonable caveats), which is an interval type. An interval type splits the delta into a tuple of months, days, nanoseconds, where months captures the ambiguity, and days & nanoseconds are unambiguous. These don't trivially convert to timedelta.

So far we have been using timedelta to do the conversion, but it can be easily switched to relativedelta.

---

Based on your comments above, I understand that Spine users indeed use the duration type always in reference to another time:

1. either a datetime in the input data, or
2. for uses like model run: "now" / "start_time" / equivalent

IMO, for both uses, the model knows best how to do the calculation. If we choose MonthDayNano (relativedelta equivalent) as the underlying PyArrow type, then we should provide APIs to covert to Duration (timedelta equivalent). It will need an input time to anchor against. So something like this:

rel_delta = ... # my relativedelta array in pyarrow
# we have to implement the following, and document how to use with examples
durations = rel_delta.to_durations(anchor_time: datetime = some_datetime_index[-1])
new_datetimes = some_datetime_index + durations

The model will have to do something equivalent to the above because only the model knows what is the appropriate value of anchor_time¹.

It seems to me this would be the preferred solution.

Footnotes

1. Unfortunately the terminology is a bit confusing, so maybe instead of to_durations, something else like make_concrete, or anchor_to_datetime. We can iterate on that. ↩