-
Notifications
You must be signed in to change notification settings - Fork 918
/
Copy pathtimeseries.py
5244 lines (4552 loc) · 210 KB
/
timeseries.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
"""
Timeseries
----------
``TimeSeries`` is the main class in `darts`.
It represents a univariate or multivariate time series, deterministic or stochastic.
The values are stored in an array of shape `(time, dimensions, samples)`, where
`dimensions` are the dimensions (or "components", or "columns") of multivariate series,
and `samples` are samples of stochastic series.
Definitions:
- A series with `dimensions = 1` is **univariate** and a series with `dimensions > 1` is **multivariate**.
- | A series with `samples = 1` is **deterministic** and a series with `samples > 1` is
| **stochastic** (or **probabilistic**).
Each series also stores a `time_index`, which contains either datetimes (:class:`pandas.DateTimeIndex`)
or integer indices (:class:`pandas.RangeIndex`).
``TimeSeries`` are guaranteed to:
- Have a monotonically increasing time index, without holes (without missing dates)
- Contain numeric types only
- Have distinct components/columns names
- Have a well defined frequency (
`date offset aliases <https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_
for ``DateTimeIndex``, and step size for ``RangeIndex``)
- Have static covariates consistent with their components, or no static covariates
- Have a hierarchy consistent with their components, or no hierarchy
``TimeSeries`` can contain global or component-specific static covariate data. Static covariates in `darts` refers
to external time-invariant data that can be used by some models to help improve predictions.
Read our `user guide on covariates <https://unit8co.github.io/darts/userguide/covariates.html>`__ and the
``TimeSeries`` documentation for more information on covariates.
"""
import itertools
import pickle
import re
import sys
from collections import defaultdict
from copy import deepcopy
from inspect import signature
from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union
import matplotlib.axes
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import xarray as xr
from pandas.tseries.frequencies import to_offset
from scipy.stats import kurtosis, skew
from .logging import get_logger, raise_if, raise_if_not, raise_log
try:
from typing import Literal
except ImportError:
from typing_extensions import Literal
if sys.version_info >= (3, 11):
from typing import Self
else:
from typing_extensions import Self
logger = get_logger(__name__)
# dimension names in the DataArray
# the "time" one can be different, if it has a name in the underlying Series/DataFrame.
DIMS = ("time", "component", "sample")
VALID_INDEX_TYPES = (pd.DatetimeIndex, pd.RangeIndex)
STATIC_COV_TAG = "static_covariates"
DEFAULT_GLOBAL_STATIC_COV_NAME = "global_components"
HIERARCHY_TAG = "hierarchy"
class TimeSeries:
def __init__(self, xa: xr.DataArray):
"""
Create a TimeSeries from a (well formed) DataArray.
It is recommended to use the factory methods to create TimeSeries instead.
See Also
--------
TimeSeries.from_dataframe : Create from a :class:`pandas.DataFrame`.
TimeSeries.from_group_dataframe : Create multiple TimeSeries by groups from a :class:`pandas.DataFrame`.
TimeSeries.from_series : Create from a :class:`pandas.Series`.
TimeSeries.from_values : Create from a NumPy :class:`ndarray`.
TimeSeries.from_times_and_values : Create from a time index and a Numpy :class:`ndarray`.
TimeSeries.from_csv : Create from a CSV file.
TimeSeries.from_json : Create from a JSON file.
TimeSeries.from_xarray : Create from an :class:`xarray.DataArray`.
"""
raise_if_not(
isinstance(xa, xr.DataArray),
"Data must be provided as an xarray DataArray instance. "
"If you need to create a TimeSeries from another type "
"(e.g. a DataFrame), look at TimeSeries factory methods "
"(e.g. TimeSeries.from_dataframe(), "
"TimeSeries.from_xarray(), TimeSeries.from_values()"
"TimeSeries.from_times_and_values(), etc...).",
logger,
)
raise_if_not(
len(xa.shape) == 3,
f"TimeSeries require DataArray of dimensionality 3 ({DIMS}).",
logger,
)
# Ideally values should be np.float, otherwise certain functionalities like diff()
# relying on np.nan (which is a float) won't work very properly.
raise_if_not(
np.issubdtype(xa.values.dtype, np.number),
"The time series must contain numeric values only.",
logger,
)
val_dtype = xa.values.dtype
if not (
np.issubdtype(val_dtype, np.float64) or np.issubdtype(val_dtype, np.float32)
):
logger.warning(
"TimeSeries is using a numeric type different from np.float32 or np.float64. "
"Not all functionalities may work properly. It is recommended casting your data to floating "
"point numbers before using TimeSeries."
)
if xa.dims[-2:] != DIMS[-2:]:
# The first dimension represents the time and may be named differently.
raise_log(
ValueError(
"The last two dimensions of the DataArray must be named {}".format(
DIMS[-2:]
)
),
logger,
)
# check that columns/component names are unique
components = xa.get_index(DIMS[1])
raise_if_not(
len(set(components)) == len(components),
"The components (columns) names must be unique. Provided: {}".format(
components
),
logger,
)
self._time_dim = str(
xa.dims[0]
) # how the time dimension is named; we convert hashable to string
# The following sorting returns a copy, which we are relying on.
# As of xarray 0.18.2, this sorting discards the freq of the index for some reason
# https://github.com/pydata/xarray/issues/5466
# We sort only if the time axis is not already sorted (monotonically increasing).
self._xa = self._sort_index(xa, copy=True)
self._time_index = self._xa.get_index(self._time_dim)
if not isinstance(self._time_index, VALID_INDEX_TYPES):
raise_log(
ValueError(
"The time dimension of the DataArray must be indexed either with a DatetimeIndex "
"or with an RangeIndex."
),
logger,
)
self._has_datetime_index = isinstance(self._time_index, pd.DatetimeIndex)
if self._has_datetime_index:
freq_tmp = xa.get_index(
self._time_dim
).freq # store original freq (see bug of sortby() above).
self._freq: pd.DateOffset = (
freq_tmp
if freq_tmp is not None
else to_offset(self._xa.get_index(self._time_dim).inferred_freq)
)
raise_if(
self._freq is None,
"The time index of the provided DataArray is missing the freq attribute, and the frequency could "
"not be directly inferred. "
"This probably comes from inconsistent date frequencies with missing dates. "
"If you know the actual frequency, try setting `fill_missing_dates=True, freq=actual_frequency`. "
"If not, try setting `fill_missing_dates=True, freq=None` to see if a frequency can be inferred.",
logger,
)
self._freq_str: str = self._freq.freqstr
# reset freq inside the xarray index (see bug of sortby() above).
self._xa.get_index(self._time_dim).freq = self._freq
# We have to check manually if the index is complete for non-empty series. Another way could
# be to rely on `inferred_freq` being present, but this fails for series of length < 3.
if len(self._time_index) > 0:
is_index_complete = (
len(
pd.date_range(
self._time_index.min(),
self._time_index.max(),
freq=self._freq,
).difference(self._time_index)
)
== 0
)
raise_if_not(
is_index_complete,
"Not all timestamps seem to be present in the time index. Does "
"the series contain holes? If you are using a factory method, "
"try specifying `fill_missing_dates=True` "
"or specify the `freq` parameter.",
logger,
)
else:
self._freq = self._time_index.step
self._freq_str = None
# check static covariates
static_covariates = self._xa.attrs.get(STATIC_COV_TAG, None)
raise_if_not(
isinstance(static_covariates, (pd.Series, pd.DataFrame))
or static_covariates is None,
"`static_covariates` must be either a pandas Series, DataFrame or None",
logger,
)
# check if valid static covariates for multivariate TimeSeries
if isinstance(static_covariates, pd.DataFrame):
n_components = len(static_covariates)
raise_if(
n_components > 1 and n_components != self.n_components,
"When passing a multi-row pandas DataFrame, the number of rows must match the number of "
"components of the TimeSeries object (multi-component/multi-row static covariates must map to each "
"TimeSeries component).",
logger,
)
static_covariates = static_covariates.copy()
elif isinstance(static_covariates, pd.Series):
static_covariates = static_covariates.to_frame().T
else: # None
pass
# prepare static covariates:
if static_covariates is not None:
static_covariates.index = (
self.components
if len(static_covariates) == self.n_components
else [DEFAULT_GLOBAL_STATIC_COV_NAME]
)
static_covariates.columns.name = STATIC_COV_TAG
# convert numerical columns to same dtype as series
# we get all numerical columns, except those that have right dtype already
cols_to_cast = static_covariates.select_dtypes(
include=np.number, exclude=self.dtype
).columns
changes = {col: self.dtype for col in cols_to_cast}
# Calling astype is costly even when there's no change...
if len(changes) > 0:
static_covariates = static_covariates.astype(changes, copy=False)
# handle hierarchy
hierarchy = self._xa.attrs.get(HIERARCHY_TAG, None)
self._top_level_component = None
self._bottom_level_components = None
if hierarchy is not None:
raise_if_not(
isinstance(hierarchy, dict),
"The hierarchy must be a dict mapping (non-top) component names to their parent(s) in the hierarchy.",
)
# pre-compute grouping informations
components_set = set(self.components)
children = set(hierarchy.keys())
# convert string ancestors to list of strings
hierarchy = {
k: ([v] if isinstance(v, str) else v) for k, v in hierarchy.items()
}
raise_if_not(
all(c in components_set for c in children),
"The keys of the hierarchy must be time series components",
)
ancestors = set().union(*hierarchy.values())
raise_if_not(
all(a in components_set for a in ancestors),
"The values of the hierarchy must only contain component names matching those of the series.",
)
hierarchy_top = components_set - children
raise_if_not(
len(hierarchy_top) == 1,
"The hierarchy must be such that only one component does "
+ "not appear as a key (the top level component).",
)
self._top_level_component = hierarchy_top.pop()
raise_if_not(
self._top_level_component in ancestors,
"Invalid hierarchy. Component {} appears as it should be top-level, but "
+ "does not appear as an ancestor in the hierarchy dict.",
)
bottom_level = components_set - ancestors
# maintain the same order as the original components
self._bottom_level_components = [
c for c in self.components if c in bottom_level
]
# Store static covariates and hierarchy in attributes (potentially storing None)
self._xa = _xarray_with_attrs(self._xa, static_covariates, hierarchy)
"""
Factory Methods
===============
"""
@classmethod
def from_xarray(
cls,
xa: xr.DataArray,
fill_missing_dates: Optional[bool] = False,
freq: Optional[Union[str, int]] = None,
fillna_value: Optional[float] = None,
) -> Self:
"""
Return a TimeSeries instance built from an xarray DataArray.
The dimensions of the DataArray have to be (time, component, sample), in this order. The time
dimension can have an arbitrary name, but component and sample must be named "component" and "sample",
respectively.
The first dimension (time), and second dimension (component) must be indexed (i.e., have coordinates).
The time must be indexed either with a pandas DatetimeIndex, a pandas RangeIndex, or a pandas Index that can
be converted to a RangeIndex. It is better if the index has no holes; alternatively setting
`fill_missing_dates` can in some cases solve these issues (filling holes with NaN, or with the provided
`fillna_value` numeric value, if any).
If two components have the same name or are not strings, this method will disambiguate the components
names by appending a suffix of the form "<name>_N" to the N-th column with name "name".
The component names in the static covariates and hierarchy (if any) are *not* disambiguated.
Parameters
----------
xa
The xarray DataArray
fill_missing_dates
Optionally, a boolean value indicating whether to fill missing dates (or indices in case of integer index)
with NaN values. This requires either a provided `freq` or the possibility to infer the frequency from the
provided timestamps. See :meth:`_fill_missing_dates() <TimeSeries._fill_missing_dates>` for more info.
freq
Optionally, a string or integer representing the frequency of the underlying index. This is useful in order
to fill in missing values if some dates are missing and `fill_missing_dates` is set to `True`.
If a string, represents the frequency of the pandas DatetimeIndex (see `offset aliases
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_ for more info on
supported frequencies).
If an integer, represents the step size of the pandas Index or pandas RangeIndex.
fillna_value
Optionally, a numeric value to fill missing values (NaNs) with.
Returns
-------
TimeSeries
A univariate or multivariate deterministic TimeSeries constructed from the inputs.
"""
xa_index = xa.get_index(xa.dims[0])
has_datetime_index = isinstance(xa_index, pd.DatetimeIndex)
has_range_index = isinstance(xa_index, pd.RangeIndex)
has_integer_index = not (has_datetime_index or has_range_index)
has_frequency = (
has_datetime_index and xa_index.freq is not None
) or has_range_index
# optionally fill missing dates; do it only when there is a DatetimeIndex (and not a RangeIndex)
if fill_missing_dates:
xa_ = cls._fill_missing_dates(xa, freq=freq)
# provided index does not have a freq; using the provided freq
elif (
(has_datetime_index or has_integer_index)
and freq is not None
and not has_frequency
):
xa_ = cls._restore_xarray_from_frequency(xa, freq=freq)
# index is an integer index and no freq is provided; try convert it to pd.RangeIndex
elif has_integer_index and freq is None:
xa_ = cls._integer_to_range_indexed_xarray(xa)
else:
xa_ = xa
if fillna_value is not None:
xa_ = xa_.fillna(fillna_value)
# clean components (columns) names if needed (if names are not unique, or not strings)
components = xa_.get_index(DIMS[1])
if len(set(components)) != len(components) or any(
[not isinstance(s, str) for s in components]
):
def _clean_component_list(columns) -> List[str]:
# return a list of string containing column names
# make each column name unique in case some columns have the same names
clist = columns.to_list()
# convert everything to string if needed
for i, column in enumerate(clist):
if not isinstance(column, str):
clist[i] = str(column)
has_duplicate = len(set(clist)) != len(clist)
while has_duplicate:
# we may have to loop several times (e.g. we could have columns ["0", "0_1", "0"] and not
# noticing when renaming the last "0" into "0_1" that "0_1" already exists...)
name_to_occurence = defaultdict(int)
for i, column in enumerate(clist):
name_to_occurence[clist[i]] += 1
if name_to_occurence[clist[i]] > 1:
clist[i] = clist[i] + "_{}".format(
name_to_occurence[clist[i]] - 1
)
has_duplicate = len(set(clist)) != len(clist)
return clist
time_index_name = xa_.dims[0]
columns_list = _clean_component_list(components)
# Note: an option here could be to also rename the component names in the static covariates
# and/or hierarchy, if any. However, we decide not to do so as those are directly dependent on the
# component names to work properly, so in case there's any name conflict it's better solved
# by the user than handled by silent renaming, which can change the way things work.
# TODO: is there a way to just update the component index without re-creating a new DataArray?
xa_ = xr.DataArray(
xa_.values,
dims=xa_.dims,
coords={
time_index_name: xa_.get_index(time_index_name),
DIMS[1]: columns_list,
},
attrs=xa_.attrs,
)
# We cast the array to float
if np.issubdtype(xa_.values.dtype, np.float32) or np.issubdtype(
xa_.values.dtype, np.float64
):
return cls(xa_)
else:
return cls(xa_.astype(np.float64))
@classmethod
def from_csv(
cls,
filepath_or_buffer,
time_col: Optional[str] = None,
value_cols: Optional[Union[List[str], str]] = None,
fill_missing_dates: Optional[bool] = False,
freq: Optional[Union[str, int]] = None,
fillna_value: Optional[float] = None,
static_covariates: Optional[Union[pd.Series, pd.DataFrame]] = None,
hierarchy: Optional[Dict] = None,
**kwargs,
) -> Self:
"""
Build a deterministic TimeSeries instance built from a single CSV file.
One column can be used to represent the time (if not present, the time index will be a RangeIndex)
and a list of columns `value_cols` can be used to indicate the values for this time series.
Parameters
----------
filepath_or_buffer
The path to the CSV file, or the file object; consistent with the argument of `pandas.read_csv` function
time_col
The time column name. If set, the column will be cast to a pandas DatetimeIndex (if it contains
timestamps) or a RangeIndex (if it contains integers).
If not set, the pandas RangeIndex will be used.
value_cols
A string or list of strings representing the value column(s) to be extracted from the CSV file. If set to
`None`, all columns from the CSV file will be used (except for the time_col, if specified)
fill_missing_dates
Optionally, a boolean value indicating whether to fill missing dates (or indices in case of integer index)
with NaN values. This requires either a provided `freq` or the possibility to infer the frequency from the
provided timestamps. See :meth:`_fill_missing_dates() <TimeSeries._fill_missing_dates>` for more info.
freq
Optionally, a string or integer representing the frequency of the underlying index. This is useful in order
to fill in missing values if some dates are missing and `fill_missing_dates` is set to `True`.
If a string, represents the frequency of the pandas DatetimeIndex (see `offset aliases
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_ for more info on
supported frequencies).
If an integer, represents the step size of the pandas Index or pandas RangeIndex.
fillna_value
Optionally, a numeric value to fill missing values (NaNs) with.
static_covariates
Optionally, a set of static covariates to be added to the TimeSeries. Either a pandas Series or a pandas
DataFrame. If a Series, the index represents the static variables. The covariates are globally 'applied'
to all components of the TimeSeries. If a DataFrame, the columns represent the static variables and the
rows represent the components of the uni/multivariate TimeSeries. If a single-row DataFrame, the covariates
are globally 'applied' to all components of the TimeSeries. If a multi-row DataFrame, the number of
rows must match the number of components of the TimeSeries (in this case, the number of columns in the CSV
file). This adds control for component-specific static covariates.
hierarchy
Optionally, a dictionary describing the grouping(s) of the time series. The keys are component names, and
for a given component name `c`, the value is a list of component names that `c` "belongs" to. For instance,
if there is a `total` component, split both in two divisions `d1` and `d2` and in two regions `r1` and `r2`,
and four products `d1r1` (in division `d1` and region `r1`), `d2r1`, `d1r2` and `d2r2`, the hierarchy would
be encoded as follows.
.. highlight:: python
.. code-block:: python
hierarchy={
"d1r1": ["d1", "r1"],
"d1r2": ["d1", "r2"],
"d2r1": ["d2", "r1"],
"d2r2": ["d2", "r2"],
"d1": ["total"],
"d2": ["total"],
"r1": ["total"],
"r2": ["total"]
}
..
The hierarchy can be used to reconcile forecasts (so that the sums of the forecasts at
different levels are consistent), see `hierarchical reconciliation
<https://unit8co.github.io/darts/generated_api/darts.dataprocessing.transformers.reconciliation.html>`_.
**kwargs
Optional arguments to be passed to `pandas.read_csv` function
Returns
-------
TimeSeries
A univariate or multivariate deterministic TimeSeries constructed from the inputs.
"""
df = pd.read_csv(filepath_or_buffer=filepath_or_buffer, **kwargs)
return cls.from_dataframe(
df=df,
time_col=time_col,
value_cols=value_cols,
fill_missing_dates=fill_missing_dates,
freq=freq,
fillna_value=fillna_value,
static_covariates=static_covariates,
hierarchy=hierarchy,
)
@classmethod
def from_dataframe(
cls,
df: pd.DataFrame,
time_col: Optional[str] = None,
value_cols: Optional[Union[List[str], str]] = None,
fill_missing_dates: Optional[bool] = False,
freq: Optional[Union[str, int]] = None,
fillna_value: Optional[float] = None,
static_covariates: Optional[Union[pd.Series, pd.DataFrame]] = None,
hierarchy: Optional[Dict] = None,
) -> Self:
"""
Build a deterministic TimeSeries instance built from a selection of columns of a DataFrame.
One column (or the DataFrame index) has to represent the time,
and a list of columns `value_cols` has to represent the values for this time series.
Parameters
----------
df
The DataFrame
time_col
The time column name. If set, the column will be cast to a pandas DatetimeIndex (if it contains
timestamps) or a RangeIndex (if it contains integers).
If not set, the DataFrame index will be used. In this case the DataFrame must contain an index that is
either a pandas DatetimeIndex, a pandas RangeIndex, or a pandas Index that can be converted to a
RangeIndex. It is better if the index has no holes; alternatively setting `fill_missing_dates` can in some
cases solve these issues (filling holes with NaN, or with the provided `fillna_value` numeric value, if
any).
value_cols
A string or list of strings representing the value column(s) to be extracted from the DataFrame. If set to
`None`, the whole DataFrame will be used.
fill_missing_dates
Optionally, a boolean value indicating whether to fill missing dates (or indices in case of integer index)
with NaN values. This requires either a provided `freq` or the possibility to infer the frequency from the
provided timestamps. See :meth:`_fill_missing_dates() <TimeSeries._fill_missing_dates>` for more info.
freq
Optionally, a string or integer representing the frequency of the underlying index. This is useful in order
to fill in missing values if some dates are missing and `fill_missing_dates` is set to `True`.
If a string, represents the frequency of the pandas DatetimeIndex (see `offset aliases
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_ for more info on
supported frequencies).
If an integer, represents the step size of the pandas Index or pandas RangeIndex.
fillna_value
Optionally, a numeric value to fill missing values (NaNs) with.
static_covariates
Optionally, a set of static covariates to be added to the TimeSeries. Either a pandas Series or a pandas
DataFrame. If a Series, the index represents the static variables. The covariates are globally 'applied'
to all components of the TimeSeries. If a DataFrame, the columns represent the static variables and the
rows represent the components of the uni/multivariate TimeSeries. If a single-row DataFrame, the covariates
are globally 'applied' to all components of the TimeSeries. If a multi-row DataFrame, the number of
rows must match the number of components of the TimeSeries (in this case, the number of columns in
``value_cols``). This adds control for component-specific static covariates.
hierarchy
Optionally, a dictionary describing the grouping(s) of the time series. The keys are component names, and
for a given component name `c`, the value is a list of component names that `c` "belongs" to. For instance,
if there is a `total` component, split both in two divisions `d1` and `d2` and in two regions `r1` and `r2`,
and four products `d1r1` (in division `d1` and region `r1`), `d2r1`, `d1r2` and `d2r2`, the hierarchy would
be encoded as follows.
.. highlight:: python
.. code-block:: python
hierarchy={
"d1r1": ["d1", "r1"],
"d1r2": ["d1", "r2"],
"d2r1": ["d2", "r1"],
"d2r2": ["d2", "r2"],
"d1": ["total"],
"d2": ["total"],
"r1": ["total"],
"r2": ["total"]
}
..
The hierarchy can be used to reconcile forecasts (so that the sums of the forecasts at
different levels are consistent), see `hierarchical reconciliation
<https://unit8co.github.io/darts/generated_api/darts.dataprocessing.transformers.reconciliation.html>`_.
Returns
-------
TimeSeries
A univariate or multivariate deterministic TimeSeries constructed from the inputs.
"""
# get values
if value_cols is None:
series_df = df.loc[:, df.columns != time_col]
else:
if isinstance(value_cols, str):
value_cols = [value_cols]
series_df = df[value_cols]
# get time index
if time_col:
if time_col not in df.columns:
raise_log(AttributeError(f"time_col='{time_col}' is not present."))
time_index = pd.Index([])
time_col_vals = df[time_col]
if np.issubdtype(time_col_vals.dtype, object):
# Try to convert to integers if needed
try:
time_col_vals = time_col_vals.astype(int)
except ValueError:
pass
if np.issubdtype(time_col_vals.dtype, np.integer):
# We have to check all integers appear only once to have a valid index
raise_if(
time_col_vals.duplicated().any(),
"The provided integer time index column contains duplicate values.",
)
# Temporarily use an integer Index to sort the values, and replace by a
# RangeIndex in `TimeSeries.from_xarray()`
time_index = pd.Index(time_col_vals)
elif np.issubdtype(time_col_vals.dtype, object):
# The integer conversion failed; try datetimes
try:
time_index = pd.DatetimeIndex(time_col_vals)
except ValueError:
raise_log(
AttributeError(
"'time_col' is of 'object' dtype but doesn't contain valid timestamps"
)
)
elif np.issubdtype(time_col_vals.dtype, np.datetime64):
time_index = pd.DatetimeIndex(time_col_vals)
else:
raise_log(
AttributeError(
"Invalid type of `time_col`: it needs to be of either 'str', 'datetime' or 'int' dtype."
)
)
time_index.name = time_col
else:
raise_if_not(
isinstance(df.index, VALID_INDEX_TYPES)
or np.issubdtype(df.index.dtype, np.integer),
"If time_col is not specified, the DataFrame must be indexed either with "
"a DatetimeIndex, a RangeIndex, or an integer Index that can be converted into a RangeIndex",
logger,
)
# BUGFIX : force time-index to be timezone naive as xarray doesn't support it
# pandas.DataFrame loses the tz information if it's not its index
if isinstance(df.index, pd.DatetimeIndex) and df.index.tz is not None:
logger.warning(
"The provided DatetimeIndex was associated with a timezone, which is currently not supported "
"by xarray. To avoid unexpected behaviour, the tz information was removed. Consider calling "
f"`ts.time_index.tz_localize({df.index.tz})` when exporting the results."
"To plot the series with the right time steps, consider setting the matplotlib.pyplot "
"`rcParams['timezone']` parameter to automatically convert the time axis back to the "
"original timezone."
)
time_index = df.index.tz_localize(None)
else:
time_index = df.index
if not time_index.name:
time_index.name = time_col if time_col else DIMS[0]
if series_df.columns.name:
series_df.columns.name = None
xa = xr.DataArray(
series_df.values[:, :, np.newaxis],
dims=(time_index.name,) + DIMS[-2:],
coords={time_index.name: time_index, DIMS[1]: series_df.columns},
attrs={STATIC_COV_TAG: static_covariates, HIERARCHY_TAG: hierarchy},
)
return cls.from_xarray(
xa=xa,
fill_missing_dates=fill_missing_dates,
freq=freq,
fillna_value=fillna_value,
)
@classmethod
def from_group_dataframe(
cls,
df: pd.DataFrame,
group_cols: Union[List[str], str],
time_col: Optional[str] = None,
value_cols: Optional[Union[List[str], str]] = None,
static_cols: Optional[Union[List[str], str]] = None,
fill_missing_dates: Optional[bool] = False,
freq: Optional[Union[str, int]] = None,
fillna_value: Optional[float] = None,
) -> List["TimeSeries"]:
"""
Build a list of TimeSeries instances grouped by a selection of columns from a DataFrame.
One column (or the DataFrame index) has to represent the time,
a list of columns `group_cols` must be used for extracting the individual TimeSeries by groups,
and a list of columns `value_cols` has to represent the values for the individual time series.
Values from columns ``group_cols`` and ``static_cols`` are added as static covariates to the resulting
TimeSeries objects. These can be viewed with `my_series.static_covariates`. Different to `group_cols`,
`static_cols` only adds the static values but are not used to extract the TimeSeries groups.
Parameters
----------
df
The DataFrame
group_cols
A string or list of strings representing the columns from the DataFrame by which to extract the
individual TimeSeries groups.
time_col
The time column name. If set, the column will be cast to a pandas DatetimeIndex (if it contains
timestamps) or a RangeIndex (if it contains integers).
If not set, the DataFrame index will be used. In this case the DataFrame must contain an index that is
either a pandas DatetimeIndex, a pandas RangeIndex, or a pandas Index that can be converted to a
RangeIndex. Be aware that the index must represents the actual index of each individual time series group
(can contain non-unique values). It is better if the index has no holes; alternatively setting
`fill_missing_dates` can in some cases solve these issues (filling holes with NaN, or with the provided
`fillna_value` numeric value, if any).
value_cols
A string or list of strings representing the value column(s) to be extracted from the DataFrame. If set to
`None`, the whole DataFrame will be used.
static_cols
A string or list of strings representing static variable columns from the DataFrame that should be
appended as static covariates to the resulting TimeSeries groups. Different to `group_cols`, the
DataFrame is not grouped by these columns. Note that for every group, there must be exactly one
unique value.
fill_missing_dates
Optionally, a boolean value indicating whether to fill missing dates (or indices in case of integer index)
with NaN values. This requires either a provided `freq` or the possibility to infer the frequency from the
provided timestamps. See :meth:`_fill_missing_dates() <TimeSeries._fill_missing_dates>` for more info.
freq
Optionally, a string or integer representing the frequency of the underlying index. This is useful in order
to fill in missing values if some dates are missing and `fill_missing_dates` is set to `True`.
If a string, represents the frequency of the pandas DatetimeIndex (see `offset aliases
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_ for more info on
supported frequencies).
If an integer, represents the step size of the pandas Index or pandas RangeIndex.
fillna_value
Optionally, a numeric value to fill missing values (NaNs) with.
Returns
-------
List[TimeSeries]
A list containing a univariate or multivariate deterministic TimeSeries per group in the DataFrame.
"""
if time_col is None and df.index.is_monotonic_increasing:
logger.warning(
"UserWarning: `time_col` was not set and `df` has a monotonically increasing (time) index. This "
"results in time series groups with non-overlapping (time) index. You can ignore this warning if the "
"index represents the actual index of each individual time series group."
)
group_cols = [group_cols] if not isinstance(group_cols, list) else group_cols
if static_cols is not None:
static_cols = (
[static_cols] if not isinstance(static_cols, list) else static_cols
)
else:
static_cols = []
static_cov_cols = group_cols + static_cols
# split df by groups, and store group values and static values (static covariates)
# single elements group columns must be unpacked for same groupby() behavior across different pandas versions
splits = []
for static_cov_vals, group in df.groupby(
group_cols[0] if len(group_cols) == 1 else group_cols
):
static_cov_vals = (
(static_cov_vals,)
if not isinstance(static_cov_vals, tuple)
else static_cov_vals
)
# check that for each group there is only one unique value per column in `static_cols`
if static_cols:
static_cols_valid = [
len(group[col].unique()) == 1 for col in static_cols
]
if not all(static_cols_valid):
# encountered performance issues when evaluating the error message from below in every
# iteration with `raise_if_not(all(static_cols_valid), message, logger)`
invalid_cols = [
static_col
for static_col, is_valid in zip(static_cols, static_cols_valid)
if not is_valid
]
raise_if(
True,
f"Encountered more than one unique value in group {group} for given static columns: "
f"{invalid_cols}.",
logger,
)
# add the static covariates to the group values
static_cov_vals += tuple(group[static_cols].values[0])
# store static covariate Series and group DataFrame (without static cov columns)
splits.append(
(
pd.DataFrame([static_cov_vals], columns=static_cov_cols),
group.drop(columns=static_cov_cols),
)
)
# create a list with multiple TimeSeries and add static covariates
return [
TimeSeries.from_dataframe(
df=split,
time_col=time_col,
value_cols=value_cols,
fill_missing_dates=fill_missing_dates,
freq=freq,
fillna_value=fillna_value,
static_covariates=static_covs,
)
for static_covs, split in splits
]
@classmethod
def from_series(
cls,
pd_series: pd.Series,
fill_missing_dates: Optional[bool] = False,
freq: Optional[Union[str, int]] = None,
fillna_value: Optional[float] = None,
static_covariates: Optional[Union[pd.Series, pd.DataFrame]] = None,
) -> Self:
"""
Build a univariate deterministic series from a pandas Series.
The series must contain an index that is either a pandas DatetimeIndex, a pandas RangeIndex, or a pandas Index
that can be converted into a RangeIndex. It is better if the index has no holes; alternatively setting
`fill_missing_dates` can in some cases solve these issues (filling holes with NaN, or with the provided
`fillna_value` numeric value, if any).
Parameters
----------
pd_series
The pandas Series instance.
fill_missing_dates
Optionally, a boolean value indicating whether to fill missing dates (or indices in case of integer index)
with NaN values. This requires either a provided `freq` or the possibility to infer the frequency from the
provided timestamps. See :meth:`_fill_missing_dates() <TimeSeries._fill_missing_dates>` for more info.
freq
Optionally, a string or integer representing the frequency of the underlying index. This is useful in order
to fill in missing values if some dates are missing and `fill_missing_dates` is set to `True`.
If a string, represents the frequency of the pandas DatetimeIndex (see `offset aliases
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_ for more info on
supported frequencies).
If an integer, represents the step size of the pandas Index or pandas RangeIndex.
fillna_value
Optionally, a numeric value to fill missing values (NaNs) with.
static_covariates
Optionally, a set of static covariates to be added to the TimeSeries. Either a pandas Series or a
single-row pandas DataFrame. If a Series, the index represents the static variables. If a DataFrame, the
columns represent the static variables and the single row represents the univariate TimeSeries component.
Returns
-------
TimeSeries
A univariate and deterministic TimeSeries constructed from the inputs.
"""
df = pd.DataFrame(pd_series)
return cls.from_dataframe(
df,
time_col=None,
value_cols=None,
fill_missing_dates=fill_missing_dates,
freq=freq,
fillna_value=fillna_value,
static_covariates=static_covariates,
)
@classmethod
def from_times_and_values(
cls,
times: Union[pd.DatetimeIndex, pd.RangeIndex, pd.Index],
values: np.ndarray,
fill_missing_dates: Optional[bool] = False,
freq: Optional[Union[str, int]] = None,
columns: Optional[pd._typing.Axes] = None,
fillna_value: Optional[float] = None,
static_covariates: Optional[Union[pd.Series, pd.DataFrame]] = None,
hierarchy: Optional[Dict] = None,
) -> Self:
"""
Build a series from a time index and value array.
Parameters
----------
times
A pandas DateTimeIndex, RangeIndex, or Index that can be converted to a RangeIndex representing the time
axis for the time series. It is better if the index has no holes; alternatively setting
`fill_missing_dates` can in some cases solve these issues (filling holes with NaN, or with the provided
`fillna_value` numeric value, if any).
values
A Numpy array of values for the TimeSeries. Both 2-dimensional arrays, for deterministic series,
and 3-dimensional arrays, for probabilistic series, are accepted. In the former case the dimensions
should be (time, component), and in the latter case (time, component, sample).
fill_missing_dates
Optionally, a boolean value indicating whether to fill missing dates (or indices in case of integer index)
with NaN values. This requires either a provided `freq` or the possibility to infer the frequency from the
provided timestamps. See :meth:`_fill_missing_dates() <TimeSeries._fill_missing_dates>` for more info.
freq
Optionally, a string or integer representing the frequency of the underlying index. This is useful in order
to fill in missing values if some dates are missing and `fill_missing_dates` is set to `True`.
If a string, represents the frequency of the pandas DatetimeIndex (see `offset aliases
<https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases>`_ for more info on
supported frequencies).
If an integer, represents the step size of the pandas Index or pandas RangeIndex.
columns
Columns to be used by the underlying pandas DataFrame.
fillna_value
Optionally, a numeric value to fill missing values (NaNs) with.
static_covariates
Optionally, a set of static covariates to be added to the TimeSeries. Either a pandas Series or a pandas
DataFrame. If a Series, the index represents the static variables. The covariates are globally 'applied'
to all components of the TimeSeries. If a DataFrame, the columns represent the static variables and the
rows represent the components of the uni/multivariate TimeSeries. If a single-row DataFrame, the covariates
are globally 'applied' to all components of the TimeSeries. If a multi-row DataFrame, the number of
rows must match the number of components of the TimeSeries (in this case, the number of columns in
``values``). This adds control for component-specific static covariates.
hierarchy
Optionally, a dictionary describing the grouping(s) of the time series. The keys are component names, and
for a given component name `c`, the value is a list of component names that `c` "belongs" to. For instance,
if there is a `total` component, split both in two divisions `d1` and `d2` and in two regions `r1` and `r2`,
and four products `d1r1` (in division `d1` and region `r1`), `d2r1`, `d1r2` and `d2r2`, the hierarchy would
be encoded as follows.
.. highlight:: python
.. code-block:: python
hierarchy={
"d1r1": ["d1", "r1"],
"d1r2": ["d1", "r2"],
"d2r1": ["d2", "r1"],
"d2r2": ["d2", "r2"],
"d1": ["total"],
"d2": ["total"],
"r1": ["total"],
"r2": ["total"]
}
..
The hierarchy can be used to reconcile forecasts (so that the sums of the forecasts at
different levels are consistent), see `hierarchical reconciliation
<https://unit8co.github.io/darts/generated_api/darts.dataprocessing.transformers.reconciliation.html>`_.
Returns
-------
TimeSeries
A TimeSeries constructed from the inputs.
"""
raise_if_not(
isinstance(times, VALID_INDEX_TYPES)
or np.issubdtype(times.dtype, np.integer),