ImputeBench: A Comprehensive Benchmark for Time Series Imputation Techniques

ImputeBench implements over 20 advanced time series imputation algorithms. It evaluates their precision and runtime on eight real-world time series datasets using six missingness patterns. Technical details can be found in our PVLDB 2020 paper: Mind the Gap: An Experimental Evaluation of Imputation of Missing Values Techniques in Time Series . The benchmark can be easily extended with new algorithms (C/C++, Python, or Matlab), datasets, and missingness patterns.

• Initial Algorithms: The original benchmark implements the following algorithms (in C++):

  • CDRec: Scalable Recovery of Missing Blocks in Time Series with High and Low Cross-Correlations, KAIS'20
  • DynaMMo: DynaMMo: mining and summarization of coevolving sequences with missing values, KDD'09
  • GROUSE: Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation, PMLR'16
  • ROSL: Robust Orthonormal Subspace Learning: Efficient Recovery of Corrupted Low-Rank Matrices, CVPR'14
  • SoftImpute: Spectral Regularization Algorithms for Learning Large Incomplete Matrices, JMLR'10
  • SPIRIT*: Streaming pattern discovery in multiple time-series, VLDB'05
  • STMVL: ST-MVL: Filling Missing Values in Geo-Sensory Time Series Data, IJCAI'16
  • SVDImpute: Missing value estimation methods for DNA microarrays, BIOINFORMATICS'01
  • SVT: A Singular Value Thresholding Algorithm for Matrix Completion, SIAM J. OPTIM'10
  • TeNMF: Nonnegative Matrix Factorization for Time Series Recovery From a Few Temporal Aggregates, PMLR'17
  • TRMF: Temporal Regularized Matrix Factorization for High-dimensional Time Series Prediction, NIPS'16
  • TKCM*: Continuous Imputation of Missing Values in Streams of Pattern-Determining Time Series, EDBT'17

• Recently Integrated Algorithms: We recently expanded the original benchmark with new algorithms (using their original code):

  • MPIN: Missing Value Imputation for Multi-attribute Sensor Data Streams via Message Propagation, PVLDB'24
  • PriSTI: PriSTI: A Conditional Diffusion Framework for Spatiotemporal Imputation, ICDE'23
  • GRIN: Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks, ICLR'22
  • DeepMVI: Missing Value Imputation on Multidimensional Time Series, PVLDB'21
  • IIM*: Learning Individual Models for Imputation, ICDE '19
  • MRNN*: Estimating Missing Data in Temporal Data Streams Using Multi-Directional Recurrent Neural Networks, Trans. On Bio Eng.'19
  • BRITS: BRITS: Bidirectional Recurrent Imputation for Time Series, NeurIPS'18
  • SSA*: Model Agnostic Time Series Analysis via Matrix Estimation, Meas. Anal. Comput. Syst'18

• Algorithms under Integration:

  • DAMR: Dynamic Adjacency Matrix Representation Learning for Multivariate Time Series Imputation, SIGMOD'23
  • HKMF-T: HKMF-T: Recover From Blackouts in Tagged Time Series With Hankel Matrix Factorization, TKDE'21
  • NAOMI: NAOMI: Non-Autoregressive Multiresolution Sequence Imputation, NeurIPS'19
  • E2EGAN: E²GAN: End-to-End Generative Adversarial Network for Multivariate Time Series Imputation, IJCAI'19

• Datasets: All the datasets used in this benchmark can be found here.

• Missingness Patterns: The full list of recovery scenarios can be found here.

• Notes: The algorithms marked with ^* cannot handle multiple incomplete time series. They produce results only for the following scenarios: miss_perc, ts_length, and ts_nbr.

Prerequisites | Build | Execution | Extension | Contributors | Award | Citation

---

Prerequisites

• Ubuntu 20 or Ubuntu 22 (including Ubuntu derivatives, e.g., Xubuntu) or the same distribution under WSL.
• Clone this repository

---

Build

• Install mono from https://www.mono-project.com/download/stable/ and reboot your terminal.

• Build the Testing Framework using the installation script located in the root folder

    $ sh install_linux.sh

---

Execution

    $ cd TestingFramework/bin/Debug/
    $ mono TestingFramework.exe [arguments]

Arguments

-alg	-d	-scen
cdrec	airq	miss_perc
dynammo	bafu	ts_length
grouse	chlorine	ts_nbr
rosl	climate	miss_disj
softimp	drift10	miss_over
svdimp	electricity	mcar
svt	meteo	blackout
stmvl	temp	all
spirit	bafu_red
tenmf	drift10_red
tkcm	all
trmf
all
--------	--------	--------
New algs
--------	--------	--------
ssa
m-rnn
brits
deepmvi
mpin
pristi
iim

Results

All results and plots will be added to the Results folder. The accuracy results of all algorithms will be sequentially added for each scenario and dataset to: Results/.../.../error/. The runtime results of all algorithms will be added to: Results/.../.../runtime/. The plots of the recovered blocks will be added to the folder Results/.../.../recovery/plots/.

Execution examples

1. Run a single algorithm (cdrec) on a single dataset (drift10) using one scenario (missing percentage)

    $ mono TestingFramework.exe -alg cdrec -d drift10 -scen miss_perc

2. Run two algorithms (cdrec, spirit) on a single dataset (drift10) using one scenario (missing percentage)

    $ mono TestingFramework.exe -alg cdrec,spirit -d drift10 -scen miss_perc

3. Run point 2 without runtime results

    $ mono TestingFramework.exe -alg cdrec,spirit -d drift10 -scen miss_perc -nort

4. Run the whole VLDB'20 benchmark (all algorithms, all datasets, all scenarios, precision and runtime)

    $ mono TestingFramework.exe -alg all -d all -scen all

Warning: Running the whole benchmark takes a sizeable amount of time (up to 4 days, depending on the hardware) and produces up to 15GB of output files with all recovered data and plots unless stopped early.

5. Create patterns of missing blocks on one complete dataset (airq) using one scenario (missing percentage)

    $ mono TestingFramework.exe -alg mvexport -d airq -scen miss_perc

Note: You must run each scenario separately on one or multiple datasets. Each time you execute one scenario, the Results folder will be overwritten with the new files.

6. Additional command-line parameters

    $ mono TestingFramework.exe --help

Parametrized execution

• You can parametrize each algorithm using the command -algx. For example, you can run the svdimp algorithm with a reduction value of 4 on the drift dataset and by varying the sequence length as follows:

    $ mono TestingFramework.exe -algx svdimp 4 -d drift10 -scen ts_nbr

• If you want to run some algorithms with default parameters and some with customized ones, you can use -alg and -algx together. For example, you can run stmvl algorithm with default parameter and cdrec algorithm with a reduction value of 4 on the airq dataset by varying the sequence length as follows:

    $ mono TestingFramework.exe -alg stmvl -algx cdrec 4 -d airq -scen ts_nbr

Remark: The command -algx cannot be executed in a group and thus must precede the name of each algorithm.

---

Executing New Algorithms

• To evaluate the newly integrated algorithms, please install the following Python packages (takes several minutes):

    $ sh install_extra.sh

• Activate the virtual environment and execute the new algorithms from the table above

    $ source bench-env/bin/activate
    $ mono TestingFramework.exe [arguments]

Extension

• To add new algorithms:
  • C/C++: https://github.com/eXascaleInfolab/bench-vldb20/tree/master/NewAlgorithms/cpp
  • Python: https://github.com/eXascaleInfolab/bench-vldb20/tree/master/NewAlgorithms/python
  • Matlab: https://github.com/eXascaleInfolab/bench-vldb20/tree/master/NewAlgorithms/matlab
• To add new datasets:
  • import the file to TestingFramework/bin/Debug/data/{name}/{name}_normal.txt (name is the name of your dataset).
  • Requirements: rows>= 1'000, columns>= 10, column separator: empty space, row separator: newline
  • Note: the benchmark can also run with rows>= 100 and columns>= 5 but with a limited number of scenarios and algorithms.

---

Contributors

• Mourad Khayati ([email protected])
• Zakhar Tymchenko ([email protected]).

---

Award

ImputeBench has received the VLDB 2020 Best Experiments and Analysis Paper Award.

---

Citation

@article{DBLP:journals/pvldb/KhayatiLTC20,
  author       = {Mourad Khayati and
                  Alberto Lerner and
                  Zakhar Tymchenko and
                  Philippe Cudr{\'{e}}{-}Mauroux},
  title        = {Mind the Gap: An Experimental Evaluation of Imputation of Missing
                  Values Techniques in Time Series},
  journal      = {Proc. {VLDB} Endow.},
  volume       = {13},
  number       = {5},
  pages        = {768--782},
  year         = {2020},
  url          = {http://www.vldb.org/pvldb/vol13/p768-khayati.pdf},
  doi          = {10.14778/3377369.3377383},
  timestamp    = {Sat, 30 Sep 2023 10:24:09 +0200},
  biburl       = {https://dblp.org/rec/journals/pvldb/KhayatiLTC20.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}