flatten_df is too slow

[manycoding]

1. Can it be rewritted to not use recursion?

2. If not, profile and see how to improve. To test, use jobs with nested fields and a considerable amount of items.

3. Add tests to check the speed

[manycoding]

Profiling:
https://notebook-qa-generic.scrapinghub.com/notebooks/GATF/Experiments/flatten_and_whitespaces.ipynb

[manycoding]

https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html

Note
It is worth noting that concat() (and therefore append()) makes a full copy of the data, and that constantly reusing this function can create a significant performance hit. If you need to use the operation over several datasets, use a list comprehension.

[manycoding]

What if instead of concatenating each time we keep new columns somewhere (in memory\disk) and concatenate in the end?

[ivankivanov]

I did several experiments with different versions. The latest one results:

• 5k items:

1. current code - 55 sec
2. new code - first level - 631 ms (5000 - 713), , second level - 4 secs - (5000 - 763)

• 150k items

1. current code - out of memory for 16 GB
2. new code - first level - 8.18 s - (141762, 777), second level - >1 min

The notebook is placed in: /shared/qa/Experiments/flat.ipynb
and the new branch for tests is flatten_df.

[manycoding]

@ivankivanov It might be a good idea to generate a df for testing so we can share notebooks here.

The idea looks doable, basically we replace recursion with a loop.
I don't see flatten_df branch.

[ivankivanov]

This is the pull request:

#74

Notebook:

shared/qa/Experiments/flattenFinal.ipynb

[ivankivanov]

sync with master
last notebook with performance: shared/qa/Experiments/flatten_df.ipynb

current code - 58.9 sec - full
new code - 6.68 sec - full
150k items
current code - out of memory for 32 GB
new code - first level - 7.96 s - (141762, 777), second level ~ 11 min