Incremental CSV destination

[ChristopheDuong]

What

see #1239

[cgardens]

short and sweet. 👌

one question that is blocking but once we resolve, will approve. good stuff.

[michel-tricot]

I am a bit nervous about appending to a file.
The day we want to expand that destination to GCS/S3 the current implementation won't work. Why don't we just create a new file with the incremental data an "sequence" the filenames?

[ChristopheDuong]

I am a bit nervous about appending to a file.
The day we want to expand that destination to GCS/S3 the current implementation won't work. Why don't we just create a new file with the incremental data an "sequence" the filenames?

With @cgardens's suggestions, we are not appending to a file anymore but we'll always overwrite the final file as a single file (not a folder with multiple partitioned files in it)

I think this consideration can be tackled once this destination is able to actually write in GCS/S3?

(plus the source-file can read only one file at a time at the moment!)

[cgardens]

nice!