Parquets (and arrow files) not being removed before a run (not a bug)

[ajstewart]

I think usually this was fine as the files are just overwritten, but with the arrow files being present a user might run with arrow files on, and then re-run with arrow files off. In this case the arrow files need to be removed from the job directory.

The forced parquets are removed here:

vast-pipeline/vast_pipeline/management/commands/runpipeline.py

Lines 102 to 113 in 432b6af

	# clean up pipeline images and forced measurements for re-runs
	if flag_exist:
	logger.info('Cleaning up pipeline run before re-process data')
	p_run.image_set.clear()
	if not pipeline.config.MONITOR:
	logger.info(
	'Cleaning up forced measurements before re-process data'
	)
	forced_parquets = remove_forced_meas(p_run.path)
	for parquet in forced_parquets:
	os.remove(parquet)

I suggest that at this point we just remove all parquets and arrows from the job directory to be clean.