Modularise process_files.jl

[tlienart]

The function is now too big and there are some parts that should be reasonably easy to cut out.

1. the continuous checking part
2. the markdown transformation part
3. going through files and populating a dictionary accordingly

also really need to write some doc or think in more details about issues (currently all NOTE and TODO in the code).

1. dealing with assets that are changing
2. dealing with infra files that are changing and must cause a re-compile
3. having some verbosity level indicating what's being seen and what's going on

[tlienart]

all done after the fixed merge in 1074d6e