makepipe compatible with git workflows?

[lschneiderbauer]

Hi!

thanks for the package! :)

I am trying to understand if I can use this package within a git workflow.
As far as I understand, the package is using the file modification dates to determine if a target should be remade or not, and it seems that this causes problems when storing the script files in a git repository. Git itself does not control the file modification time, so any time a checkout/something similar happends, the file modification date is adjusted (even though no meaningful modification might occur).

Are there any recommendations on how to use the package with R scripts within a git repository to support "correct" make behavior?`

Thanks!

[kinto-b]

Hey Lukas, thanks for the question. This is a delicate problem since in many instances the current behavior is desired. Imagine you modify foo.R, run your pipeline, think better of it and switch back to main. If original file modification times were preserved, makepipe wouldn't know that your targets are out of date.

You get the same behavior with GNU make and it's to defend against this problem. Actually it's part of the reason git doesn't preserve modification times. See:

https://archive.kernel.org/oldwiki/git.wiki.kernel.org/index.php/Git_FAQ.html#Why_isn.27t_Git_preserving_modification_time_on_files.3F

Note however that when you do a checkout, git won't modify files that haven't changed, so they should retain the same modification time. For example, if you also had bar.R that was unchanged, its file modification time would be the same before/after git checkout main.

---

All that being said, I do see the motivation for a workaround here.

I suppose one option would be to add a function to neutralize the effect of a git-checkout by updating the modification times of all the targets to the current time. So after git checkout you'd run makepipe::touch_targets(pipeline=p) or something.

Another would be to hash and cache everything and then compare hashes. However this would mean adding a ./.makepipe directory and would require much more time/work. (Let me know if you'd consider helping out with this). I believe this is closer to what targets does, so that package might be an option for you in the meantime, if you're willing to take on the overhead.

[kinto-b]

I've added a touch() method to the Pipeline object that can be used as a workaround in this instance. Once you've constructed your pipeline (you can do this without executing it by setting build=FALSE), you can run

p <- get_pipeline()
p$touch()

to update the file modification of all targets to the current time. This effectively 'forces' all targets into an up-to-date state.

You can use it any time you know your targets are up-to-date but, for whatever reason, the file mtimes are out of whack.

Edit: will be in next release

[lschneiderbauer]

Thanks @kinto-b !

Sorry I didn't reply earlier, your answer is much appreciated, there was just nothing to add from my side to your answer.

Thanks also for providing this workaround.

Best, Lukas