Repository navigation
Remove work directories for processes where input has a specific value #7123
Closed
valterschutz
started this conversation in
Ideas
Replies: 1 comment 2 replies
|
Hi @valterschutz . It's hard to say without a specific example, but it sounds like you have some processes that are not invalidating the cache when you expect them too (e.g. when modifying a python script), and you are working around that by deleting cache entries Instead I would make sure that you are defining your processes correctly, and calling your python scripts in a way that allows Nextflow to include it in the task hash. When done correctly, editing the python script should automatically re-execute the task |
2 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
My nextflow processes often invoke python scripts, which might change as I develop them. Thus, I often end up wanting to rerun a process but only for a very limited set of inputs.
Unlike #5308 and #6683 , I don't want to change the caching behavior. I just want to remove specific work directories, so that nextflow understands that some processes need to rerun.
I first propose that
nextflow cleanis extended with an option to invalidate a specific process, e.g.nextflow clean -process <process>. I currently achieve the same behavior by temporarily disabling thecachefor that specific process, but I think this would be a more convenient solution.My second proposal is that the above command is additionally extended to support the syntax
nextflow clean -process <process> -input-name <input_name> -input-value <input_value>. For the time being, I mostly care about this working with integers and strings.For my personal workflow --- which is largely based on training reinforcement learning agents --- this is (IMO) the biggest issue of Nextflow, which I am otherwise very impressed by. I'm surprised that I haven't found anyone else suggesting this yet. Am I missing something obvious?
All reactions