Safe runs of multiple instances?

[noyannus]

Is it safe to run multiple instances of dude in parallel?

For huge reorganizations it would be good to set up several of the large dupe searches and let them run overnight.

If it is not possible -- would it be hard to implement? I imagine separate caches and logs for each instance, or merged caches and logs that contain a field to identify and filter for entries of individual instances.

[PJDude]

It would be the best to first identify the problem that prevents using single instance :) Is there any ?
Dude crc calculations are being run in parallel - one thread per device (yes - GIL is still there) Cache is stored also separately - per device. The log file is created with a resolution of seconds so running two instances in the same second is probably asking for trouble. Starting instances with an interval of >1s on different physical devices maybe have some chances to work however not recommended. I can imagine (hardly :) ) running separate instances to maybe little faster cache preparation and then use all the data in single instance. I suspect that more problems may be caused by memory usage or the amount of data to be shown in the gui.

[noyannus]

Thanks for the quick reply!

It would be the best to first identify the problem that prevents using single instance :) Is there any ?

None. :-))) -- No, wait, there is one "luxury lament": The panes of "Paths To Scan" and "Exclude from scan" enlarge vertically with the window height; they always have the same height. For use cases with many (8) paths and no Excludes, or several Excludes and only 1 path it would be convenient to move the separator between these panes manually (or try vertical autofit). (And the "To" in "Paths To Scan" does not match the capitalization of the other fields.)

My problem is about running some dedupes in parallel that are large enough for an overnight run, and where the data do not overlap much. I imagine separate caches may require a bit more calculation (one extra time each for each data overlap), but then will provide a smaller search space or i/o load.

Dude crc calculations are being run in parallel - one thread per device (yes - GIL is still there) Cache is stored also separately - per device.

Ah, I see. For cache separation it will suffice to set up one dedupe job per device -- just copy the data over and I'm safe. A garbled log is tolerable for a one-time operation.