What is the best way to merge directories, sub-directories and files with Rclone? Is it to use 'rclone copy' or 'rclone sync' -- without deleting files from the target/destination location? We have a large data and file consolidation project that requires a consistent and efficient way of moving data to a single archive location from many other directories and locations.
We have reviewed the link here - but we use a different cloud storage system (pCloud).
The Goal
The command we would like to run will do the following:
Get source directory and all subdirectories and files from the mount point.
Identify target/destination directory (with recursive merge of the source files) on the mount point.
Verify the files using time, date, checksum - to make sure they are the same
Merge/copy ONLY differences from the source location to the target location.
Remove duplicates from the source location.
So it is: rclone copy DriveMountPoint://Source DriveMountPoint://Archive --transfers 16 --progress --dry-run --stats --remove-source-files --log-file=mylogfile.txt
Does anyone have ideas on the best way too do this? What we are trying to eliminate is unnecessary copying on the pCloud storage drive from source to destination location.
Copy makes the source and destination the same, but does not remove any changes on the destination.
Sync makes the source identical to the destination by deleting items on the destination.
Generally this is the case, but does depend on the backend and if you are using crypt or not.
Copy and sync both only work with changes from source to destination.
You'd dedupe your source first as normally a tool that does copy or sync doesn't dedupe the source. What would that look like? What's a duplicate?
Over a long period of time, we have layers of backups (and backups of backups!). They have been stored in directories of different names. So the file IMG0001.JPG, for example, may be in 3 different named directories.
So merge for us means to copy from two or more different source directories and only keep one copy of the file after it is validated as being exactly the same as others with the same file name. We just want to make sure that "rclone copy" is going to be our best approach for that. We previously used RsyncOSX that had flags to not delete files on the target destination. It was, in effect, a one-way sync - just adding new files. That would work - but we don't see those flags in Rclone.
We have looked at the rclone dedupe command as well, but that appears to only dedupe on Google Drive.
So it appears the extra step of dedupe on the client systems, before uploading via rclone copy, is the workflow we need to be safe.
rlcone is one source to one destination so you can't do that in a single command.
I think that 'feels' like copy to me and not sync as would delete the destination.
Dedupe use case is just Google Drive really. It removes two of the same name files in the same directory as Google allows for this, but most file systems/mounts do not. I don't think that does what you want it to do.
Many thanks, again. The goal is ultimately consolidation of all backups into one logical and more manageable structure going forward. We will go with a workflow of manual thinking (oldest backups first) and rclone copy.... It appears to be the safest and most logical series of steps to achieve the goal. It will take a little bit longer, but we can be sure of the result.
The problem is not Rclone, the problem comes from decades of a mix of systems Win/OSX/Multiple Linux Distros/Unix/IOS... backups.
OSX backups, for example, had a few years where Apple kept moving iTunes and Photo directories around - so the backups have files that were copied into different locations. Just came across backups from this thing which was a blast when it came out. Somewhere back there we used it as a file server, and there were backups on that!
Rclone sync and Rclone copy are doing what we need them to do. So thanks, again - for your help.