Hi,
I have an rclone server copying files from an sftp remote to a SharePoint document library. It is quite a lot of files. I would like to change the copy to a move as we dont need the files on the sftp storage once we have it in SharePoint.
I did a dry run and it seems that it tries to copy over all files again and not just the deltas, did i read this correctly or did i misread it? I assumed it will just move the files that are not present in the Sharepoint Library and leave the rest as is?
rclone version)rclone copy /tmp remote:tmp)-vv flag (eg output from rclone -vv copy /tmp remote:tmp)rclone move -v --contimeout=3m --min-age 5m --checkers 1 --ignore-existing --user-agent "ISV|rclone|rclone/v1.42" --no-traverse sftp: sharepoint:
rclone v1.51.0
OS: Amazon Linux 6bit
I did not grab the log output and the question was more a rclone process question rather then an issue, wanted to know if the move command checks the destination for an existing file before it moves it or does it overwrite existing files?
Copy and move both do the same checking but move deletes from the source.
You'd need to share a debug with -vv to answer why it was being copied again.
It checks first and only moves/copies files if they do not exist on destination or are not up to date. The checking procedures are identical for move and copy - and pretty much the same on sync. rclone is pretty smart about not doing unnecessary work. This is default behavior.
Don't use --ignore-existing unless you understand fully what that means. It will cause rclone to not update any older versions of files if required. It will just skip it if it exists at all. While it has it's uses in niche cases it could easily result in your backup not being actually up-to-date with the most recent files, which is usually not what you want. To put it another way is is default behavior to "skip identical".
Usually this check will be on size+modtime, but if both the source and destination can support the same checksum then you can get a 100% accurate "paranoia-mode" check using checksum comparison. When transferring from local storage that does not natively store checksums (most typical filesystems) this is not on by default. You can force it with --checksum but this does of course necessitate that all files are fully read from disk and checksummed by CPU - which can be time-consuming and also not good for the load on the harddrives if it a often recurring script dealing with large amounts of data.
This may be a moot point if SFTP does not support checksum attributes for files though. I am not sure about that. rclone will tell you if you try to force --checksum but it is not possible to do.
I suspect you are probably just mis-reading the --dry-run info as it can be a little confusing. A copypaste of that would be a good start.
But if you can confirm that it is actually overwriting files again that it should not need to, then supply a logfile with -vv like Animosity says. You may find --log-file MyRcloneLog.txt useful to dump the output into a file directly because -vv (debug output) is overwhelmingly verbose.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.