Local copy exceeds 9 hours

What is the problem you are having with rclone?

When copying ~1TB of data from source to destination on the same drive exceeds 9 hours. Is this normal?

What is your rclone version (output from rclone version)

1.56.0

Which cloud storage system are you using? (eg Google Drive)

Local

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --stats 10s --progress --checksum copy x:/source x:/destination

Without a log file, no idea as we have no information on what is going on.

@Animosity022 , there are no errors in the log. It simply takes longer than 9 hours to copy the data across.

You'd have to actually post the log to allow me to see it.

using that flag, rclone needs to checksum the files.
i was curious about the worst case so i did a quick test

DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "D:\\source\\" "D:\\dest\\" "--checksum" "--log-level=DEBUG" "--log-file=rclone.log" "--progress"]
DEBUG : Creating backend with remote "D:\\source\\"
DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : fs cache: renaming cache item "D:\\source\\" to be canonical "//?/D:/source/"
DEBUG : Creating backend with remote "D:\\dest\\"
DEBUG : fs cache: renaming cache item "D:\\dest\\" to be canonical "//?/D:/dest/"
DEBUG : Local file system at //?/d:/dest/: Waiting for checks to finish
DEBUG : file.txt: md5 = a77a31b8fd044e906defd48dd4474b26 (Local file system at //?/D:/source/)
DEBUG : file.txt: md5 = 8ab108942d1cef9982481c517d128450 (Local file system at //?/D:/dest/)
DEBUG : file.txt: md5 differa
DEBUG : Local file system at //?/d:/dest/: Waiting for transfers to finish
DEBUG : file.txt: md5 = a77a31b8fd044e906defd48dd4474b26 OK
INFO  : file.txt: Copied (replaced existing)

I'm not sure how best to upload the logs. It exceeds 100MB is size and redacting would be time consuming. Is there a test I could possibly run?

@asdffdsa, is there recommendation not to use checksum? If yes, how do I ensure that the integrity of the file transfer is maintained?

when rclone copies a file, it compares the checksum of the source and dest.

as a test, i suggest that you try to copy a single file, use debug output.
that way you can see what rclone is doing.
for example, notice the md5=

c:\data\rclone\scripts\rclone.exe copy D:\source d:\dest -vv 
DEBUG : rclone: Version "v1.56.0" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "copy" "D:\\source" "d:\\dest" "-vv"]
DEBUG : Creating backend with remote "D:\\source"
DEBUG : Using config file from "C:\\data\\rclone\\scripts\\rclone.conf"
DEBUG : fs cache: renaming cache item "D:\\source" to be canonical "//?/D:/source"
DEBUG : Creating backend with remote "d:\\dest"
DEBUG : fs cache: renaming cache item "d:\\dest" to be canonical "//?/d:/dest"
DEBUG : Local file system at //?/d:/dest: Waiting for checks to finish
DEBUG : Local file system at //?/d:/dest: Waiting for transfers to finish
DEBUG : file.txt: md5 = c4ca4238a0b923820dcc509a6f75849b OK
INFO  : file.txt: Copied (new)

As I understand it from the documentation, that the default is time & size. --checksum calculates the hash of each file which I am assuming is a byte for byte comparison. I'm unsure how limiting the check to only time & size is sufficient unless I've missed something.

In your example, it suggests the default appears to be a md5 hash. I'm unclear since the documentation suggests otherwise.

Just to make it clear: Time and size is checked to decide if files are different, and then if transfer is necessary it will be verified with checksum. By default.

each use-case if different and without knowing the details, i cannot give a good answer.

for local to local, md5 is used for hashing.

What are the odds of a conflict? I am assuming that if the default is md5, there is a risk of collisions. Can this be changed?

Can you give me an example of a use case where it may be insufficient so that I have an idea?

sorry, but insufficient is a fuzzy word, does not mean anything on its own.
you need to define in detail your use-case

MD5: The probability of just two hashes accidentally colliding is approximately: 1.47*10-29.

That would be the risk of file being corrupted in transfer, but in such a way that md5 does not change? Think the odds are pretty low..

No, rclone uses whatever the backend supports. Not sure what algo rclone check command with --download would use.

Put it another way, I would be keen to learn in what scenario that time and size would not work so that I have a better understanding.

what is your concern?
the comparing of two files to decide if a file transfer is required.
or
comparing of two files after a file transfer has completed.

The first and second. If there are no differences in the files, there should be no transfer.

Assuming the files don't match or does not exist at the destination, transfer the file and that it has done so successfully ensuring integrity.

Is this theoretical or actually possible? If the latter, that's concerning as there is data that is valuable.

i shared that earlier in the post.

147000000000000000000000000000

if you are a doctor from the future and your job is to transfer the contents of my brain to a new host, and need to validate the transfer, then i might have a concern.
other than that, i would not worry.