The copy (and sync) command only shows the Elapsed time increasing. No bytes are Transferred, even after waiting for days (tested with smaller files, e.g. 10GB, these work although only after two minutes of Elapsed time pass).
Run the command 'rclone version' and share the full output of the command.
rclone v1.58.0
os/version: freebsd 12.0-release-p4 (64 bit)
os/kernel: 12.0-release-p4 (amd64)
os/type: freebsd
os/arch: amd64
go/version: go1.17.8
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Microsoft azureblob
The command you were trying to run (eg rclone copy /tmp remote:tmp)
If you check the system, do you see disk IO? The first part is checksumming the file so if you have very slow storage, I'd imagine that is the bottleneck.
A 10GB file for me takes about 20 seconds to sum on slower spinning storage so 2 minutes seems really long.
felix@gemini:/data$ time md5sum testsum
2dd26c4d4799ebd29fa31e48d49e8e53 testsum
real 0m19.099s
user 0m14.426s
sys 0m4.671s
felix@gemini:/data$ du -sh testsum
10G testsum
I think that dd command is giving you some odd info as my slow spinning disk hits some 'not right' numbers as you have disk caching in the mix with that command.
felix@gemini:/data$ dd if=testsum of=/dev/null bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.635587 s, 1.7 GB/s
felix@gemini:/data$ dd if=testsum of=/dev/null bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.464849 s, 2.3 GB/s
felix@gemini:/data$ dd if=testsum of=/dev/null bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.361735 s, 3.0 GB/s
This is more what I'd expect from a slow spinning disk:
felix@gemini:/data$ mount | grep data
/dev/sdd1 on /data type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/)
felix@gemini:/data$ sudo hdparm -Tt /dev/sdd
/dev/sdd:
Timing cached reads: 38294 MB in 2.00 seconds = 19184.28 MB/sec
Timing buffered disk reads: 684 MB in 3.01 seconds = 227.46 MB/sec
Each block is checksummed and if I remember rightly there is a checksum of checksums.
What you are missing is storing an MD5 sum as metadata on that large file's object. That comes in useful for bitrot detection (eg with rclone check which won't work without it) but doesn't make the transfer any more secure.
It's not the fastest storage around the block, also this environment is hammered on - regardless, reading through the whole file takes finite amount of time:
time gdd if=largefile.dat of=/dev/null bs=1G
641+1 records in
641+1 records out
688751047612 bytes (689 GB, 641 GiB) copied, 3326.52 s, 207 MB/s
real 55m26.612s
user 0m0.008s
sys 8m33.407s
Could it be that the procedure which @ncw called "checksum of checksums" is not sequential and generates a lot of smaller IOPS or even random I/O patterns? Any options left to work with checksum while switching to something less intensive for large files on this somewhat slower data source? ( it's spinning rust )
@ncw chime in if you like - didn't want to skip you
That should be sequential, though remember rclone can transfer multiple files in parallel depending on the value of --transfers and scan multiple directories in parallel (controlled by --checkers). Often setting --checkers too high is counterproductive with hard disk based systems.
It looks like from the above you are just transferring one file though so this probably isn't a --checkers or --transfers problem.
If you want to simulate what rclone is doing making the checksum then use
rclone -P md5sum largefile.dat
I'd expect that to complete at about disk speed though unless you have a really slow CPU so for 641 GB file it should take about 55m not > 24hr.
You could disable checksums and transfer with a --min-size 100G (say) then complete the transfer without the --min-size and the checksum disable to hoover up the remaining things.
