Rclone copy stuck/hangs/does not finish

Help and Support
1

What is the problem you are having with rclone?

The file transfer is completed and shown as 100% but rclone does not finish the transfer. This means the temporary file extension is not changed to the final and no other transfers are started, because it seems to think it is not finished.

I am downloading bigger files 700MB to 4GB from my OneDrive. I am using copy and have tried to modify the timeout to double the max file size as suggested in another thread. I have also tried to limit the transfer to one or two files to see if that changes anything. Both measures have not worked.

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0
- os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
- os/kernel: 10.0.19045.4780 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.22.4
- go/linking: static
- go/tags: cmount

Are you on the latest version of rclone?
Yes

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy msod:/"Videos" "E:\Videos" --progress --timeout 8m --transfers=1

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[msod]
type = onedrive
region = global
token = XXX
drive_id = XXX
drive_type = personal

A log from the command that you were trying to run with the -vv flag

2024/08/22 10:54:17 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 28/29 (1811939328-1879048192) size 64Mi starting
2024/08/22 10:54:17 DEBUG : Couldn't decode error response: EOF
2024/08/22 10:54:17 DEBUG : DJI_20240728112205_0099_D.MP4.lejikor8.partial: writing chunk 26
2024/08/22 10:54:19 DEBUG : DJI_20240728112205_0099_D.MP4.lejikor8.partial: writing chunk 27
2024/08/22 10:55:14 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 25/29 (1610612736-1677721600) size 64Mi finished
2024/08/22 10:55:14 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 29/29 (1879048192-1931987608) size 50.487Mi starting
2024/08/22 10:55:14 DEBUG : Couldn't decode error response: EOF
2024/08/22 10:55:15 DEBUG : DJI_20240728112205_0099_D.MP4.lejikor8.partial: writing chunk 28
2024/08/22 10:55:18 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 26/29 (1677721600-1744830464) size 64Mi finished
2024/08/22 10:55:21 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 27/29 (1744830464-1811939328) size 64Mi finished
2024/08/22 10:55:22 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 28/29 (1811939328-1879048192) size 64Mi finished
2024/08/22 10:55:34 DEBUG : DJI_20240728112205_0099_D.MP4: multi-thread copy: chunk 29/29 (1879048192-1931987608) size 50.487Mi finished
2024/08/22 10:55:34 DEBUG : DJI_20240728112205_0099_D.MP4: Finished multi-thread copy with 29 parts of size 64Mi
2024/08/22 10:57:41 DEBUG : DJI_20240722095358_0090_D.MP4: quickxor = 62c545be9fa98e47bbd1cf0f46a929cc11e8e6b0 OK
2024/08/22 10:57:41 INFO  : DJI_20240722095358_0090_D.MP4: Updated modification time in destination
2024/08/22 10:57:41 DEBUG : DJI_20240722095358_0090_D.MP4: Unchanged skipping
2024/08/22 10:57:41 DEBUG : DJI_20240722103236_0096_D.MP4: Modification times differ by 645h7m36s: 2024-07-24 07:18:46 +0000 UTC, 2024-08-20 06:26:22 +0200 CEST
2024/08/22 10:59:55 DEBUG : DJI_20240722095139_0089_D.MP4: quickxor = 79f3f7630de85852e51f49253e2f29aaecff36f7 OK
2024/08/22 10:59:55 INFO  : DJI_20240722095139_0089_D.MP4: Updated modification time in destination
2024/08/22 10:59:55 DEBUG : DJI_20240722095139_0089_D.MP4: Unchanged skipping
2024/08/22 10:59:55 DEBUG : DJI_20240722103548_0097_D.MP4: Modification times differ by 645h6m50s: 2024-07-24 07:18:46 +0000 UTC, 2024-08-20 06:25:36 +0200 CEST

Transferred:        1.799 GiB / 162.813 GiB, 1%, 101.489 KiB/s, ETA 2w5d6h
Checks:                18 / 88, 20%
Transferred:            0 / 147, 0%
Elapsed time:     11m15.4s
Checking:
 *                 DJI_20240721131704_0085_D.MP4: checking
 *                 DJI_20240722094159_0086_D.MP4: checking
 *                 DJI_20240722094827_0088_D.MP4: checking
 *                 DJI_20240722095139_0089_D.MP4: checking
 *                 DJI_20240722095358_0090_D.MP4: checking
 *                 DJI_20240722102359_0093_D.MP4: checking
 *                 DJI_20240722102727_0094_D.MP4: checking
 *                 DJI_20240722102921_0095_D.MP4: checking
Transferring:
 *                 DJI_20240728112205_0099_D.MP4:100% /1.799Gi, 101.339Ki/s, 0s
2

When the file finishes downloading, it's doing a checksum and you are probably on slow storage.

You can reduce checkers to 1 and see if that helps as you are most likely IO bound with the default.

3

Thanks. I doubt that I am on slow storage as I am downloading at close 100 MBit/s, am paying for it and have a total of 2TB space available.

But at this point I am trying everything and will try to set checkers to 1. Thanks!

Edit: No. That did not make a difference. As a workaround I will try to increase the number of transfers to a ridiculous number and see if that will allow to download more efficiently.

4

Is your local storage a SSD or a mechanical HDD?

When file hits 100%, rclone will try to generate a checksum and compare it to the number received from OneDrive to confirm there were no discrepancies. Seems your local computer takes a long time doing that.

Checking the logs, you see the file stuck in 100% has finished copying:

2024/08/22 10:55:34 DEBUG : DJI_20240728112205_0099_D.MP4: Finished multi-thread copy with 29 parts of size 64Mi

You could also try using

--multi-thread-streams 1

This would basically stop multi thread download and just do all in one stream. Might help on the IO side locally.

5

Easy to rule out local checksums checking responsibility - turn it off and test:

--ignore-checksum

Normally rclone will check that the checksums of transferred files match, and give an error "corrupted on transfer" if they don't.

not recommended to be used for "production" but perfect for testing

IMO also it looks like local disk can not handle such volume of reads and writes at the same time. Is it actually a disk? or some SD card? 99% problem is here.

Try the same transfer to your local C: drive.

Other thing to check - I would remove:

not sure what you might need it for...

6

Wow, what a great forum. Thanks for the answers. I succeeded in downloading all the files I needed. It was slow (see above), but still way more comfortable than using the OneDrive interface.
I currently lack the time and energy to try out if the suggestions work, but will try at a later point to evaluate if any of those would have helped speed up the process.

Thanks again for the suggestions!!

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.