Resuming downloads

Help and Support
1

What is the problem you are having with rclone?

I am failing to understand how to properly download large files over inconsistent links.

After i copy something and it is not fully done, the file is removed. Ideally, I would want to

  1. keep the downloaded portion.
  2. resume it next time.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0
- os/version: arch (64 bit)
- os/kernel: 6.7.8-2024q1-2 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google drive. Downloading only.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy mydrive:drive_dir/singlefile.mov local_dir

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[mydrive]
type = drive
root_folder_id = XXX
client_id = XXX
client_secret = XXX
scope = drive
pacer_min_sleep = 151ms
token = XXX
team_drive =

A log from the command that you were trying to run with the -vv flag

<6>INFO  : 
Transferred:        1.956 GiB / 44.937 GiB, 4%, 1.125 MiB/s, ETA 10h52m14s
Transferred:            0 / 1, 0%
Elapsed time:      30m4.7s
Transferring:
 *      20230725.mov:  4% /44.937Gi, 1.124Mi/s, 10h52m25s

^C<6>INFO  : Signal received: interrupt
<7>DEBUG : 20230725.mov: multi-thread copy: cancelling transfer on exit
<7>DEBUG : 20230725.mov: Reopening on read failure after offset 61402962 bytes: retry 1/10: context canceled
<7>DEBUG : 20230725.mov: Reopening on read failure after offset 59295042 bytes: retry 1/10: context canceled
<7>DEBUG : 20230725.mov: Reopen failed after offset 61402962 bytes read: open file failed: Get "https://www.googleapis.com/drive/v3/files/hash123?alt=media": context canceled
<7>DEBUG : 20230725.mov: multi-thread copy: chunk 32/719 failed: multi-thread copy: failed to write chunk: context canceled
<7>DEBUG : 20230725.mov: Reopening on read failure after offset 13971050 bytes: retry 1/10: context canceled
<7>DEBUG : 20230725.mov: multi-thread copy: chunk 34/719 (2214592512-2281701376) size 64Mi starting
<7>DEBUG : 20230725.mov: Reopen failed after offset 59295042 bytes read: open file failed: Get "https://www.googleapis.com/drive/v3/files/hash123?alt=media": context canceled
<7>DEBUG : 20230725.mov: Reopening on read failure after offset 20926406 bytes: retry 1/10: context canceled
<7>DEBUG : 20230725.mov: multi-thread copy: chunk 31/719 failed: multi-thread copy: failed to write chunk: context canceled
<7>DEBUG : 20230725.mov: Reopen failed after offset 13971050 bytes read: open file failed: Get "https://www.googleapis.com/drive/v3/files/hash123?alt=media": context canceled
<7>DEBUG : 20230725.mov: multi-thread copy: chunk 29/719 failed: multi-thread copy: failed to write chunk: context canceled
<7>DEBUG : 20230725.mov: multi-thread copy: chunk 34/719 failed: multi-thread copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/hash123?alt=media": context canceled
<7>DEBUG : 20230725.mov: Reopen failed after offset 20926406 bytes read: open file failed: Get "https://www.googleapis.com/drive/v3/files/hash123?alt=media": context canceled
<7>DEBUG : 20230725.mov: multi-thread copy: chunk 33/719 failed: multi-thread copy: failed to write chunk: context canceled
<6>INFO  : Exiting...
<3>ERROR : 20230725.mov: Failed to copy: multi-thread copy: failed to write chunk: context canceled
2

Yes it would be nice but is not possible with rclone today.

It is well known thing on rclone features wish list since 2015:

3

i saw all those discussions and they were either about upload or publicly shared gdrive issues. Assumed it was easier for downloads :slight_smile:

thanks for confirming it also affects downloads from static sources. :sob:

4

I think for downloads it could be easier to do than for uploads - but equally such functionality does not exist today.

5

What you could try is to use rclone mount. VFS cache reads data in chunks and assembles original file in local cache using sparse file.

rclone mount mydrive:drive_dir /path/to/mountpoint --vfs-cache-mode=full --vfs-read-chunk-size-limit 0 --vfs-cache-max-age 9999h

--vfs-read-chunk-size-limit 0 makes all chunks the same size (default 128M)
--vfs-cache-max-age 9999h makes sure that data is valid in cache for longer than default 1h

All together means that if reading file from rclone mount is disrupted then next attempt will only re-download missing chunks.

I have not tested mount behaviour myself for unstable network connection but in theory it should be good substitute of download resume.

1 Like
6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.