Best way to get rclone uploads to continue after resume from hibernate?

Help and Support
1

What is the problem you are having with rclone?

I have a mount going with some long living uploads, but whenever I hibernate my computer and later resume it the network temporarily drops out as Windows renegotiates with the router, the small network drop causes rclone to drop the currently uploading file and start a new one once the network resumes. Was wondering the best way to keep rclone from dropping the file and just have it continually retry or wait until the network becomes sane again

Run the command 'rclone version' and share the full output of the command.

Yes, 1.57.0

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "archiveenc:" X: -vvvvvvvv --vfs-cache-mode full --cache-dir Z:\rclone --vfs-read-chunk-size 200M --vfs-cache-max-age 1s --volname 'The Archive' --attr-timeout 1h --drive-skip-gdocs --drive-skip-shortcuts --dir-cache-time 5h --cache-dir 'Z:\rclone' --rc --rc-web-gui --rc-user=admin --rc-pass=admin --transfers 2 --order-by 'size,ascending'

The rclone config contents with secrets removed.

[archive]
type = drive
client_id = 
client_secret = 
scope = drive
token = 

[archiveenc]
type = crypt
remote = archive:Encrypted
filename_encryption = standard
directory_name_encryption = true
password = 
password2 =

A log from the command with the -vv flag

	couldn't list directory: Get "https://www.googleapis.com/drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks%29%2CnextPageToken%2CincompleteSearch&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271MaosAIJKLBpK2IWEiagxcqq3iJocxjnW%27+in+parents%29+and+%28name%3D%276nvo0dfn0ejmef6b9j703pfmo8%27%29&supportsAllDrives=true": dial tcp: lookup www.googleapis.com: no such host
2

With --vfs-cache-mode full rclone should retry the uploads indefinitely. Is it not?

3

The problem isnt that it errors out the file and doesn't retry it later, the problem is that all progress on the file is lost, which can sometimes amount to a good amount of time if the file is large enough. For instance, if a file is at 64% uploaded and rclone loses internet even for a small second or two, the file will be errored out and restart at 0 the next time it loops through the cache

I absolutely love the tool and use it constantly, this is the only major hurdle I have ever come across

4

hi,

rclone cannot restart failed uploads.

tho rclone can deal with a intermittent internet connections, certainly for "small second or two" and for longer periods.

2022-03-08 09:17:13 DEBUG : HBCD_PE_x64.iso: multipart upload starting chunk 7 size 5Mi offset 30Mi/1.262Gi
2022-03-08 09:17:30 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-east-2.wasabisys.com/zork/HBCD_PE_x64.iso?partNumber=4&uploadId=9Id4shwEbUvgLbOEUTYyK3jtikUjd9lzGIBgDi0dAd5K9ikI4S6EU8fnQjjnIQlm6GTOhDJpbFfDP5zcpHRAHFaiu7sHrZBVG_uAVzrXpPSLWD-arQ8M4tkWthP_s-OO": write tcp 192.168.62.234:58367->38.73.225.20:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022-03-08 09:17:30 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2022-03-08 09:17:31 DEBUG : pacer: low level retry 1/2 (error RequestError: send request failed
caused by: Put "https://s3.us-east-2.wasabisys.com/zork/HBCD_PE_x64.iso?partNumber=5&uploadId=9Id4shwEbUvgLbOEUTYyK3jtikUjd9lzGIBgDi0dAd5K9ikI4S6EU8fnQjjnIQlm6GTOhDJpbFfDP5zcpHRAHFaiu7sHrZBVG_uAVzrXpPSLWD-arQ8M4tkWthP_s-OO": write tcp 192.168.62.234:58364->38.73.225.20:443: wsasend: An existing connection was forcibly closed by the remote host.)
2022-03-08 09:17:31 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2022-03-08 09:17:51 DEBUG : pacer: Reducing sleep to 15ms
2022-03-08 09:17:51 DEBUG : HBCD_PE_x64.iso: multipart upload starting chunk 8 size 5Mi offset 35Mi/1.262Gi
5

Rclone retries temporary networking error messages.

It might be that there is a network error we aren't retrying and we should be.

Can you post the error message when the internet goes away for a moment which kills the transfer?

6

Yeah, though it will have to wait until later tonight when I am back at my computer from work

1 Like
7

Couldn't get the error to happen tonight, its seemingly related to timing though, I'm guessing if something lines up just wrong between Windows bringing the network back up and rclone trying to use it.

If I can get the error to pop up again I'll make sure to post it here. Of course when I try to get something to error to report it it never does :smile:

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.