What happens if the connection is interrupted while uploading a file to remote?

What is the problem you are having with rclone?

It is not a problem, just curiosity. I am downloading some big files using JDownloader2. With "--vfs-cache-mode writes" enabled in my mount I'm able to write to cache folder and to upload to my rclone remote immediately when a file is completely downloaded into cache.
Then, after file is completely uploaded to a remote, rclone removes the file from cache (configured --vfs-cache-max-age to 0h0m0s to do this).

Well, my ISP sometimes reboots my router to clean cache, apply new firmware updates or simply fix random problems. What happens if a file is being uploaded to the remote and during the upload the router reboots?

The file seems to upload successfully!

Why? does rclone have an infinite number of attempts to upload a file to the remote? If not, how many times does the rclone try? Sometimes my router takes four minutes to reboot...
Does rclone stop the upload and then, after recovering the connection, restart the upload to the remote? rclone knows the file is being used even when the router is rebooting?

Last question: How does rclone know if that a JDownloader2 file is completely downloaded into cache in order to start the upload to the remote?

It is simply awesome!!

Thank you to all of you for your incredible work, and for sharing your talent with all of us.

What is your rclone version (output from rclone version)

v1.53.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

QNAP NAS (QTS/Linux)

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag


INFO  : Google drive root 'myremote/d7567ljafm514ho0qm5tlf76ke0o': Change notify listener failure: Get "https://www.googleapis.com/drive/v3/changes?alt=json&fields=nextPageToken%2CnewStartPageToken%2Cchanges%28fileId%2Cfile%28name%2Cparents%2CmimeType%29%29&includeItemsFromAllDrives=true&pageSize=1000&pageToken=4458598&prettyPrint=false&supportsAllDrives=true": dial tcp 142.250.184.170:443: connect: network is unreachable
ERROR : BigFile1.zip.part: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ABg5-UxPVQGXF188Hv9oIw7EaB1XWOdraRQ0pIdY9xnk280SuyiMey7t8YIpxTBCGnqcY-v6xBEVL3v67t2ulgSIRgU8": context canceled
INFO  : BigFile1.zip.part: vfs cache: upload canceled
ERROR : BigFile2.zip.part: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ABg5-Uyuh4ByRkFOzHPk18U0GRxFNLLA4x9HaWbGD79K0BGOCJcR61X6v_6btHDNLogJY0JW1aRujVh4cxNE1nWcGvY": context canceled
INFO  : BigFile2.zip.part: vfs cache: upload canceled

Depends on how long the connection is down. Rclone retries for a bit and if it fails enough, the transfer fails.

There are a few spots that rclone reties and it depends.

      --low-level-retries int                Number of low level retries to do. (default 10)

and

    --retries int                          Retry operations this many times if they fail (default 3)
      --retries-sleep duration               Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)

See earlier comment as it depends on how long it's down.

Rclone like me has no idea what JDownloader is. If the file is changing, rclone will notice that and skip the file being uploaded.

Best thing to do is run what you want, make a log and rclone does a great job as showing what happens.

Thank you very much. So I assume if rclone is uploading a file and misses the connection with the remote , then tries 10 times (default) and if fails to connect, then rclone removes the file chunk hosted on the remote? or leaves the incomplete files at the remote?

Anyway and luckily for me, seems like it's working despite the router reboots and the files one way or another are uploading correctly. It's so interesting.

Love rclone :heart_eyes:

JDownloader is a download management tool like MiPony.

Currently, there isn't any resume uploads so there is nothing on the remote side.

No idea what MiPony is either :slight_smile:

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.