[WebGUI] File too large when uploading to Scaleway?

Help and Support
1

What is the problem you are having with rclone?

I keep getting an error when uploading a file of 26GB from rclone WebGUI to Scaleway storage. The error is:
rc: "operations/uploadfile": error: multipart upload failed to read source: context canceled
The error also occurs on smaller files, but they seem to get uploaded eventually. This big 26GB file does not. What can I try to get a succesful upload? Thanks in advance for any help!

What is your rclone version (output from rclone version)

1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64 bit

Which cloud storage system are you using? (eg Google Drive)

Scaleway Glacier

The command you were trying to run (eg rclone copy /tmp remote:tmp)

No command since I am using the WebGUI. Just uploading a file.

The rclone config contents with secrets removed.

[scaleway]
type = s3
acl = private
endpoint = s3.nl-ams.scw.cloud
provider = Scaleway
region = nl-ams
secret_access_key = xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
storage_class = GLACIER
access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxxxx
chunk_size = 40M
max_upload_parts = 1000

A log from the command with the -vv flag

I'm not sure how to obtain this log while using the WebGUI? I tried this but the log is extremely sparse:
rclone rcd --rc-web-gui --log-level INFO --log-file=rclonelogfile.txt

2

hello and welcome to the forum,

  • i would use the command line, for easier testing
  • for debug log --log-level DEBUG 
chunk_size = 40M
max_upload_parts = 1000

is there a reason not to use the defaults?

3

Yes there is. Scaleway only allows 1000 upload parts (it's in the FAQ). Since my biggest file is 36GB, I came to a 40M chunk size.

I will try with the DEBUG log level. Guess I might have to figure out the command line as well :slight_smile:

Edit: do you have any idea what the error message might be pointing at?

4

the rclone gui is experimental, so not sure if the problem is with the gui, the command itself or what.
seems rclone is having a problem reading from the source.
if you need a mature gui, try RcloneBrowser | Simple cross platform GUI for rclone. Supports macOS, GNU/Linux, BSD family and Windows.

the debug should show what is going on.

what is the source path?

5

Here is the debug log (still using WebGUI). It's the end of the log, where the upload starts to retry then fails.
The source path is a local directory on my Windows 10 machine.
Can you see anything wrong here?

2021/05/10 21:09:11 DEBUG : Fotos 2020.7z: multipart upload starting chunk 218 size 40M offset 8.477G/off
2021/05/10 21:09:15 DEBUG : Fotos 2020.7z: multipart upload starting chunk 219 size 40M offset 8.516G/off
2021/05/10 21:09:16 DEBUG : Fotos 2020.7z: multipart upload starting chunk 220 size 40M offset 8.555G/off
2021/05/10 21:09:28 DEBUG : Fotos 2020.7z: multipart upload starting chunk 221 size 40M offset 8.594G/off
2021/05/10 21:09:35 DEBUG : Fotos 2020.7z: multipart upload starting chunk 222 size 40M offset 8.633G/off
2021/05/10 21:09:45 DEBUG : pacer: low level retry 1/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:45 DEBUG : Fotos 2020.7z: Cancelling multipart upload
2021/05/10 21:09:45 DEBUG : pacer: low level retry 1/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:45 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2021/05/10 21:09:45 DEBUG : pacer: low level retry 1/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:45 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2021/05/10 21:09:45 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2021/05/10 21:09:45 DEBUG : pacer: low level retry 2/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:45 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2021/05/10 21:09:45 DEBUG : pacer: low level retry 2/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:45 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2021/05/10 21:09:46 DEBUG : pacer: low level retry 2/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:46 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2021/05/10 21:09:46 DEBUG : pacer: low level retry 3/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:46 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2021/05/10 21:09:47 DEBUG : pacer: low level retry 3/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:47 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
2021/05/10 21:09:47 DEBUG : pacer: low level retry 3/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:47 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021/05/10 21:09:48 DEBUG : pacer: low level retry 4/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:50 DEBUG : pacer: low level retry 4/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:53 DEBUG : pacer: low level retry 4/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:55 DEBUG : pacer: low level retry 5/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:57 DEBUG : pacer: low level retry 5/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:09:59 DEBUG : pacer: low level retry 5/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:01 DEBUG : pacer: low level retry 6/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:01 DEBUG : pacer: Reducing sleep to 1.5s
2021/05/10 21:10:01 DEBUG : Fotos 2020.7z: Failed to cancel multipart upload: context canceled
2021/05/10 21:10:01 ERROR : rc: "operations/uploadfile": error: multipart upload failed to read source: context canceled
2021/05/10 21:10:03 DEBUG : pacer: low level retry 6/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:03 DEBUG : pacer: Rate limited, increasing sleep to 2s
2021/05/10 21:10:04 DEBUG : pacer: low level retry 6/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:06 DEBUG : pacer: low level retry 7/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:08 DEBUG : pacer: low level retry 7/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:10 DEBUG : pacer: low level retry 7/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:12 DEBUG : pacer: low level retry 8/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:14 DEBUG : pacer: low level retry 8/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:16 DEBUG : pacer: low level retry 8/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:18 DEBUG : pacer: low level retry 9/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:20 DEBUG : rc: "core/version": with parameters map[]
2021/05/10 21:10:20 DEBUG : rc: "core/version": reply map[arch:amd64 decomposed:[1 55 1] goTags:cmount goVersion:go1.16.3 isBeta:false isGit:false linking:dynamic os:windows version:v1.55.1]: <nil>
2021/05/10 21:10:20 DEBUG : rc: "core/bwlimit": with parameters map[]
2021/05/10 21:10:20 DEBUG : rc: "core/bwlimit": reply map[bytesPerSecond:-1 bytesPerSecondRx:-1 bytesPerSecondTx:-1 rate:off]: <nil>
2021/05/10 21:10:20 DEBUG : rc: "core/stats": with parameters map[]
2021/05/10 21:10:20 DEBUG : rc: "core/stats": reply map[bytes:25753088173 checks:0 deletedDirs:0 deletes:0 elapsedTime:43858.6631988 errors:1 eta:0 fatalError:false lastError:multipart upload failed to read source: context canceled renames:0 retryError:true speed:5.891547980511701e+06 totalBytes:25753088173 totalChecks:0 totalTransfers:1 transferTime:4371.1921312 transfers:1]: <nil>
2021/05/10 21:10:20 DEBUG : rc: "core/stats": with parameters map[]
2021/05/10 21:10:20 DEBUG : rc: "core/stats": reply map[bytes:25753088173 checks:0 deletedDirs:0 deletes:0 elapsedTime:43858.6641982 errors:1 eta:0 fatalError:false lastError:multipart upload failed to read source: context canceled renames:0 retryError:true speed:5.891547980511701e+06 totalBytes:25753088173 totalChecks:0 totalTransfers:1 transferTime:4371.1921312 transfers:1]: <nil>
2021/05/10 21:10:20 DEBUG : pacer: low level retry 9/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:22 DEBUG : pacer: low level retry 9/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:24 DEBUG : pacer: low level retry 10/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:25 DEBUG : rc: "core/stats": with parameters map[]
2021/05/10 21:10:25 DEBUG : rc: "core/stats": reply map[bytes:25753088173 checks:0 deletedDirs:0 deletes:0 elapsedTime:43863.9121174 errors:1 eta:0 fatalError:false lastError:multipart upload failed to read source: context canceled renames:0 retryError:true speed:5.891547980511701e+06 totalBytes:25753088173 totalChecks:0 totalTransfers:1 transferTime:4371.1921312 transfers:1]: <nil>
2021/05/10 21:10:25 DEBUG : rc: "core/stats": with parameters map[]
2021/05/10 21:10:25 DEBUG : rc: "core/stats": reply map[bytes:25753088173 checks:0 deletedDirs:0 deletes:0 elapsedTime:43863.9126322 errors:1 eta:0 fatalError:false lastError:multipart upload failed to read source: context canceled renames:0 retryError:true speed:5.891547980511701e+06 totalBytes:25753088173 totalChecks:0 totalTransfers:1 transferTime:4371.1921312 transfers:1]: <nil>
2021/05/10 21:10:27 DEBUG : pacer: low level retry 10/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:28 DEBUG : pacer: low level retry 10/10 (error RequestCanceled: request context canceled
caused by: context canceled)
2021/05/10 21:10:30 DEBUG : rc: "core/stats": with parameters map[]
2021/05/10 21:10:30 DEBUG : rc: "core/stats": reply map[bytes:25753088173 checks:0 deletedDirs:0 deletes:0 elapsedTime:43868.596109 errors:1 eta:0 fatalError:false lastError:multipart upload failed to read source: context canceled renames:0 retryError:true speed:5.891547980511701e+06 totalBytes:25753088173 totalChecks:0 totalTransfers:1 transferTime:4371.1921312 transfers:1]: <nil>
2021/05/10 21:10:30 DEBUG : rc: "core/stats": with parameters map[]
2021/05/10 21:10:30 DEBUG : rc: "core/stats": reply map[bytes:25753088173 checks:0 deletedDirs:0 deletes:0 elapsedTime:43868.6051038 errors:1 eta:0 fatalError:false lastError:multipart upload failed to read source: context canceled renames:0 retryError:true speed:5.891547980511701e+06 totalBytes:25753088173 totalChecks:0 totalTransfers:1 transferTime:4371.1921312 transfers:1]: <nil>
6

not sure what is going as i do not use rc and the experiential gui.
i use the command line and if i want a gui, i use rclone browser, which is rock stable.
hopefully someone with more experience in that stuff will stop by soon.

until then,
scaleway is throttling rclone, as seen in the pacer messages.
i would increase the chunk size to 256M or higher and/or reduce "--s3-upload-concurrency.

for what it worth, i evaluated scaleway, found them very expensive and not worth the price premium.
and if i was getting throttled like that, i would not use them.

7

Thanks @asdffdsa. I am trying Scaleway because they are the least expensive option I could find. Price of their Glacier storage is €0.002/GB/Month. Downloading is €0.01/GB, but for me it is a last resort backup, so I will only be downloading occassionaly. I have not found any other or hidden costs in their pricing structure. I am very curious about where you found the 'price premium' you mentioned?

Edit: the pacer messages only occur between the retries, after something has gone wrong. Pacing makes sense to me in that context. Too bad there is no clear error message about the reason the upload is failing. I will try other means of uploading, CLI or rclone browser.

8

I think what is happening is that the webgui is timing out the connection hence these messages.

These happen when the API connection is closed - rclone then shuts down the activity that it caused.

Can you try using the command line? I think that probably will work.

9

Coincidentally, the command line upload just finished succesfully a minute ago :partying_face:

Do you think it is worth mentioning in the experimental WebGUI documentation that large uploads should not be attempted?

10

It would probably be worth opening an issue here - put a link to the forum thread in it.

11

Here is the issue:

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.