Transfer files between aws s3 and gdrive

What is the problem you are having with rclone?

Transferring files between aws s3 and gdrive is subject to significant speed limits. aws capacity is 1 gigabyte per second (8gbps) Google's reception capacity is 40 MB per second (320mbps) per service account. when running 25 pokots at 320 Mbps per service account, the total speed is 150 Mbps.
When transferring files from local storage, the speed is 320 Mbps. what could be the problem. how do i speed up transfer between amazon s3 and gdrive as much as possible

Run the command 'rclone version' and share the full output of the command.

rclone v1.56.0

  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.19042.1586 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.16.5
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

aws s3
gdrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move s3:aws6401-1 gdrive: --drive-stop-on-upload-limit --transfers 26 -P --drive-service-account-file "accounts/1.json" -v --log-file rclone1.txt

The rclone config contents with secrets removed.

[gdrive]
type = drive
scope = drive
token = ***
team_drive = ***
root_folder_id = 

[s3]
type = s3
provider = AWS
access_key_id = ***
secret_access_key = ***
region = us-east-2
acl = private


hello,

--- what are pokots?
--- hard to know what is going on, without a rclone debug log.

sorry, maybe i missed it, but what are those speed limits?

https://rclone.org/drive/#limitations
"Drive has quite a lot of rate limiting. This causes rclone to be limited to transferring about 2 files per second only"

pokots - streams, I start the transfer to different service accounts,
each file is transferred using a separate service account to avoid the google drive download speed limit, in my case I see that the speed limit is 150mbps for all downloads together. files are very large and take a long time to transfer

i have no problem transferring from local disk, or downloading to local from s3, the problem occurs when transferring remote to remote (s3 to gdrive)

i thought that you were uploading to gdrive, not downloading?

fyi, rent/use a cheap/free vm from google, and run rclone on that.

and as i mentioned, need to post a debug log, so we can see what rclone is doing.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.