What is the problem you are having with rclone?
Transferring files between aws s3 and gdrive is subject to significant speed limits. aws capacity is 1 gigabyte per second (8gbps) Google's reception capacity is 40 MB per second (320mbps) per service account. when running 25 pokots at 320 Mbps per service account, the total speed is 150 Mbps.
When transferring files from local storage, the speed is 320 Mbps. what could be the problem. how do i speed up transfer between amazon s3 and gdrive as much as possible
Run the command 'rclone version' and share the full output of the command.
rclone v1.56.0
- os/version: Microsoft Windows 10 Pro 2009 (64 bit)
- os/kernel: 10.0.19042.1586 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.16.5
- go/linking: dynamic
- go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
aws s3
gdrive
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone move s3:aws6401-1 gdrive: --drive-stop-on-upload-limit --transfers 26 -P --drive-service-account-file "accounts/1.json" -v --log-file rclone1.txt
The rclone config contents with secrets removed.
[gdrive]
type = drive
scope = drive
token = ***
team_drive = ***
root_folder_id =
[s3]
type = s3
provider = AWS
access_key_id = ***
secret_access_key = ***
region = us-east-2
acl = private