Slow Upload with too many files

Help and Support
1

What is the problem you are having with rclone?

Hello, I have slow upload with many files but size of folder is less than 2Gb, it takes more than 6 hours, even with 100Mb of Internet. The folder has almost 200,000 files but most of then are small. The problem doesn't occur when fewer files even with more sizes.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: arch "rolling" (64 bit)
  • os/kernel: 5.18.7-zen1-1-zen (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy -P  --transfers=30 --stats-one-line --fast-list --check-first --copy-links /mnt/folder GDrive:/folder

The rclone config contents with secrets removed.

1 [GDrive].                                                                                                                                                                               
    2 type = drive.
    3 client_id =
    4 client_secret =
    5 scope = drive.
    6 token =
2

Google Drive limits file creation to roughly 2-3 per second as there is no way around that as it is a limitation of the API.

1 Like
3

I see, thank you.

4

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.