Improving speed with many files and complicated folder structure

Help and Support
1

What is the problem you are having with rclone?

Hi, fairly new to rclone. In the past when uploading/downloading folders with no folder structure below it and a handful of large files has been stupendously fast. What I'm looking for is help to optimize transfer speeds when working in a complicated folder structure with tons of small files.

In the first example i'll have 10gb uploads/downloads take 10 minutes with full speed. In the 2nd example, similar total file size takes up to 3 hours to complete. Getting fluctuating speeds down to 1.5mbps generally.

Ideally I'm just looking for some generic flags/commands to help transfer speeds when working with complex folder structures and many files.

What is your rclone version (output from rclone version)

rclone version v1.53.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

primarily OneDrive.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync C:\*filepath*\Uploaded_Folder RemoteName:*filepath*\Uploaded_Folder -P

The rclone config contents with secrets removed.

[remotename]
type = onedrive
token = {"access_token":"Tokenremoved","expiry":"Dateremoved"}
drive_id = Driveidremoved
drive_type = business

A log from the command with the -vv flag

I don't have a full log file prepared, as it would take roughly 3 hours to have a complete round to look at. I'll generate one to update later with an edit.
2

hi,

in the second example, how many files and folders are there, what is the average file size

you can tweak this flags, which apply to all remotes
--transfers
--checkers
--fast-list

if you use a debug log, you can see if there are any errors, pacing or throttling.

if you search the forum you will find many posts about onedrive being slow.

3

Thanks for the reply!

Okay I'll take a look at those flags and search a little more deeply on onedrive posts on the forum.

In this example, file size ranges from 1 to 100 MB, but generally it looks like around 1-10 for most. with about 10,000 files in 1500 folders.

4

Increasing your concurrent transfers using the --transfers flag really improves upload speed when uploading very small files. OneDrive might also have a speed limiter that could be the bottleneck.

5

another option is to use filters on the source.
the less files that need to be transferred, the less checks on the dest.

for example, i run a sync command every day on a schedule.
i use a https://rclone.org/filtering/#max-age-don-t-transfer-any-file-older-than-this.
rclone sync source: dest: --max-age=25h
i use 25, not 24 on purpose,

then once a week, i run the sync without using max-age, to make sure the dest matches the source.

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.