Hard drive barely accessible when rclone sync is running

Help and Support
1

What is the problem you are having with rclone?

When running rclone sync opening any directory on my hard drive becomes super slow. Some of them with lots of pictures or videos (> 1000) won't even open it just keeps loading in nautilus. Is there any way to throttle rclone sync? Other than that rclone is doing its job well I can't complain about any real bug.

What is your rclone version (output from rclone version)

rclone v1.54.0-beta.4793.2b7994e73
- os/arch: linux/amd64
- go version: go1.15.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Pop!_OS 20.4 64 bits

Which cloud storage system are you using? (eg Google Drive)

Microsoft Onedrive Personal

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone runs in systemd with this service file (main_hdd is a 6TB mounted NTFS drive):

[Unit]
Description=rclone onedrive local -> onedrive remote
 
[Service]
ExecStart=sh -c "rclone sync /mnt/main_hdd/Onedrive onedrive:/ --progress --exclude-from /mnt/main_hdd/Onedrive/exclude.txt"

[Install]
WantedBy=default.target

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"","token_type":"Bearer","refresh_token":"","expiry":"2020-10-06T15:01:34.403731271+02:00"}
drive_id = 123
drive_type = personal
2

hello and welcome to the forum,

you can try
https://rclone.org/docs/#bwlimit-bandwidth-spec

3

Rclone can be very aggressive. I'm guessing you have an HDD rather than SSD.

To slow rclone down --bwlimit is good but only works when transferring data. Reducing --checkers maybe to 1 will help with the directory traversals as will reducing --transfers. You can also add

  --check-first   Do all the checks before starting transfers.

Which I think helps with HDDs so you do all the checking first before starting any transfers.

4

Thanks a lot for all your suggestions :slight_smile:. I had no idea which global flag could help. This really improves my experience of switching to Linux! While the --bwlimit argument did not seem to change much the other arguments do seem to fix the issue :slight_smile:!

I am not sure whether it is due to system overload (well the computer is a bit old but the CPU is at 30% under normal use) or if it is the HDD or the NTFS file system :thinking:.

Also I suppose that --progress impacts the performance a tiny bit negatively (with --vv it seemed to be worse) but I wanted to have some sort of feedback as I don't like a "blind" process.

So I ended up with this command: rclone sync /mnt/main_hdd/Onedrive onedrive:/ --progress --exclude-from /mnt/main_hdd/Onedrive/exclude.txt --check-first --checkers 4 --transfers 1. I kept 4 checkers as I realized that this does not impact performance too much and with just 1 it takes ages (there are still lots of changes to process).

5

Great - glad you got it working.

I suspect that the --check-first flag is doing most of the work here - it might be worth increasing --transfers and see if you can improve the throughput.

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.