Optimized for low memory/high bandwidth

I run on a low memory (1000mb) machine, but it has a 1GiB connection. I’m always maxed out on memory.

Currently my setup is this:
rclone copy -v --bwlimit 8M --checkers 2 --transfers 2 --buffer-size 0 --http-url http://old-releases.ubuntu.com/releases/ :http: googledrive:ubuntu.com

I’m wondering if I could have a better balance of speed and memory to optimize this transfer. My 8M bwlimit is for staying under GDrive’s 750GB/day limit (and allow me to upload other stuff on my own apart from the script).
I only set the buffer size to 0 and checkers/transfers down to 2 from 3 after reading another post earlier this morning.

EDIT: Since making those changes by dropping checkers and transfers down to 2 and buffer size to 0, I’m now using 162MB of memory instead of maxing out to 1GB. I’m concerned that my speed may be reduced though.

Now I’m in search of the perfect balance. Can anyone recommend better settings?

Also, I’m mirroring another repository that has some large files in it (double digit GB). How will this setup handle those?

Thanks,
Matt

That does seem like a bit of memory to use with a 0 buffer size.

You can try adding in

--use-mmap

and see if that comes down a bit.

Assuming you are on 1.46 or higher / hopefully latest 1.47.

The goal is to create another mirror copy of that basically?

Maybe he is r/DataHoarder/
He can consider using --bwlimit 2M, to avoid overload the server of canonical

@Animosity022, thank you, I’ll try that.

Actually, I’m still getting 100% memory usage. Not all is from rclone, but a lot is, from what I can tell.

When I run htop, I see a lot of rclone instances running. Any idea why?

I have a screenshot below. I suspect something is going wrong with the way I’m running the script. I’m simply using crontab -e to call /root/copy.sh, which contains a single line of rclone commands per server.

The only way to get more than 1 copy going is to run it multiple times as that’s nothing that rclone itself does.

I’m guessing your script is calling it multiple times or not checking if a previous one is done.

Hm, I’m not sure how to adjust the script. It’s nothing more than a simple shell script. Looks like this inside:

#!/bin/bash

/usr/bin/rclone copy -v --bwlimit 8M --checkers 2 --transfers 2 --buffer-size 0 --http-url http://download.kiwix.org :http: GDrive:kiwix.org
/usr/bin/rclone copy -v --bwlimit 8M --checkers 2 --transfers 2 --buffer-size 0 --http-url http://old-releases.ubuntu.com :http: GDrive:ubuntu.com

Then in crontab -e
@reboot sleep 60 && /root/copy.sh

Maybe check your cron log or did you run some other way as it looks like it ran 4 times.

After making a few changes, I think the actual issue is instrumentation- a false alarm. htop shows those multiple instances, but top only shows one.

I’ve also switched away from using crontab to call the script. Instead, I’m using tmux and launching each shell script in it’s own tmux session, so that I can connect to each one individually and watch the current progress with rclone -P. This is a much simpler solution, and my memory usage has dropped significantly by changing --checkers 2 --transfers 2 to --checkers 1 --transfers 1 and by adding the --use-mmap flag. Thank you.

Yes, it’s this flag here. Sorry as I really thought I remember that showed the PIDs differently but that is the case. If you hide userland process threads, you get similar top output as top by default does not show threads.

image

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.