Looking for advice on copying files from local server to ACD

Help and Support
1

Hello,

I recently just signed up for ACD & started to use rclone and it’s been good in a few basic tests I ran. I’m currently looking for guidance for my particular scenario, as what would be the best approach to transfer data from my in-home NAS server to ACD.

I have rclone installed on my windows desktop that has a network shared drive to the NAS server where my data is stored. The problem is I have to transfer close to 1.1 TB of data from my server to ACD, but having bad upload speeds this is going to take me awhile. It takes me about an hour to transfer 5.9GB roughly.

The biggest problem right now is when I upload anything using rsync, it takes up all my network bandwidth. Essentially when uploading, I can’t even browser the web as all the bandwidth is eaten up by rclone. I guess this boils down to:

  • Figuring out how can I use rclone copy to transfer files without impacting the rest of my network (perhaps lowering the upload speed if possible)?
  • Figuring out how best I can automate this potentially (on windows). This relies on figuring out the first bullet, as I can’t just dedicate hours or days just to wait for rclone to finish uploading.

Appreciate any thoughts or guidance someone can provide to make my situation a bit easier.

2

Rclone has a built in bandwidth limit option. From the documentation:

–bwlimit BwTimetable
Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.

You can use speedtest.net to get a better gauge of your max upload speed. And remeber rclone takes the speed limit in BYTES not BITS.

Edit: For reference- speedtest.net shows my upload speed as 12mbps. That is roughly 1.5MB/s. To be safe I use the following rclone command:

rclone sync /path/to/source remote:path/to/destination --bwlimit 1M
3

Thanks! I actually have about the same upload speed as you and I tried a quick test with a file and it looks good. I’ll copy a few more files in parallel when I can later today but as a follow up, do you have any issues in doing anything else in parallel that might be bandwidth intensive while uploading?

The quick test I did allowed me to limit the speed so I could browse the internet while the upload was progressing but I was thinking along the lines of if I were to play a video game + uploading at the same time. I guess at this point is finding the correct balance that would allow me to do both things in parallel…

4

As long as your uplink isn’t congested there shouldn’t be any issues with latency. Which means you can play games.

If you have such a shitty connection i’d recommend that you build a bash script that runs rclone over and over and over (infinite loop) for you with 750k/1M upload speed, this way you keep everything backed 24/7.

What brand is your NAS? Most run Linux, if you can SSH into the box you can build bash scripts.

while : do
    rclone sync /path/to/source remote:path/to/destination --bwlimit 1M
    sleep 1
done

add it to “startup” in cron (@reboot /path/to/script/ &)

TL;DR:
Build a script to run rclone on the NAS rather then on your Windows machine.

GL =)

5

I’m not sure if I would really want to do an infinite loop but I would definitely consider doing it in a cron job like fashion of some sorts in hourly intervals. I think first I have to upload majority of the stuff using copy before sync so I’m comfortable im not messing anything up.

The NAS server runs a version of FreeNAS. I haven’t done much to it besides setting it up initially along with a Plex Jail. I’ll look into it some more. My content is broken up into 3 main folders, each worth several hundred GB’s, so I don’t want to just copy a folder at a time since uploading several hundred GB will take me days…