Using rclone for a complete backup of linux machine onto onedrive

[mattasdata]

What is the problem you are having with rclone?

I am attempting to use rclone to perform a backup of the home directory on my Linux machine onto OneDrive. This is a work laptop and it is OneDrive for Business (or Enterprise, whatever they call it now). If it was my own laptop, I would just use an external drive for backups, but this OneDrive space is what the IT people at work want me to use.

The problem is that the transfer of files with rclone sync seems to go on forever. I have left it overnight on a few occasions, and it is still chugging away in the morning. I have looked through a few previous posts on this forum about OneDrive throttling, and although I am getting some throttle warnings, I suspect the real issue is that I am trying to sync far too many really small files and this is always going to struggle even with a high throttle limit..

The solution I used for a long time was to make a tar backup locally on my machine and then just upload that single file. Worked a dream. But now, my machine is running out of space and I can't make the tar backup locally anymore.

Does anyone have a better solution for doing something like this with rclone? I attempted to pipe the output of tar to the rclone sync command accepting stdin, but the same problem arises because rclone creates spools in the tmp file space that eventually also cause my system to run out of hd space.

Run the command 'rclone version' and share the full output of the command.

rclone v1.50.2
- os/arch: linux/amd64
- go version: go1.13.8

Which cloud storage system are you using? (eg Google Drive)

OneDrive as part of O365 (my company's plan)

The command you were trying to run

rclone sync ./ onedrive:<my_username> --exclude-from=configs/rclone_exclude --links --progress --user-agent "ISV|rclone.org|rclone/v1.50.2" --tpslimit=10 -vv

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"<redacted>","token_type":"Bearer","refresh_token":"<redacted>","expiry":"2022-11-28T19:13:45.259887618Z"}
drive_id = <not sure if this needs to be redacted, can provide if helpful>
drive_type = business

A log from the command with the -vv flag

I'll post this when the current command I am running reaches some throttling messages. Currently, it is just doing a bunch of checks of files that have already been synced.

[mattasdata]

One place the command seems to spend a lot of time is in my miniforge3 directory, which is where I have some conda environments. Again, lots and lots of little files there.

[asdffdsa (jojothehumanmonkey)]

hi,
that version of rclone is years old, best to update at
https://rclone.org/downloads/#script-download-and-install

@mattasdata

The problem is that the transfer of files with rclone sync seems to go on forever

--- should use a client_id/secret with onedrive business.
"You may choose to create and use your own Client ID, in case the default one does not work well for you. For example, you might see throttling."

--- can reduce the number of api calls to onedrive.
rclone sync ./ onedrive:<my_username> --max-age=24h
if you run the backup once a day:
for each source file, if the modtime is older than 24 hours, then rclone will not check onedrive.

--- can create a type of forever forward incremental backup

rclone sync ./ remote:full --backup-dir=remote:incrementals/`date +%Y%m%d.%I%M%S`

[ncw (Nick Craig-Wood)]

Onedrive has a horrendous amount of rate limiting.

One thing you could do is use restic along with rclone - this will pack lots of small files together into larger blobs. Its how I backup my laptop.

[mattasdata]

Thanks @ncw and @asdffdsa for the replies.

I will have a go at implementing some of your proposals and report back how that goes.