What is the problem you are having with rclone?
I am trying to run a borg backup script to make a local repository and then use rclone to sync with OneDrive. Once an 'initial' backup has been made manually, this script first fetches the entire repo from OneDrive, adds in a new archive for the day, prunes, uploads it back to OneDrive with the additional archive and then deletes local files. When fetching the repo (of ~1 GB), RCLONE does it almost instantaneously (~20s) like it is local file, I've repeatedly run the script several times and ensured that the destination of the sync is empty but the issue is, possibly after a long duration AND restart later, rsync is unable to fetch the repo instantaneously as it previously did possibly because the cache is getting deleted and takes horribly long as if it is downloading all the files from OneDrive. How do I prevent this from happening? I am not totally sure about what is resetting this as the shift is seemingly random. I have tested after ~8hrs of inactivity (in sleep mode) and was able to fetch the repo in seconds. Reboot and run the script again, was able to fetch the repo in seconds. But a combination of these two might be affecting the speed. I have also referred to
https://rclone.org/cache/
and tried to access
How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache.
Accepted units are: "s", "m", "h".
these settings from
rclone config
but to no avail, so I hastily added
info_age = 48h
at the end of rclone.conf but that didn't really change anything
Side notes:
- Speed of "instantaneous" sync (200-600 MBps )is well beyond my network speed so it is evident that it is happening locally and this is not just a bad network issue (also very less network usage from System Monitor)
- Archiving directly to OneDrive with Borg throws a lot of errors, so I've gone through this process of syncing from local directory as suggested here
https://forum.rclone.org/t/rclone-as-destination-for-borgbackup/12717/4
This is only feasible if I am able to fetch repo in a few seconds.
3. I am able to recreate conditions to fetch instantaneously by deleting both repos (local and drive), initializing a repository and archive locally and uploading to Drive. Deleting local repo and fetching it back from drive is instantaneous. After this point, I can make changes in the backup files and still run the script repeatedly with instant download but this desired outcome is not sustained.
4. First upload (or any upload for that matter) doesn't take too long. First upload taking around 10 min to consequent uploads ranging from near instantaneous to 2-4 minutes
5. Ubuntu and rclone newbie so please be gentle
What is your rclone version (output from rclone version
)
rclone v1.50.2
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 20.04.2 LTS, 64-bit
Which cloud storage system are you using? (eg Google Drive)
OneDrive for business
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync --fast-list --progress "/home/.../OneDrive/Ubuntu Backups/Borg/" /home/.../BorgbackupSync
The rclone config contents with secrets removed.
[one drive]
type = onedrive
token = ***
drive_id = ***
drive_type = business
chunk_size = 10M
info_age = 48h
A log from the command with the -vv
flag
https://pastebin.com/k4UWUPrX
Desired outcome: https://pastebin.com/uFnX4eBA