Rclone sync on OneDrive causes local leak of files

Hi! I am new to Rclone, and found it as a way to sync with a remote MS OneDrive folder. It works wonders, except for files leaked into the local OneDrive folder similarly randomly, which I'm not able to track down. I'd welcome any suggestion to debug this, I don't even know where to start.

What is the problem you are having with rclone?

  • I basically followed the instructions here to setup my OneDrive account. It works well, I can mount the OneDrive remote account into /home/mathieu/Public/OneDrive/.

  • My rclone script looks like:

    rclone sync /home/mathieu onedrive:backup_home/
    --exclude "/Music/"
    --exclude "/Pictures/
    --exclude "/Public/"
    --exclude "/Downloads/temp
    --exclude "/Downloads/tmp**"

(I have removed a bunch of --exclude of the same form, see full script below)

  • The sync starts well, files are copied on the remote OneDrive the way I expect, and I can check on the OneDrive account that it works. The sync should be around 110–120 GB.
  • However, every now and then, rclone will start copying many (all?) files locally in /home/mathieu/Public/OneDrive/ until my hard drive is full (for instance, I currently have about 70 GB free on a 200 GB HD). I then need to delete the duplicated files (with sudo rm -r /home/mathieu/Public/OneDrive/home/ because some of the copied content is owned by root).
  • Oddly, the script goes on as if nothing happened, and keep copying to the remote.

What is your rclone version (output from rclone version)

rclone v1.50.2
- os/arch: linux/amd64
- go version: go1.13.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian GNU/Linux 9.5 (stretch) [64 bit].

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /home/mathieu onedrive:backup_home/ \
  --exclude "/Music/**" \
  --exclude "/Pictures/**" \
  --exclude "/Public/**" \
  --exclude "/Downloads/temp**" \
  --exclude "/Downloads/tmp**" \
  --exclude "/Downloads/torrents/**" \
  --exclude "/Videos/**" \
  --exclude "/.emacs.d/cache/**" \
  --exclude "/.emacs.d/elpa/**" \
  --exclude "/.R-site/site-library/**" \
  --exclude "/.cache/**" \
  --exclude "/.gftp/cache/**" \
  --exclude "/.googleearth/Cache/**" \
  --exclude "/.config/chromium/Default/Service Worker/CacheStorage/**" \
  --exclude "/.config/chromium/Default/Service Worker/ScriptCache/**" \
  --exclude "/.config/google-chrome/Default/Service Worker/CacheStorage/**" \
  --exclude "/.config/google-chrome/Default/Service Worker/ScriptCache/**" \
  --exclude "/.config/Microsoft/Microsoft Teams/Application Cache/**" \
  --exclude "/.config/Microsoft/Microsoft Teams/Cache/**" \
  --exclude "/.config/Slack/Cache/**" \
  --exclude "/.config/Slack/Service Worker/CacheStorage/**" \
  --exclude "/.config/skypeforlinux/Cache/**" \
  --exclude "/.thumbnails/**" \
  --exclude "/.local/share/torbrowser/**" \
  --exclude "/.local/share/Trash/**" \
  --exclude "/.icedove/82cgbxbx.default/ImapMail/**" \
  --progress \
  --verbose \
  --links \

The rclone config contents with secrets removed.

type = onedrive
token = {"access_token":"eyJ0eXAiOiJKV<snip>","token_type":"Bearer","refresh_token":"OAQABAAAAAAAm<snip>","expiry":"2020-05-19T13:26:42.415148467+02:00"}
drive_id = b!hrraSE<snip>
drive_type = business

A log from the command with the -vv flag

Since the script keeps working fine, I have what seems like a normal log… but I'd be happy to report specific pieces if instructed to.

To see why a file is being copied, you'd have to post a debug log so you'd want to create the problem and share the log so we can help identify why it was being copied.

OK, I'm working on a log file now… will post it next time the problem arises (sync in progress).


you should update the rclone version.

curl https://rclone.org/install.sh | sudo bash

I just updated to 1.51.0 (amd64), and am running the script again. I will post here the results…

"mount the OneDrive remote account into /home/mathieu/Public/OneDrive/."
can you post that rclone mount command?
is there a reason you are using a mount instead of a simple rclone sync /path/to/local/folder onedrive:backup_home/

your command is sync /home/mathieu onedrive:backup_home/

if i understand what you are doing
you are mounting onedrive to a local folder and then syncing to onedrive, which is mounted as local storage, kind of a infinite loop

Ah, sorry I was not clear on that! I do not mount the remote to sync — I just meant that I am able to successfully mount the remote (using rclone mount) when I need it, as a proof that the OneDrive config works.

However, when the file duplication happens, sometimes after the beginning of the sync (with rclone sync), it does happen in the mount point.

(still checking if that happens with v1.51 as I write)

so the mount is not active at the same time the rclone sync is running?

No, it is not. The rclone sync command is the only thing active from the rclone side of things. As a matter of fact, the /home/mathieu/Public/OneDrive/folder (the mount folder) is completely empty when I run the sync script. It only starts to fill up when files are leaking locally (this issue), sometimes hours after the beginning of the sync.

I have now a log file when this happened (with v1.51)… The entire file is ~60 MB, 400,000 lines. Any suggestion to share this? (the file leak occurred towards the end, but I don't have an exact time…)

You could make a OneDrive link :slight_smile:

No, the whole Public folder is excluded from the sync script.

LOL, I did not think of using OneDrive to actually share files… It's uploading ATM, I'll post the link if I can before the end of the day.

rclone sync /home/mathieu onedrive:backup_home/
there is no way that a one-way sync from local to cloud would cause new files to appear in the local.
so either this is a major rclone bug or something else.

at one point, you did run a rclone mount pointing inside /home
perhaps there is a rclone mount running in the background.
and that rclone mount is pointed to a local folder under /home
that would explain how files in the cloud somehow end up in a local folder at the same time a rclone sync is running.

i am not a linux expert, but i would try
ps -ef | grep rclone

mount | grep rclone

fusermount -uz /home/mathieu/Public/OneDrive

I see what you mean, so let's try… Here's the output from your commands:

ps -ef | grep rclone
mathieu   3334  3301  1 10:47 pts/0    00:05:40 rclone sync /home/mathieu onedrive:backup_home/ --exclude /Music/** --exclude /Pictures/** --exclude /Public/** --exclude /Downloads/temp** --exclude /Downloads/tmp** --exclude /Downloads/torrents/** --exclude /Videos/** --exclude /.emacs.d/cache/** --exclude /.emacs.d/elpa/** --exclude /.R-site/site-library/** --exclude /.cache/** --exclude /.gftp/cache/** --exclude /.googleearth/Cache/** --exclude /.config/chromium/Default/Service Worker/CacheStorage/** --exclude /.config/chromium/Default/Service Worker/ScriptCache/** --exclude /.config/google-chrome/Default/Service Worker/CacheStorage/** --exclude /.config/google-chrome/Default/Service Worker/ScriptCache/** --exclude /.config/Microsoft/Microsoft Teams/Application Cache/** --exclude /.config/Microsoft/Microsoft Teams/Cache/** --exclude /.config/Slack/Cache/** --exclude /.config/Slack/Service Worker/CacheStorage/** --exclude /.config/skypeforlinux/Cache/** --exclude /.thumbnails/** --exclude /.local/share/torbrowser/** --exclude /.local/share/Trash/** --exclude /.icedove/** --links --delete-during --progress -vv --log-file=/home/mathieu/Downloads/rclone.log
mathieu  15378 15342  0 18:45 pts/2    00:00:00 grep rclone

mount | grep rclone


fusermount -uz /home/mathieu/Public/OneDrive
fusermount: entry for /home/mathieu/Public/OneDrive not found in /etc/mtab

Does not look like there's much happening beside the sync.

So it turns out I cannot post a link to this thread (I guess because I'm new to the forum). :roll_eyes:

Thanks all for the support — I'm now leaving for the weekend with no Internet connection…

Ive upgraded your account. For new users to protect from spam we only allow whitelisted domains. what domain was denied?

Thanks for your help! This is a OneDrive link on sharepoint.com.

Here is the link to the huge log file:


Note that I stopped the script after I noticed the leak, so it stops abruptly before the end. The leak happened minutes before that (really not sure how much though).

strange that sometimes rclone can find the config file and other times not?
each time you run the rclone command, are you always running it from the same folder?
if rclone cannot find the config file, rclone might consider onedrive:backup_home/ to be a local folder, not a remote.

2020/05/21 20:15:42 DEBUG : Using config file from "/home/mathieu/.config/rclone/rclone.conf"

2020/05/21 23:35:07 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2020/05/21 23:35:07 Failed to create file system for "onedrive:backup_home/": didn't find section in config file

If you check the paths, he runs it as his user sometimes and as root some other times.

yeah, now i see what you mean about root.

but what i wrote is correct?

if rclone cannot find the config file,
then rclone might consider onedrive:backup_home" to be a local folder.