Rclone sync on OneDrive causes local leak of files

basille · May 20, 2020, 9:38am

Hi! I am new to Rclone, and found it as a way to sync with a remote MS OneDrive folder. It works wonders, except for files leaked into the local OneDrive folder similarly randomly, which I'm not able to track down. I'd welcome any suggestion to debug this, I don't even know where to start.

What is the problem you are having with rclone?

I basically followed the instructions here to setup my OneDrive account. It works well, I can mount the OneDrive remote account into /home/mathieu/Public/OneDrive/.
My rclone script looks like:

rclone sync /home/mathieu onedrive:backup_home/
--exclude "/Music/"
--exclude "/Pictures/"
--exclude "/Public/"
--exclude "/Downloads/temp"
--exclude "/Downloads/tmp**"
--progress
--verbose
--links
--delete-during

(I have removed a bunch of --exclude of the same form, see full script below)

The sync starts well, files are copied on the remote OneDrive the way I expect, and I can check on the OneDrive account that it works. The sync should be around 110–120 GB.
However, every now and then, rclone will start copying many (all?) files locally in /home/mathieu/Public/OneDrive/ until my hard drive is full (for instance, I currently have about 70 GB free on a 200 GB HD). I then need to delete the duplicated files (with sudo rm -r /home/mathieu/Public/OneDrive/home/ because some of the copied content is owned by root).
Oddly, the script goes on as if nothing happened, and keep copying to the remote.

What is your rclone version (output from `rclone version`)

rclone v1.50.2
- os/arch: linux/amd64
- go version: go1.13.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian GNU/Linux 9.5 (stretch) [64 bit].

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /home/mathieu onedrive:backup_home/ \
  --exclude "/Music/**" \
  --exclude "/Pictures/**" \
  --exclude "/Public/**" \
  --exclude "/Downloads/temp**" \
  --exclude "/Downloads/tmp**" \
  --exclude "/Downloads/torrents/**" \
  --exclude "/Videos/**" \
  --exclude "/.emacs.d/cache/**" \
  --exclude "/.emacs.d/elpa/**" \
  --exclude "/.R-site/site-library/**" \
  --exclude "/.cache/**" \
  --exclude "/.gftp/cache/**" \
  --exclude "/.googleearth/Cache/**" \
  --exclude "/.config/chromium/Default/Service Worker/CacheStorage/**" \
  --exclude "/.config/chromium/Default/Service Worker/ScriptCache/**" \
  --exclude "/.config/google-chrome/Default/Service Worker/CacheStorage/**" \
  --exclude "/.config/google-chrome/Default/Service Worker/ScriptCache/**" \
  --exclude "/.config/Microsoft/Microsoft Teams/Application Cache/**" \
  --exclude "/.config/Microsoft/Microsoft Teams/Cache/**" \
  --exclude "/.config/Slack/Cache/**" \
  --exclude "/.config/Slack/Service Worker/CacheStorage/**" \
  --exclude "/.config/skypeforlinux/Cache/**" \
  --exclude "/.thumbnails/**" \
  --exclude "/.local/share/torbrowser/**" \
  --exclude "/.local/share/Trash/**" \
  --exclude "/.icedove/82cgbxbx.default/ImapMail/**" \
  --progress \
  --verbose \
  --links \
  --delete-during

The rclone config contents with secrets removed.

[onedrive]
type = onedrive
token = {"access_token":"eyJ0eXAiOiJKV<snip>","token_type":"Bearer","refresh_token":"OAQABAAAAAAAm<snip>","expiry":"2020-05-19T13:26:42.415148467+02:00"}
drive_id = b!hrraSE<snip>
drive_type = business

A log from the command with the `-vv` flag

Since the script keeps working fine, I have what seems like a normal log… but I'd be happy to report specific pieces if instructed to.

Animosity022 · May 20, 2020, 11:03am

To see why a file is being copied, you'd have to post a debug log so you'd want to create the problem and share the log so we can help identify why it was being copied.

basille · May 20, 2020, 12:50pm

OK, I'm working on a log file now… will post it next time the problem arises (sync in progress).

asdffdsa · May 20, 2020, 12:59pm

hello,

you should update the rclone version.

curl https://rclone.org/install.sh | sudo bash

basille · May 20, 2020, 1:48pm

I just updated to 1.51.0 (amd64), and am running the script again. I will post here the results…

asdffdsa · May 20, 2020, 2:08pm

"mount the OneDrive remote account into /home/mathieu/Public/OneDrive/."
can you post that rclone mount command?
is there a reason you are using a mount instead of a simple rclone sync /path/to/local/folder onedrive:backup_home/

your command is sync /home/mathieu onedrive:backup_home/

if i understand what you are doing
you are mounting onedrive to a local folder and then syncing to onedrive, which is mounted as local storage, kind of a infinite loop

basille · May 20, 2020, 4:00pm

Ah, sorry I was not clear on that! I do not mount the remote to sync — I just meant that I am able to successfully mount the remote (using rclone mount) when I need it, as a proof that the OneDrive config works.

However, when the file duplication happens, sometimes after the beginning of the sync (with rclone sync), it does happen in the mount point.

(still checking if that happens with v1.51 as I write)

asdffdsa · May 20, 2020, 4:04pm

so the mount is not active at the same time the rclone sync is running?

basille · May 22, 2020, 3:32pm

No, it is not. The rclone sync command is the only thing active from the rclone side of things. As a matter of fact, the /home/mathieu/Public/OneDrive/folder (the mount folder) is completely empty when I run the sync script. It only starts to fill up when files are leaking locally (this issue), sometimes hours after the beginning of the sync.

I have now a log file when this happened (with v1.51)… The entire file is ~60 MB, 400,000 lines. Any suggestion to share this? (the file leak occurred towards the end, but I don't have an exact time…)

Animosity022 · May 22, 2020, 3:33pm

You could make a OneDrive link

basille · May 22, 2020, 3:48pm

No, the whole Public folder is excluded from the sync script.

basille · May 22, 2020, 3:49pm

LOL, I did not think of using OneDrive to actually share files… It's uploading ATM, I'll post the link if I can before the end of the day.

asdffdsa · May 22, 2020, 3:53pm

rclone sync /home/mathieu onedrive:backup_home/
there is no way that a one-way sync from local to cloud would cause new files to appear in the local.
so either this is a major rclone bug or something else.

at one point, you did run a rclone mount pointing inside /home
perhaps there is a rclone mount running in the background.
and that rclone mount is pointed to a local folder under /home
that would explain how files in the cloud somehow end up in a local folder at the same time a rclone sync is running.

i am not a linux expert, but i would try
ps -ef | grep rclone

mount | grep rclone

fusermount -uz /home/mathieu/Public/OneDrive

basille · May 22, 2020, 4:48pm

I see what you mean, so let's try… Here's the output from your commands:

ps -ef | grep rclone
mathieu   3334  3301  1 10:47 pts/0    00:05:40 rclone sync /home/mathieu onedrive:backup_home/ --exclude /Music/** --exclude /Pictures/** --exclude /Public/** --exclude /Downloads/temp** --exclude /Downloads/tmp** --exclude /Downloads/torrents/** --exclude /Videos/** --exclude /.emacs.d/cache/** --exclude /.emacs.d/elpa/** --exclude /.R-site/site-library/** --exclude /.cache/** --exclude /.gftp/cache/** --exclude /.googleearth/Cache/** --exclude /.config/chromium/Default/Service Worker/CacheStorage/** --exclude /.config/chromium/Default/Service Worker/ScriptCache/** --exclude /.config/google-chrome/Default/Service Worker/CacheStorage/** --exclude /.config/google-chrome/Default/Service Worker/ScriptCache/** --exclude /.config/Microsoft/Microsoft Teams/Application Cache/** --exclude /.config/Microsoft/Microsoft Teams/Cache/** --exclude /.config/Slack/Cache/** --exclude /.config/Slack/Service Worker/CacheStorage/** --exclude /.config/skypeforlinux/Cache/** --exclude /.thumbnails/** --exclude /.local/share/torbrowser/** --exclude /.local/share/Trash/** --exclude /.icedove/** --links --delete-during --progress -vv --log-file=/home/mathieu/Downloads/rclone.log
mathieu  15378 15342  0 18:45 pts/2    00:00:00 grep rclone

mount | grep rclone

(empty)

fusermount -uz /home/mathieu/Public/OneDrive
fusermount: entry for /home/mathieu/Public/OneDrive not found in /etc/mtab

Does not look like there's much happening beside the sync.

basille · May 22, 2020, 4:54pm

So it turns out I cannot post a link to this thread (I guess because I'm new to the forum).

Thanks all for the support — I'm now leaving for the weekend with no Internet connection…

calisro · May 22, 2020, 4:56pm

Ive upgraded your account. For new users to protect from spam we only allow whitelisted domains. what domain was denied?

basille · May 22, 2020, 5:23pm

Thanks for your help! This is a OneDrive link on sharepoint.com.

Here is the link to the huge log file:

https://uflorida-my.sharepoint.com/:u:/g/personal/basille_ufl_edu/EdO9g303pqBEtMdbwq_tVmUBo-j1W41EMgBado5lATzQ5A?e=1uUeZp

Note that I stopped the script after I noticed the leak, so it stops abruptly before the end. The leak happened minutes before that (really not sure how much though).

asdffdsa · May 22, 2020, 5:37pm

strange that sometimes rclone can find the config file and other times not?
each time you run the rclone command, are you always running it from the same folder?
if rclone cannot find the config file, rclone might consider onedrive:backup_home/ to be a local folder, not a remote.

2020/05/21 20:15:42 DEBUG : Using config file from "/home/mathieu/.config/rclone/rclone.conf"

2020/05/21 23:35:07 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2020/05/21 23:35:07 Failed to create file system for "onedrive:backup_home/": didn't find section in config file

Animosity022 · May 22, 2020, 6:23pm

If you check the paths, he runs it as his user sometimes and as root some other times.

asdffdsa · May 22, 2020, 6:29pm

yeah, now i see what you mean about root.

but what i wrote is correct?

if rclone cannot find the config file,
then rclone might consider onedrive:backup_home" to be a local folder.