Rclone kept dying

suchan · November 27, 2024, 10:35am

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

My rclone union remote frequently dies and I can't seem to get it to run again without restarting. Yet after I restarted, after a few minutes, it died again.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

os/version: darwin 13.6.6 (64 bit)
os/kernel: 22.6.0 (arm64)
os/type: darwin
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.1
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

/usr/local/bin/rclone mount UnionOneDrive: /Users/suchan/SecureDrive --vfs-cache-mode=full --vfs-cache-max-size=10GiB --vfs-refresh --volname="OneDriveUnion" -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[UnionOneDrive]
type = union
upstreams = onedrive1:od1 onedrive2:od2 onedrive3:od3 onedrive4:od4 onedrive5:od5
create_policy = mfs

[onedrive1]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal

[onedrive2]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal

[onedrive3]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal

[onedrive4]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal

[onedrive5]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

2024/11/27 21:31:30 DEBUG : rclone: Version "v1.68.1" starting with parameters ["/usr/local/bin/rclone" "mount" "UnionOneDrive:" "/Users/suchan/SecureDrive" "--vfs-cache-mode=full" "--vfs-cache-max-size=10GiB" "--vfs-refresh" "--volname=OneDriveUnion" "-vv"]
2024/11/27 21:31:30 DEBUG : Creating backend with remote "UnionOneDrive:"
2024/11/27 21:31:30 DEBUG : Using config file from "/Users/suchan/.config/rclone/rclone.conf"
2024/11/27 21:31:30 DEBUG : Creating backend with remote "onedrive1:od1"
2024/11/27 21:31:30 DEBUG : Creating backend with remote "onedrive5:od5"
2024/11/27 21:31:30 DEBUG : Creating backend with remote "onedrive4:od4"
2024/11/27 21:31:30 DEBUG : Creating backend with remote "onedrive3:od3"
2024/11/27 21:31:30 DEBUG : Creating backend with remote "onedrive2:od2"
2024/11/27 21:31:32 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.Mfs, searchPolicy = *policy.FF
2024/11/27 21:31:32 DEBUG : vfs cache: root is "/Users/suchan/Library/Caches/rclone"
2024/11/27 21:31:32 DEBUG : vfs cache: data root is "/Users/suchan/Library/Caches/rclone/vfs/UnionOneDrive"
2024/11/27 21:31:32 DEBUG : vfs cache: metadata root is "/Users/suchan/Library/Caches/rclone/vfsMeta/UnionOneDrive"
2024/11/27 21:31:32 DEBUG : Creating backend with remote ":local,encoding='Slash,InvalidUtf8,Dot':/Users/suchan/Library/Caches/rclone/vfs/UnionOneDrive/"
2024/11/27 21:31:32 DEBUG : :local: detected overridden config - adding "{DDwBV}" suffix to name
2024/11/27 21:31:32 DEBUG : union root '': Refreshing VFS directory cache
2024/11/27 21:31:32 DEBUG : : Reading directory tree
2024/11/27 21:31:32 DEBUG : fs cache: renaming cache item ":local,encoding='Slash,InvalidUtf8,Dot':/Users/suchan/Library/Caches/rclone/vfs/UnionOneDrive/" to be canonical ":local{DDwBV}:/Users/suchan/Library/Caches/rclone/vfs/UnionOneDrive"
2024/11/27 21:31:32 DEBUG : Creating backend with remote ":local,encoding='Slash,InvalidUtf8,Dot':/Users/suchan/Library/Caches/rclone/vfsMeta/UnionOneDrive/"
2024/11/27 21:31:32 DEBUG : :local: detected overridden config - adding "{DDwBV}" suffix to name
2024/11/27 21:31:32 DEBUG : fs cache: renaming cache item ":local,encoding='Slash,InvalidUtf8,Dot':/Users/suchan/Library/Caches/rclone/vfsMeta/UnionOneDrive/" to be canonical ":local{DDwBV}:/Users/suchan/Library/Caches/rclone/vfsMeta/UnionOneDrive"
2024/11/27 21:31:32 INFO  : Pictures/Camera Roll/2024/11/20241102_003517344_iOS.heic: vfs cache: removed cache file as Removing old cache file not in use
2024/11/27 21:31:32 DEBUG : Pictures/Camera Roll/2024/11/20241102_003517344_iOS.heic: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/11/27 21:31:32 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Pictures/Camera Roll/2024/11/20241102_003517344_iOS.heic was removed, freed 0 bytes
2024/11/27 21:31:32 INFO  : Pictures/Camera Roll/2024/11/20241102_015117144_iOS.heic: vfs cache: removed cache file as Removing old cache file not in use
2024/11/27 21:31:32 DEBUG : Pictures/Camera Roll/2024/11/20241102_015117144_iOS.heic: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/11/27 21:31:32 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Pictures/Camera Roll/2024/11/20241102_015117144_iOS.heic was removed, freed 0 bytes
2024/11/27 21:31:32 INFO  : Pictures/Camera Roll/2024/11/20241102_015238786_iOS.heic: vfs cache: removed cache file as Removing old cache file not in use
2024/11/27 21:31:32 DEBUG : Pictures/Camera Roll/2024/11/20241102_015238786_iOS.heic: vfs cache: removed metadata from cache as Removing old cache file not in use
2024/11/27 21:31:32 INFO  : vfs cache RemoveNotInUse (maxAge=3600000000000, emptyOnly=false): item Pictures/Camera Roll/2024/11/20241102_015238786_iOS.heic was removed, freed 0 bytes
2024/11/27 21:31:32 INFO  : vfs cache: cleaned: objects 0 (was 3) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2024/11/27 21:31:32 CRITICAL: Fatal error: failed to mount FUSE fs: cannot open: /Users/suchan/SecureDrive: open /Users/suchan/SecureDrive: device not configured

kapitainsky · November 27, 2024, 10:55am

Instead of macFUSE try FUSE-T

suchan · November 28, 2024, 12:39am

Hello again,

And Thank you for the quick response!
I uninstalled MacFUSE, installed Fuse-t as you suggested then restarted and it's now running stable.

Thanks for that!

Now, I'm not sure whether this is a related issue though, for some reason I'm unable to see the contents of my files, especially photos or videos. I thought it's waiting for it to be downloaded, but it's just running in a perpetual loading state (see screenshot). Tried opening in Preview but still not showing the contents. Do I need to configure something else?

asdffdsa · November 28, 2024, 2:57pm

hi,

check the rclone debug log.
none of your remotes are using client id+secret which can cause performance issues.

suchan · November 30, 2024, 10:47am

Hi asdffdsa,

Thank you for your reply.
I tried creating client id+secret on all my remotes.
Will report back if it improves.

Sorry what do you mean : check rclone debug log?
I didn't see anything that could help me in there.

asdffdsa · November 30, 2024, 1:43pm

ok, mabye there is nothing to see or not sure what to look for.

kill the mount
start the mount
try to view a photo
kill the mount
post the full debug log.

suchan · December 1, 2024, 2:04am

Thank you.
You are exactly correct in saying that I'm not sure what to look for.

I did the steps you mentioned and have posted the full debug log

Here's the link to the pastebin

asdffdsa · December 1, 2024, 3:54pm

did you do that or not?

for testing, i would simplify the setup, test on a single remote, not union.

suchan · December 1, 2024, 9:50pm

did you do that or not?

Yes I did sir, got client id+secret on all the remotes

will post update soon with test on single remote.

asdffdsa · December 1, 2024, 9:54pm

ok, good. can you post rclone config redacted name.of.remote:

actually, i would create a new folder, with just one small file .jpg

and mount to that new folder, something like
rclone mount name.of.remote:new.folder ...

suchan · December 1, 2024, 10:52pm

Below is the output

% rclone config redacted UnionOneDrive:

[UnionOneDrive]
type = union
upstreams = onedrive1:od1 onedrive2:od2 onedrive3:od3 onedrive4:od4 onedrive5:od5
create_policy = mfs

This is the output for one of the remote, the other 4 looks exactly the same (other than the name)

% rclone config redacted onedrive1:

[onedrive2]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal
client_id = XXX
client_secret = XXX

I ran this command

/usr/local/bin/rclone mount onedrive2:od2/TestFolder/ /Users/suchan/SecureDrive --vfs-cache-mode=full --vfs-cache-max-size=10GiB --vfs-refresh --volname="OneDrive2" -vv

Then pasted the log here: rclone debug log single folder file - Pastebin.com

This one worked well though, file immediately shows no issues, almost no waiting time, only seems to be an issue when in Union?

suchan · December 5, 2024, 7:08am

Hello @asdffdsa Not meaning to rush you but may I ask if you need me to include any other information?

asdffdsa · December 5, 2024, 2:21pm

i do not use union remotes, maybe there is a setting that needs to be tweaked.
maybe @kapitainsky has an idea?
or
perhaps you are using onedrive in a way it is not designed to be used and pounding it with api calls from union remote.

kapitainsky · December 5, 2024, 2:35pm

Indeed I do use rclone union with OneDrive and have zero issues.

However your point might be very true. OneDrive has rather aggressive throttling - and its details are not really documented (or at least I've never found it). It does not affect my usage but I had issues in the past when transferring too much and too fast.

@suchan - try your union with only subset of data. Maybe you have so many files that when you open it in Finder it all dies (throttled to death) when creating thumbnails?

Try to disable Finder thumbnails creation.

Experiment with some throttling - for the start add to your mount:

--tpslimit 5
--tpslimit-burst 0
--onedrive-delta

It will be slow but if it works you will know that issue is with throttling. You can then increase tpslimit (tpslimit-burst - leave 0 always) until you find sweet spot.

FYI - I have tpslimit set to 20 but it can differ depending on factors like number of union members, your usage patterns etc. And for my usage (sort of archive) I do not care if it is slower but works 100%.

In addition. I switched to create_policy = rand as rclone about stats on OneDrive are so much delayed that mfs does not really work anyway. It might be a bit tricky if you use your drives for something else than union though - then you have to manually (or by script) balance union members from time to time.

asdffdsa · December 5, 2024, 3:04pm

given the OP wants thumbnails, what about this?
--vfs-read-chunk-size=5M --vfs-read-chunk-size-limit=0

kapitainsky · December 5, 2024, 3:09pm

Nice idea. Definitely worth a try.

Maybe also add:

--vfs-cache-max-age 9999h --dir-cache-time 9999h

This way the heaviest usage will be limited to first mount run only.

suchan · December 8, 2024, 9:53am

Hello,

Thank you for both your thoughts and responses. Being a first time user I am learning slowly, but surely, because both of you are so helpful and so patient with me.

My use-case was definitely similar to @kapitainsky in that I am using it as semi-archive. From what I've observed, perhaps the reason it got throttled and hitting the tpslimit is due to the following reasons, not in any particular order:

I started with approximately 100-120 Gb worth of files on my first remote with many pictures / photos already in there. Then I added a whole lot more by disabling iCloud Backup and exporting about nearly 10,000 worth of photos from my iPhone.
Having just found the greatness of rclone union, I am like a little boy let loose in a giant candy store for the first time in his life. I then began moving stuffs (even more photos from various places like Dropbox, GDrive, local folders) which quickly grew my usage by > 100Gb worth of stuffs, again mostly photo (and video) archives. All of that? Into the union drive I just created in one swoop. Finder probably sees all these new files and quickly attempt to create gazillion thumbnails.
Myself being overzealous and frequently checking that my files are safe and I can access it easily, checking files & folders - all those seemingly harmless operations probably adds up and OneDrive didn't like it.

My new command to run rclone, per all your suggestions above is now:

/usr/local/bin/rclone mount UnionOneDrive: /Users/suchan/SecureDrive --tpslimit 5 --tpslimit-burst 0 --onedrive-delta --vfs-read-chunk-size=5M --vfs-read-chunk-size-limit=0 --vfs-cache-max-age 9999h --dir-cache-time 9999h --vfs-cache-mode=full --vfs-cache-max-size=10GiB --vfs-refresh --volname="OneDriveUnion" --daemon --daemon-wait 0

I started with tpslimit 5 as your suggestions and might increase it if I felt necessary.

I'll try the create_policy = rand too