Rclone Mount not letting data pass

Hey guys,

I apologize for the nondescript title. I struggled to put into words what my issue I was having.

Some info before i get started explaining the issue:
-OS: Centos 7
-Cloud Service being mounted: Google Drive
-remotes:Gdrive, gcache, crypt

Basically Rclone mount works fine for a while and after a while, the mount doesn't let any data pass through it. If I run a 'df -h' it comes back showing the drive mounted. If I run a 'ls -al' on the directory that Rclone mounts, it lists the correct permissions, owner and other info. If I manually copy a file into the mounted folder (to test it though) though, it just sits at 0% done. It doesn't fail. it just sits there trying to copy. I am posting my log file just in case it helps though that seems to show other issues with the cache file.

Interestingly, if I check the status of rclone.service, it says running with no issues.

If I stop the service and restart it though it gives me this message and the only way to make it go away that I've found is to restart the server:
Message: 'Assertion failed on job for rclone.service.'

EDIT: okay so after looking at the log file, it looks like I get that error because of the last line in the log, that its having issues unmounting the directory. As root, if I run that command manually, it works and I can start the Rclone service again without the above message.

Your help is appreciated!

I am not sure that this is relevant to the error im having but wanted to mention that I am using Rclone to encrypt data and store it on Google drive. That server runs Seafile Server and stores its library with rclone.

Here is my mount service.

#/etc/systemd/system/rclone.service
[Install]
WantedBy=default.target

[Unit]
Description=crypt (rclone)
AssertPathIsDirectory=/opt/seafile/seafile-data
After=plexdrive.service

[Service]
Type=simple
User=seafile
ExecStart=/usr/bin/rclone mount
--config=/opt/seafile/.config/rclone/rclone.conf
--dir-cache-time 96h
--cache-info-age 100h
--allow-other
--drive-chunk-size 32M
--allow-non-empty
--vfs-cache-mode writes
--cache-tmp-upload-path=/opt/seafile/sea-cache/upload
--cache-chunk-path=/opt/seafile/sea-cache/chunks
--cache-workers=8
--cache-writes
--cache-dir=/opt/seafile/sea-cache/vfs
--cache-db-path=/opt/seafile/sea-cache/db
--drive-use-trash
--log-file=/var/log/rclone.log
--checkers=16 crypt: /opt/seafile/seafile-data
ExecStop=/bin/fusermount -u /opt/seafile/seafile-data
Restart=always
RestartSec=10

Rclone Log:

2019/09/15 00:41:47 ERROR : worker-0 <8q4b2ivikavamb3ahj8b2vabl3grneo9j91fca1ekb4ak19u3t4bhsc0t70fppoj9vq44q769jp26>: object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:41:47 ERROR : worker-4 <8q4b2ivikavamb3ahj8b2vabl3grneo9j91fca1ekb4ak19u3t4bhsc0t70fppoj9vq44q769jp26>: object open failed 5242880: open /opt/seafile/sea-cache/upload/9um46shqo$
2019/09/15 00:43:04 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/3p4mhejbm7ntant3bspqc73dia2vesrm9oqmm2c3qeoj7ngf0stjcvr3cgne7hn588lf29g4e7vac: failed to rename queued file for upload: pending upload $
2019/09/15 00:43:04 ERROR : worker-0 <3p4mhejbm7ntant3bspqc73dia2vesrm9oqmm2c3qeoj7ngf0stjcvr3cgne7hn588lf29g4e7vac>: object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:43:04 ERROR : worker-1 <3p4mhejbm7ntant3bspqc73dia2vesrm9oqmm2c3qeoj7ngf0stjcvr3cgne7hn588lf29g4e7vac>: object open failed 5242880: open /opt/seafile/sea-cache/upload/9um46shqo$
2019/09/15 00:43:11 ERROR : tmpfiles/953c29fb052a27cf18b2ffd7f59d113c343a0e9c.9EK37Z: Dir.Rename error: file does not exist
2019/09/15 00:43:12 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/kv7fans5efcla9l35ft2d3k0h81qr08lfvfrb773dt3scdu8pfll3g4ioot62avmecgm1fchdj0pk: failed to rename queued file for upload: pending upload $
2019/09/15 00:43:13 ERROR : worker-0 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:44:21 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/2kiom395spivbm25h1ljremanidn7pt4tjiqbmq65d8p0rkibvmm5s29d4fimtriodochbl1j9d92: error refreshing object in : in cache fs Google drive ro$
2019/09/15 00:44:21 ERROR : tmpfiles/a5650dabd1e153b112cb274c0bc8acd261901d42.5BK47Z: File.Remove file error: in cache fs Google drive root 'Rclone/Gdrive2': object not found
2019/09/15 00:44:27 ERROR : tmpfiles/: Dir.Remove error: file does not exist
2019/09/15 00:44:31 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/25aab1095ssq6prjqvj5b7v3jpqsvepbfgarqfu9m3arsmegcmu7i9vl53ptdcpgj76tm2timpiv0: failed to rename queued file for upload: pending upload $
2019/09/15 00:44:32 ERROR : worker-0 <25aab1095ssq6prjqvj5b7v3jpqsvepbfgarqfu9m3arsmegcmu7i9vl53ptdcpgj76tm2timpiv0>: object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:44:32 ERROR : worker-1 <25aab1095ssq6prjqvj5b7v3jpqsvepbfgarqfu9m3arsmegcmu7i9vl53ptdcpgj76tm2timpiv0>: object open failed 5242880: open /opt/seafile/sea-cache/upload/9um46shqo$
2019/09/15 00:44:32 ERROR : worker-2 <25aab1095ssq6prjqvj5b7v3jpqsvepbfgarqfu9m3arsmegcmu7i9vl53ptdcpgj76tm2timpiv0>: object open failed 10485760: open /opt/seafile/sea-cache/upload/9um46shq$
2019/09/15 00:44:56 ERROR : tmpfiles/e116753a861fe99012f6c9f8a4bcc4b0f9e9aa8b.GID47Z: Dir.Rename error: file does not exist
2019/09/15 00:45:04 ERROR : tmpfiles/ba56bfc748d687b4f2d1191aca398ec1ccc22be8.QLR77Z: Dir.Rename error: file does not exist
2019/09/15 00:45:50 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/0sam0atqvplls3h2fif7t483r88llmmqcj7i467bap692msipmlducgai3e92q1ia6jmnhthq55i2: failed to rename queued file for upload: pending upload $
2019/09/15 00:45:51 ERROR : worker-0 <0sam0atqvplls3h2fif7t483r88llmmqcj7i467bap692msipmlducgai3e92q1ia6jmnhthq55i2>: object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:46:06 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/bpqdv6e6hs7q9tan7g3jtuia8rk6uaq8a1ga6i9r0h5su35l101dndf9qqj0560gs3q0i3tlkm1o2: failed to rename queued file for upload: pending upload $
2019/09/15 00:46:07 ERROR : worker-0 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:46:07 ERROR : worker-4 : object open failed 5242880: open /opt/seafile/sea-cache/upload/9um46shqo$
2019/09/15 00:46:23 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/nktajbfpkctkodjl92p3nqquf410s1ff8f157rl1a4btpbddk59q7lvnlmri3i6jnn6l56jc0r6n4: error refreshing object in : in cache fs Google drive ro$
2019/09/15 00:46:23 ERROR : tmpfiles/9145cd83e12b6be5c6276d1c20b23e62b05c9279.GQ4Q7Z: File.Remove file error: in cache fs Google drive root 'Rclone/Gdrive2': object not found
2019/09/15 00:46:38 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/rpm9vjcjnranj7b06osi5s1m0i80lmspsblslflbmcclj1vvi0ct1i0eu6o65mgaa0937qa3na1s2: error moving: can't move object - incompatible remotes
2019/09/15 00:46:38 ERROR : worker-0 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:46:38 ERROR : worker-1 : object open failed 5242880: open /opt/seafile/sea-cache/upload/9um46shqo$
2019/09/15 00:46:38 ERROR : worker-2 : object open failed 10485760: open /opt/seafile/sea-cache/upload/9um46shq$
2019/09/15 00:46:58 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/b5kgn7auqjjiacep2t6io67ka34ado6ptutun97rs87uu5scj1qumo0usimid76p15pshhs2boqlk: error refreshing object in : in cache fs Google drive ro$
2019/09/15 00:46:58 ERROR : tmpfiles/f6eb5a4a1b2cb765ef62317ff5b613162e11b547.B9Q17Z: File.Remove file error: in cache fs Google drive root 'Rclone/Gdrive2': object not found
2019/09/15 00:47:12 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/a70n1avb3t810f74ucns01rhghk7idigllbf98jec769kba2p4ju85aanrqeppgvgdj0kvjelqn3m: failed to rename queued file for upload: pending upload $
2019/09/15 00:47:38 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/lhh5740f7qvrvmi3og7td8slvfduc7lvufvlrtnbke3ko3n4l7vish789a7b6ig33grvrq042eoss: failed to rename queued file for upload: pending upload $
2019/09/15 00:47:39 ERROR : worker-0 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:47:59 ERROR : tmpfiles/77388e0c26c8a5a8f0a983151cb23fcbe0fb4ca8.6AD37Z: Dir.Rename error: file does not exist
2019/09/15 00:48:00 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/ackasenlcm5i886sg3bvh2phr7hkv15lgpf676ngcp5enaqfl69d0sm35e76a3ngv2pnv5ope535u: failed to rename queued file for upload: pending upload $
2019/09/15 00:48:00 ERROR : worker-0 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:48:07 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/0kr4781437juh6o63lgkukbol8up1u5mk8vups3ei31jemuqf296vrh0ttk9k4l01uesdj90o5uqa: failed to rename queued file for upload: pending upload $
2019/09/15 00:48:08 ERROR : worker-3 <0kr4781437juh6o63lgkukbol8up1u5mk8vups3ei31jemuqf296vrh0ttk9k4l01uesdj90o5uqa>: object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:49:24 ERROR : tmpfiles/38d66e54fac91864b9ed111fcaf74e5346c53afa.O5OU7Z: Dir.Rename error: file does not exist
2019/09/15 00:49:38 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/tqpbdeti73rsf2cq3hm38pd28ltu6dk8ks3r9j23gegpdtsel74q1gr2st85oaev2h7hhh4hki2ca: failed to rename queued file for upload: pending upload $
2019/09/15 00:49:38 ERROR : worker-3 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 00:49:38 ERROR : worker-4 : object open failed 5242880: open /opt/seafile/sea-cache/upload/9um46shqo$
2019/09/15 00:50:39 ERROR : 9um46shqo2m47c1uqqvd2e5aj8/cmv37mt2lenenb9dhbtqjm6aptkacg3nvhmlqtdhn0ct6e87sfvt7c4t5frku3invtbqjisq9sma0: failed to rename queued file for upload: pending upload $
2019/09/15 00:50:40 ERROR : worker-0 : object open failed 0: open /opt/seafile/sea-cache/upload/9um46shqo2m47c1$
2019/09/15 01:27:36 Fatal error: failed to umount FUSE fs: exit status 1: fusermount: failed to unmount /opt/seafile/seafile-data: Device or resource busy

I had some kind of similiar issue at the beginning following some online tutorial using Plexdrive where newly added files just wouldn't appear while on the Google Drive website you could see newly created encrypted files.

And since you use After=plexdrive.service I just assume you use it. In that case I suggest just throwing that out. After that I haven't had a similiar issue yet. Also as far as now is a Cache dir not the fastest way anymore tho, rather VFS. But I'm not very into that topic honestly...

Thank you for your reply.

Honestly I am new to system services so I just copied a script I found online and modified it. I am not using Plex, rather im using Seafile.

It seems like the caching directory is causing issues anyway so im going to get rid of that and just use vfs cache mode.

Great.

If you don't use plexdrive, After=plexdrive.service will cause the mount never to happen after reboot.

Okay thanks for the info. What should I replace it with?

Also, one question about caching. Does vfs-cache-mode work with the cache remote or are they t separate things? In other words, would you recommend I get rid of my caching remote and just use vfs-cache-mode on crypt?

I just see I still have the same since I moved from plexdrive 2 weeks ago. :sweat_smile:

In that case you could leave it off, "start after" would just set an order of startups, like plexdrive BEFORE the rclone mount. But that is not needed anymore in this case and it will just startup somewhen with the other system services.

Yeah I never setup a cache remote. As far as I understood those are seperate things and you can leave off the remote. I'm not very into it tho and look for optimizations in another thread too.
However you could look at the Git of Animosity, he also says he uses the cache remote only for testing purposes.

Thank you for the help! I deleted the cache remote and edited the config file to remove all mentions of cache with the exception of vfs-cache-mode full and now caching is working beautifully!

Thank you all!