Rclone mount systemd service doesnt keep data in sync and shows slow performance

vitachaos · January 11, 2022, 10:01pm

I have written this service that uses rclone to read the data from Minio S3 and mounts to /data location on disk.

# cat /etc/systemd/system/rclone-mount-data.service 
[Unit]
Description=RClone Service
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/etc/rclone.conf
KillMode=none
RestartSec=5
ExecStart=/usr/local/bin/rclone --rc-addr 0.0.0.0:5574 mount  minio:data /data \
--allow-other \
--buffer-size 256M \
--dir-cache-time 1000h \
--log-level DEBUG \
--log-file /var/log/rclone.log \
--timeout 1h \
--umask 002 \
--rc

ExecStop=/bin/fusermount -uz /data 
Restart=on-failure

the problem is if I delete the file from minio it is not deleted from the local mount on disk where thsi service is running ?

I kept sqllite db file in the /data but I am getting this error:

What is the problem you are having with rclone?

I kept sqllite db file in the /data but I am getting this error:

Connecting to the Database
Trace: KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_SQLite3.acquireConnection (/app/node_modules/knex/lib/client.js:305:26)
    at async Runner.ensureConnection (/app/node_modules/knex/lib/execution/runner.js:259:28)
    at async Runner.run (/app/node_modules/knex/lib/execution/runner.js:30:19)
    at async RedBeanNode.normalizeRaw (/app/node_modules/redbean-node/dist/redbean-node.js:570:22)
    at async RedBeanNode.exec (/app/node_modules/redbean-node/dist/redbean-node.js:534:9)
    at async Function.connect (/app/server/database.js:115:9)
    at async initDatabase (/app/server/server.js:1429:5)
    at async /app/server/server.js:180:5 {
  sql: 'PRAGMA foreign_keys = ON',
  bindings: []
}
    at process.<anonymous> (/app/server/server.js:1542:13)
    at process.emit (events.js:400:28)
    at processPromiseRejections (internal/process/promises.js:245:33)
    at processTicksAndRejections (internal/process/task_queues.js:96:32)
If you keep encountering errors, please report to https://github.com/louislam/uptime-kuma/issues
events.js:377
      throw er; // Unhandled 'error' event
      ^

Error: EPERM: operation not permitted, close
Emitted 'error' event on WriteStream instance at:
    at emitErrorNT (internal/streams/destroy.js:106:8)
    at emitErrorCloseNT (internal/streams/destroy.js:74:3)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  errno: -1,
  code: 'EPERM',
  syscall: 'close'
}

am I doing something wrong in the rclone systtemd unit file to mount to disk ?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount

The rclone config contents with secrets removed.

Paste config here

A log from the command with the `-vv` flag

2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: Open: flags=OpenReadWrite+0x20000
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: Open: flags=O_RDWR|0x20000
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: >Open: fd=uptime_kuma/kuma.db (w), err=<nil>
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: >Open: fh=&{uptime_kuma/kuma.db (w)}, err=<nil>
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Read: len=16384, offset=0
2022/01/11 20:16:44 ERROR : uptime_kuma/kuma.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Read: read=0, err=operation not permitted
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Read: len=4096, offset=0
2022/01/11 20:16:44 ERROR : uptime_kuma/kuma.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Read: read=0, err=operation not permitted
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: Attr: 
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: >Attr: a=valid=1s ino=0 size=27176960 mode=-rw-rw-r--, err=<nil>
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Flush: 
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: WriteFileHandle.Flush unwritten handle, writing 0 bytes to avoid race conditions
2022/01/11 20:16:44 ERROR : uptime_kuma/kuma.db: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Flush: err=operation not permitted
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Release: 
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: WriteFileHandle.Release closing
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Release: err=<nil>
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: Open: flags=OpenReadWrite+0x20000
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: Open: flags=O_RDWR|0x20000
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: >Open: fd=uptime_kuma/kuma.db (w), err=<nil>
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: >Open: fh=&{uptime_kuma/kuma.db (w)}, err=<nil>
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Read: len=16384, offset=0
2022/01/11 20:16:44 ERROR : uptime_kuma/kuma.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Read: read=0, err=operation not permitted
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Read: len=4096, offset=0
2022/01/11 20:16:44 ERROR : uptime_kuma/kuma.db: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Read: read=0, err=operation not permitted
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: Attr: 
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: >Attr: a=valid=1s ino=0 size=27176960 mode=-rw-rw-r--, err=<nil>
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: Flush: 
2022/01/11 20:16:44 DEBUG : uptime_kuma/kuma.db: WriteFileHandle.Flush unwritten handle, writing 0 bytes to avoid race conditions
2022/01/11 20:16:44 ERROR : uptime_kuma/kuma.db: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes
2022/01/11 20:16:44 DEBUG : &{uptime_kuma/kuma.db (w)}: >Flush: err=operation not permitted

asdffdsa · January 12, 2022, 2:09am

hello and welcome to the forum,

rclone mount tries to emulate a local file system.

different programs have different behaviors
some are read-only
some write only, either sequential access or random

so rclone has different flags to tweaks its behavior to match the program behavior.

as per the debug log,
WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes

so i would test --vfs-cache-mode writes
if still problems, test using --vfs-cache-mode full

After-Shock · January 30, 2022, 10:06am

Did you end up figuring this out? Running into a similar issue and I already had --vfs-cache-mode full on my vfs service.

agneev · January 30, 2022, 12:02pm

Opening a database stored in the cloud is a terrible idea.

Animosity022 · January 30, 2022, 12:37pm

Best to open a new post with all the details as it should work without issue albeit maybe a little slow.

vitachaos · February 6, 2022, 6:29pm

Well, I think using rclone to access db from an s3 won't be a good idea.
I cannot get it to work even with
--vfs-cache-mode full

system · March 8, 2022, 6:30pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.