Gdrive mount with the smallest memory footprint possible

Help and Support
1

What is the problem you are having with rclone?

I'm currently looking to mount a gdrive shared directory in a VPS that has very limited resources. The directory only contains small files (< 1MB) and the software that monitors the directory will never copy huge amounts of data. It will possibly rename the files inside the directory, so read-only is not an option.

The two conditions that I'm looking to achieve are:

  • The lowest memory footprint possible, without compromising stability.
  • High directory refresh rate. The contents of the directory are constantly changing, and the application should watch for these changes.

What is your rclone version (output from rclone version)

rclone v1.53.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian Server, running Docker 19.03.9 with rclone:latest image.

Which cloud storage system are you using? (eg Google Drive)

Google Drive (Gsuite)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mount gdrive: /mnt/gdrive --config=/config/rclone/rclone.conf --allow-other --gid ${PGID} --uid ${PUID} --buffer-size 8M --log-level ${LOG_LEVEL} --stats 2m --use-mmap --timeout 5m
2

perhaps --buffer-size 0M
" Set to 0 to disable the buffering for the minimum memory usage."

don't need this, as "The default is 5m"
--timeout 5m

3

Thanks!
I was thinking on adding --no-checksum and --no-modtime don't know if that will make any significant impact.

4

not sure that those two will reduce ram usage.

for sure, would not remove --no-checksum, need to know if a file was copied successfully.
and given the small file size, not sure it is a issue.
that is is more of a time constraint than a memory constraint.
the time is takes to compute the checksum..

same for --no-modtime, again about saving time, not memory.
some backends, it is expensive in terms of time, to get the modtime, an extra api call.

and this vps, what are the specs?

5

1GB RAM, 1vCPU. Rclone won't be the only thing running so memory is tight.

6

might set --checkers and --transfers both to 1.

7

Certain flags do nothing on a mount like no-checksum and no-modtime.

felix@gemini:~$ rclone mount GD: /home/felix/test --no-checksum --no-modtime -vv
2020/12/28 19:38:42 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "mount" "GD:" "/home/felix/test" "--no-checksum" "--no-modtime" "-vv"]
2020/12/28 19:38:42 DEBUG : Creating backend with remote "GD:"
2020/12/28 19:38:42 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/12/28 19:38:42 DEBUG : Google drive root '': Mounting on "/home/felix/test"
2020/12/28 19:38:42 DEBUG : : Root:
2020/12/28 19:38:42 DEBUG : : >Root: node=/, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Attr:
2020/12/28 19:38:45 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/12/28 19:38:45 DEBUG : /: ReadDirAll:
2020/12/28 19:38:45 DEBUG : /: >ReadDirAll: item=8, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="backups"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=backups/, err=<nil>
2020/12/28 19:38:45 DEBUG : backups/: Attr:
2020/12/28 19:38:45 DEBUG : backups/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="blah"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=blah, err=<nil>
2020/12/28 19:38:45 DEBUG : blah: Attr:
2020/12/28 19:38:45 DEBUG : blah: >Attr: a=valid=1s ino=0 size=413 mode=-rw-rw-r--, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="crypt"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=crypt/, err=<nil>
2020/12/28 19:38:45 DEBUG : crypt/: Attr:
2020/12/28 19:38:45 DEBUG : crypt/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="hosts"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=hosts, err=<nil>
2020/12/28 19:38:45 DEBUG : hosts: Attr:
2020/12/28 19:38:45 DEBUG : hosts: >Attr: a=valid=1s ino=0 size=225 mode=-rw-rw-r--, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="hosts.old"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=hosts.old, err=<nil>
2020/12/28 19:38:45 DEBUG : hosts.old: Attr:
2020/12/28 19:38:45 DEBUG : hosts.old: >Attr: a=valid=1s ino=0 size=224 mode=-rw-rw-r--, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="test"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=test/, err=<nil>
2020/12/28 19:38:45 DEBUG : test/: Attr:
2020/12/28 19:38:45 DEBUG : test/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="test.pdf"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=test.pdf, err=<nil>
2020/12/28 19:38:45 DEBUG : test.pdf: Attr:
2020/12/28 19:38:45 DEBUG : test.pdf: >Attr: a=valid=1s ino=0 size=2224380 mode=-rw-rw-r--, err=<nil>
2020/12/28 19:38:45 DEBUG : /: Lookup: name="test.txt"
2020/12/28 19:38:45 DEBUG : /: >Lookup: node=test.txt, err=<nil>
2020/12/28 19:38:45 DEBUG : test.txt: Attr:
2020/12/28 19:38:45 DEBUG : test.txt: >Attr: a=valid=1s ino=0 size=0 mode=-rw-rw-r--, err=<nil>
2020/12/28 19:38:51 DEBUG : /: Lookup: name="test2"
2020/12/28 19:38:51 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/12/28 19:38:51 DEBUG : /: Lookup: name="test2"
2020/12/28 19:38:51 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/12/28 19:38:51 DEBUG : /: Lookup: name="test2"
2020/12/28 19:38:51 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2020/12/28 19:38:51 DEBUG : /: Create: name="test2"
2020/12/28 19:38:51 DEBUG : test2: Open: flags=O_WRONLY|O_CREATE|O_EXCL
2020/12/28 19:38:51 DEBUG : : Added virtual directory entry vAdd: "test2"
2020/12/28 19:38:51 DEBUG : test2: >Open: fd=test2 (w), err=<nil>
2020/12/28 19:38:51 DEBUG : /: >Create: node=test2, handle=&{test2 (w)}, err=<nil>
2020/12/28 19:38:51 DEBUG : test2: Attr:
2020/12/28 19:38:51 DEBUG : test2: >Attr: a=valid=1s ino=0 size=0 mode=-rw-rw-r--, err=<nil>
2020/12/28 19:38:51 DEBUG : &{test2 (w)}: Write: len=225, offset=0
2020/12/28 19:38:51 DEBUG : : Added virtual directory entry vAdd: "test2"
2020/12/28 19:38:51 DEBUG : &{test2 (w)}: >Write: written=225, err=<nil>
2020/12/28 19:38:51 DEBUG : &{test2 (w)}: Flush:
2020/12/28 19:38:51 DEBUG : Google drive root '': File to upload is small (225 bytes), uploading instead of streaming
2020/12/28 19:38:52 DEBUG : test2: MD5 = 0e85d4dc7562b55116a9d144da989dfb OK
2020/12/28 19:38:52 INFO  : test2: Copied (new)
2020/12/28 19:38:52 DEBUG : : Added virtual directory entry vAdd: "test2"
2020/12/28 19:38:52 DEBUG : &{test2 (w)}: >Flush: err=<nil>
2020/12/28 19:38:52 DEBUG : &{test2 (w)}: Release:
2020/12/28 19:38:52 DEBUG : test2: WriteFileHandle.Release nothing to do
2020/12/28 19:38:52 DEBUG : &{test2 (w)}: >Release: err=<nil>

Google Drive is a polling remote so you can reduce the time for polling down as well as it defaults a 60 seconds. I use 15 seconds myself but you could do 10 or something smaller I suppose.

--poll-interval 15s

If your files are ~1M, I'd just leave the buffer size at 1M.

The only thing really eating up memory is your directory cache as that can't be tuned really as if you are searching the entire file system, you'll do much better if you just have a high dir-cache-time since it's polling anyway. It really depends on what your application does.

8

do you mean just gdrive or what?

because i have done testing, and it does matter, a lot, on some backends like s3

9

I only use Google Drive as from my log, you can see no-checksum does nada.

10

Fortunately the mount point has only files in it, not directories. Roughly 1000 files, but it gets cleaned quite often.

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.