Out of memory: error killed process

Help and Support
1

Hi guys! This is my first time here. I hope everyone is doing well.
I currently have a VPS where Jellyfin personal is running (for those who don't know, it's a Plex-style tool, a service to create my own home Netflix).

I'm starting to use rclone with S3-compatible storage to set it up as a directory for metadata (images, movie synopses, actor images, etc.) which currently, for about 160 movies and 5 series, all metadata takes up nearly 600 MB.

My VPS has 2 GB of RAM and I added some swap memory.

The problem is with rclone. As I've noticed, rclone uses RAM when migrating files. However, I'm not actually performing a migration, so that out of memory error would occur regardless.

The process that gives me an error is when I scan the library and save metadata. In this process, images of added movies, their synopses, relevant info, as well as their actors' info, are obtained.

So, every time I upload a large amount of content or analyze the entire library and replace all existing metadata, is when the error occurs precisely because I don't have enough RAM, and I would like it not to consume so much for intensive processes.

I would appreciate any help you could provide with this since I don't know what parameters to use when running the rclone mount command to prevent this error.

Currently, the parameters I was using were --s3-chunk-size 5M and --s3-upload-concurrency 1, and I also tried the speed limit one.

Unfortunately, none of that worked for me. I would appreciate any help you could provide.

Thank you very much.

2

welcome to the forum,

not sure what you mean?

might test
--transfers=1 --vfs-read-chunk-size=5M --vfs-read-chunk-size-limit=0 --use-mmap --buffer-size=0

when you posted, there was a template of questions for you to answer.
help us to help you and please answer all of them?

1 Like
3

Ok, it seems to be working for now, although I notice that when it searches and stores files (writing task to S3), it takes longer. What should I adjust to speed up this process?

Also, I think I had a failure yesterday and I don't know what happened, but I think rclone is storing a duplicate cache on my VPS?? If so, how can I limit this from happening? I opened a 15 GB file in an S3 and then my VPS magically filled up when the heavy file was not on the VPS but in the S3. It's useless if I have an S3 for additional storage but end up duplicating the content on the VPS (which has little storage). Do the parameters you gave me slow down the reading of files?

Thank you very much for your help.

4

welcome

post the output of
rclone version
rclone confg redacted
and the top 30 lines from debug output of rclone mount command

and might check out my summary of the two vfs caches

  • vfs dir cache
  • vfs file cache
5

Rclone version:

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.8

Rclone mount:

rclone mount remote:path /path/to/mountpoint [flags]

Flags:
--allow-non-empty Allow mounting over a non-empty directory.
--allow-other Allow access to other users.
--allow-root Allow access to root user.
--attr-timeout duration Time for which file/directory attributes are cached. (default 1s)
--daemon Run mount as a daemon (background mode).
--daemon-timeout duration Time limit for rclone to respond to kernel (not supported by all OSes).
--debug-fuse Debug the FUSE internals - needs -v.
--default-permissions Makes kernel enforce access control based on the file mode.
--dir-cache-time duration Time to cache directory entries for. (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
--file-perms FileMode File permissions (default 0666)
--fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required.
--gid uint32 Override the gid field set by the filesystem.
-h, --help help for mount
--max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads. (default 128k)
--no-checksum Don't compare checksums on up/download.
--no-modtime Don't read/write the modification time (can speed things up).
--no-seek Don't allow seeking in files.
-o, --option stringArray Option for libfuse/WinFsp. Repeat if required.
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable. (default 1m0s)
--read-only Mount read-only.
--uid uint32 Override the uid field set by the filesystem.
--umask int Override the permission bits set by the filesystem.
--vfs-cache-max-age duration Max age of objects in the cache. (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off)
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s)
--vfs-case-insensitive If a file name not found, find a case insensitive match.
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--volname string Set the volume name (not supported by all OSes).
--write-back-cache Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command mount needs 2 arguments minimum: you provided 0 non flag arguments:

The command "rclone config redacted" is not recognized as such. I think it might be due to the version or some error.

Thank you very much.

6

need to uninstall the ancient version from 2019-11-19.

update to latest v1.66.0
https://rclone.org/install/

7

Rclone config redacted:

[maxytv]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://gateway.storjshare.io
Double check the config for sensitive info before posting publicly

Rclone mount:

Usage:
rclone mount remote:path /path/to/mountpoint [flags]

Flags:
--allow-non-empty Allow mounting over a non-empty directory (not supported on Windows)
--allow-other Allow access to other users (not supported on Windows)
--allow-root Allow access to root user (not supported on Windows)
--async-read Use asynchronous reads (not supported on Windows) (default true)
--attr-timeout Duration Time for which file/directory attributes are cached (default 1s)
--daemon Run mount in background and exit parent process (as background output is suppressed, use --log-file with --log-format=pid,... to monitor) (not supported on Windows)
--daemon-timeout Duration Time limit for rclone to respond to kernel (not supported on Windows) (default 0s)
--daemon-wait Duration Time to wait for ready mount from daemon (maximum time on Linux, constant sleep time on OSX/BSD) (not supported on Windows) (default 1m0s)
--debug-fuse Debug the FUSE internals - needs -v
--default-permissions Makes kernel enforce access control based on the file mode (not supported on Windows)
--devname string Set the device name - default is remote:path
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
--file-perms FileMode File permissions (default 0666)
--fuse-flag stringArray Flags or arguments to be passed direct to libfuse/WinFsp (repeat if required)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows)
-h, --help help for mount
--max-read-ahead SizeSuffix The number of bytes that can be prefetched for sequential reads (not supported on Windows) (default 128Ki)
--mount-case-insensitive Tristate Tell the OS the mount is case insensitive (true) or sensitive (false) regardless of the backend (auto) (default unset)
--network-mode Mount as remote network drive, instead of fixed disk drive (supported on Windows only)
--no-checksum Don't compare checksums on up/download
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
--noappledouble Ignore Apple Double (._) and .DS_Store files (supported on OSX only) (default true)
--noapplexattr Ignore all "com.apple.*" extended attributes (supported on OSX only)
-o, --option stringArray Option for libfuse/WinFsp (repeat if required)
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
--uid uint32 Override the uid field set by the filesystem (not supported on Windows)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 18)
--vfs-block-norm-dupes If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost)
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
--vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
--vfs-case-insensitive If a file name not found, find a case insensitive match
--vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
--vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
--vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
--vfs-refresh Refreshes the directory cache recursively in the background on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
--volname string Set the volume name (supported on Windows and OSX only)
--write-back-cache Makes kernel buffer writes before sending them to rclone (without this, writethrough caching is used) (not supported on Windows)

This is what came up. I had an issue and had to download version 1.67 beta of rclone.

8

ok, best not use beta, what was the issue?

post the output of rclone version
and
mount command

9

Nowadays storj is supported provider so you can change it to:

provider = Storj

Not that it magically will fix all your issues but good to have all details right.

10

rclone version comand:

rclone v1.66.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-173-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

mount command:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=967256k,nr_inodes=241814,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=196972k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13221)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)

The command rclone mount shows me the flags just as I showed you in the previous message.
I was able to reinstall version 1.66.
The mount command shows me what I indicated above.

11

Yes, I know that Storj is compatible, but I prefer manual mounting since I'm only using Storj for testing purposes (since it's free), and then I'll pay for Contabo (which is not compatible). That's why I did it this way, to be able to learn and carry out the procedures in the same manner and not get lost in the process.

12

run your rclone mount command with -vv flag here. So we can see what you are using and what are results.

13

are you using rclone mount or what?
post whatever commands that you are using.

can you please post the rclone mount command.

14

This is wrong approach. Different providers are not the same and rclone adjust some parameters automatically based on what you configure.

E.g. for storj:

--chunk-size is forced to be 64 MiB or greater. This will use more memory than the default of 5 MiB.
Server side copy is disabled as it isn't currently supported in the gateway.
GetTier and SetTier are not supported.

Some of these is needed for your remote to work properly.

If you do not understand implications of changing parameters, especially when you are learning you should take the easy path

15

Yes, sorry, I understand now. I need to mount my network drive with the -vv flag. My mistake.

Rclone mount with -vv:

2024/03/17 16:28:59 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "mount" "maxytv:" "/home/ubuntu/s3-mount/" "--allow-other" "--vfs-read-chunk-size=5M" "--vfs-read-chunk-size-limit=0" "--buffer-size=0" "--transfers=1" "-vv"]
2024/03/17 16:28:59 DEBUG : Creating backend with remote "maxytv:"
2024/03/17 16:28:59 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/03/17 16:28:59 DEBUG : Resolving service "s3" region "us-east-1"
2024/03/17 16:28:59 INFO  : S3 root: poll-interval is not supported by this remote
2024/03/17 16:28:59 DEBUG : S3 root: Mounting on "/home/ubuntu/s3-mount/"
2024/03/17 16:28:59 DEBUG : : Root:
2024/03/17 16:28:59 DEBUG : : >Root: node=/, err=<nil>
16

I understand you perfectly. But will it really benefit me? As I mentioned, I'm only using storj for testing and getting to know the indicated flags I'll need when mounting my S3 from contabo (I'll have to mount it as a generic S3). Therefore, I'd like to test the flags and configurations I need by installing storj as a compatible S3, so that when I use contabo, I'll have to apply the same flags and configurations I applied with storj. Correct me if I'm wrong. Otherwise, I'll install it natively as storj and that's it.

Basically, I don't want to start paying for object storage on contabo that isn't natively supported by rclone and then encounter errors because I'm missing specific flags that, for example, storj automatically applied for me. And of course, it's not much to pay on contabo, but until I manage to find the right configuration and flags... that's why I mount storj as a generic S3.

17

ok, that looks good.

18

I'm relieved that you see everything is correct. However, I would appreciate it if you could help me with the cache and temporary files. I confess that I've read your message and I would be interested in using --vfs-cache-mode=off but, as I mentioned, it gives me an error when I scan the library.

Additionally, by using this flag, is it completely safe that nothing from the consulted file or the movie I'm playing on my Jellyfin server is stored on my VPS?

Well... at least it used to work for me before. Let me try with the flags you provided me at the beginning.

19

Okay, using the flags
--allow-others --transfers=1 --vfs-read-chunk-size=5M --vfs-read-chunk-size-limit=0 --buffer-size=0 --vfs-cache-mode=off, it works, although library scanning and fetching metadata from the internet are slow.

20

that is the default setting, and that is what your command is using
if you were using a cache, that would be shown in the debug log.

in any event, that flag is about the vfs file cache, using local storage, not about memory.
memory is consumed by the vfs dir cache.

up above, i shared a link about a post of mine, summary of the two caches, did you read it?

many topics in the forum about tweaking scan settings in plex/jellyin, to reduce the amount of data that needs to be scanned per file.
in my case, a scan of a file, which used reads approx 15MiB
so on my system, if --vfs-read-chunk-size=20M, then one chunk per file.

and fwiw, storage on contabo looks very expensive, 5TiB is $59.80/month?
i use hetzner, get 5TiB for $11.87/month