Rclone mount Minio内存使用过高，服务崩溃

xigua88-66 · June 21, 2023, 7:48am

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

我在服务器上搭建了MinIO对象存储集群服务，并使用rclone mount 存储桶到本地服务器目录中。
目前存储桶的总大小在5TB左右，数据量在250万个左右。
我在本地服务器上运行了web服务，需要近实时的读取挂载目录下的文件，对rclone挂载的实时性能和稳定性要求比较高。
但是现在使用rclone挂载后，几乎在24小时内，rclone对内存的使用会超过50%，加上我其他的服务，内存使用率超过90%，48小时内，rclone mount服务会崩溃，挂载目录会无法访问，目录的元数据信息会变成一堆问号“？”。我不得不重新挂载rclone服务。

rclone mount服务崩溃后，挂载目录的信息：
OBS目录无法访问了，且元数据信息变成一堆问号“？？？？”，但是rclone进程还在。

Run the command 'rclone version' and share the full output of the command.

[root@host-10-10-2-150 home]# rclone version
rclone v1.62.2

os/version: centos 7.6.1810 (64 bit)
os/kernel: 3.10.0-957.el7.x86_64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: none
[root@host-10-10-2-150 home]#

Which cloud storage system are you using? (eg Google Drive)

MinIO

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

我的挂载命令：
rclone mount minio-108:obs-isf /home/OBS --daemon --allow-non-empty --allow-other --dir-cache-time 1s --no-seek --poll-interval 1s --umask 000 --vfs-cache-mode full --vfs-cache-poll-interval 1s --log-file /home/logs/rclone/mount.log

The rclone config contents with secrets removed.

我的rclone配置 ：
[root@host-10-10-2-150 home]# cat /root/.config/rclone/rclone.conf
[minio-108]
type = s3
provider = Minio
region = cn-east-1
access_key_id = ***********
secret_access_key = ***********************
endpoint = http://10.10.2.110:9000
[root@host-10-10-2-150 home]#

A log from the command with the `-vv` flag

2023/06/16 11:51:51 ERROR : material_library/model_feedback/610581012/2023-06-16/区域人员安全帽/E4C3AFCEDE6B4CA791E37EED2EA634A3/02010005153000000.zip: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: multipart upload failed to upload part: RequestCanceled: request context canceled
caused by: context canceled
2023/06/16 11:51:56 ERROR : material_library/model_feedback/610581012/2023-06-16/区域人员安全帽/E4C3AFCEDE6B4CA791E37EED2EA634A3/original_videos/02010005153000000.mp4: Failed to copy: multipart upload failed to initialise: RequestCanceled: request context canceled
caused by: context canceled
2023/06/16 11:51:56 ERROR : material_library/model_feedback/610581012/2023-06-16/区域人员安全帽/E4C3AFCEDE6B4CA791E37EED2EA634A3/original_videos/02010005153000000.mp4: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: multipart upload failed to initialise: RequestCanceled: request context canceled
caused by: context canceled
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/config/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/data/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/data/dataset/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/data/scripts/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/__pycache__/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/functions/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/functions/__pycache__/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/modules/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/modules/__pycache__/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/: Dir.Remove error: file does not exist
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/log/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/scripts/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/utils/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/utils/__pycache__/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/bin/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/include/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/lib/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/stb/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/stb/include/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/__pycache__/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/cmake/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/cmake/Modules/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/config/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/data/: Dir.Remove not empty
2023/06/16 17:03:40 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/include/: Dir.Remove not empty
2023/06/16 17:03:40 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/obj/: Dir.Remove not empty
2023/06/16 17:03:41 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/scripts/: Dir.Remove not empty
2023/06/16 17:03:41 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/scripts/log_parser/: Dir.Remove not empty
2023/06/16 17:03:42 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/scripts/windows/: Dir.Remove not empty
2023/06/16 17:03:42 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/: Dir.Remove error: file does not exist
2023/06/16 17:03:42 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/src/: Dir.Remove not empty
2023/06/16 17:03:45 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/src-auto-label/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/bin/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/include/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/lib/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/stb/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/stb/include/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/__pycache__/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/cmake/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/cmake/Modules/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/config/: Dir.Remove not empty
2023/06/16 17:24:24 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/data/: Dir.Remove not empty
2023/06/16 17:24:24 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/include/: Dir.Remove not empty
2023/06/16 17:24:24 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/obj/: Dir.Remove not empty
2023/06/16 17:24:25 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/scripts/: Dir.Remove not empty
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/scripts/log_parser/: Dir.Remove not empty
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/scripts/windows/: Dir.Remove not empty
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/: Dir.Remove error: file does not exist
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/src/: Dir.Remove not empty
2023/06/16 17:24:29 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/src-auto-label/: Dir.Remove not empty
2023/06/17 18:41:02 ERROR : docker/registry/docker/registry/v2/repositories/ocr_over_speed/_uploads/7e6d5f6e-3359-40b1-95eb-0147ac6b0b83/: Dir.Remove not empty
2023/06/17 18:41:02 ERROR : docker/registry/docker/registry/v2/repositories/ocr_over_speed/_uploads/7e6d5f6e-3359-40b1-95eb-0147ac6b0b83/hashstates/: Dir.Remove not empty
2023/06/17 18:41:02 ERROR : docker/registry/docker/registry/v2/repositories/ocr_over_speed/_uploads/7e6d5f6e-3359-40b1-95eb-0147ac6b0b83/hashstates/sha256/: Dir.Remove not empty

kapitainsky · June 21, 2023, 8:28am

As more and more data is cached memory usage increases - 5TB/2.5m objects when fully cached will consume big amount of both disk space and memory. Even if you have enough disk space it might be too much for your available memory.

I suggest you limit cache size by adding --vfs-cache-max-size 1000G to your mount command. Decrease/increase this value based on your observed memory usage.

Also please use some online translator when posting. E.g. https://translate.google.com/ - most people here do not speak mandarin.

system · July 21, 2023, 8:29am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.