STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.
What is the problem you are having with rclone?
我在服务器上搭建了MinIO对象存储集群服务,并使用rclone mount 存储桶到本地服务器目录中。
目前存储桶的总大小在5TB左右,数据量在250万个左右。
我在本地服务器上运行了web服务,需要近实时的读取挂载目录下的文件,对rclone挂载的实时性能和稳定性要求比较高。
但是现在使用rclone挂载后,几乎在24小时内,rclone对内存的使用会超过50%,加上我其他的服务,内存使用率超过90%,48小时内,rclone mount服务会崩溃,挂载目录会无法访问,目录的元数据信息会变成一堆问号“?”。我不得不重新挂载rclone服务。
rclone mount服务崩溃后,挂载目录的信息:
OBS目录无法访问了,且元数据信息变成一堆问号“????”,但是rclone进程还在。
Run the command 'rclone version' and share the full output of the command.
[root@host-10-10-2-150 home]# rclone version
rclone v1.62.2
- os/version: centos 7.6.1810 (64 bit)
- os/kernel: 3.10.0-957.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none
[root@host-10-10-2-150 home]#
Which cloud storage system are you using? (eg Google Drive)
MinIO
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
我的挂载命令:
rclone mount minio-108:obs-isf /home/OBS --daemon --allow-non-empty --allow-other --dir-cache-time 1s --no-seek --poll-interval 1s --umask 000 --vfs-cache-mode full --vfs-cache-poll-interval 1s --log-file /home/logs/rclone/mount.log
The rclone config contents with secrets removed.
我的rclone配置 :
[root@host-10-10-2-150 home]# cat /root/.config/rclone/rclone.conf
[minio-108]
type = s3
provider = Minio
region = cn-east-1
access_key_id = ***********
secret_access_key = ***********************
endpoint = http://10.10.2.110:9000
[root@host-10-10-2-150 home]#
A log from the command with the -vv
flag
2023/06/16 11:51:51 ERROR : material_library/model_feedback/610581012/2023-06-16/区域人员安全帽/E4C3AFCEDE6B4CA791E37EED2EA634A3/02010005153000000.zip: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: multipart upload failed to upload part: RequestCanceled: request context canceled
caused by: context canceled
2023/06/16 11:51:56 ERROR : material_library/model_feedback/610581012/2023-06-16/区域人员安全帽/E4C3AFCEDE6B4CA791E37EED2EA634A3/original_videos/02010005153000000.mp4: Failed to copy: multipart upload failed to initialise: RequestCanceled: request context canceled
caused by: context canceled
2023/06/16 11:51:56 ERROR : material_library/model_feedback/610581012/2023-06-16/区域人员安全帽/E4C3AFCEDE6B4CA791E37EED2EA634A3/original_videos/02010005153000000.mp4: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: multipart upload failed to initialise: RequestCanceled: request context canceled
caused by: context canceled
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/config/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/data/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/data/dataset/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/data/scripts/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/__pycache__/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/functions/: Dir.Remove not empty
2023/06/16 16:57:45 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/functions/__pycache__/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/modules/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/layers/modules/__pycache__/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/: Dir.Remove error: file does not exist
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/log/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/scripts/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/utils/: Dir.Remove not empty
2023/06/16 16:57:46 ERROR : data/model/4730217b-3ea7-4346-8dd6-7d89175e2584/yolact_auto/utils/__pycache__/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/bin/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/include/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/pthreads/lib/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/stb/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/3rdparty/stb/include/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/__pycache__/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/cmake/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/cmake/Modules/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/config/: Dir.Remove not empty
2023/06/16 17:03:39 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/data/: Dir.Remove not empty
2023/06/16 17:03:40 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/include/: Dir.Remove not empty
2023/06/16 17:03:40 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/obj/: Dir.Remove not empty
2023/06/16 17:03:41 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/scripts/: Dir.Remove not empty
2023/06/16 17:03:41 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/scripts/log_parser/: Dir.Remove not empty
2023/06/16 17:03:42 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/scripts/windows/: Dir.Remove not empty
2023/06/16 17:03:42 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/: Dir.Remove error: file does not exist
2023/06/16 17:03:42 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/src/: Dir.Remove not empty
2023/06/16 17:03:45 ERROR : data/model/f4ffe76f-6ad5-46f6-a835-f21883b03e83/darknet_yolov3/src-auto-label/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/bin/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/include/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/pthreads/lib/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/stb/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/3rdparty/stb/include/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/__pycache__/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/cmake/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/cmake/Modules/: Dir.Remove not empty
2023/06/16 17:24:23 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/config/: Dir.Remove not empty
2023/06/16 17:24:24 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/data/: Dir.Remove not empty
2023/06/16 17:24:24 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/include/: Dir.Remove not empty
2023/06/16 17:24:24 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/obj/: Dir.Remove not empty
2023/06/16 17:24:25 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/scripts/: Dir.Remove not empty
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/scripts/log_parser/: Dir.Remove not empty
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/scripts/windows/: Dir.Remove not empty
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/: Dir.Remove error: file does not exist
2023/06/16 17:24:26 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/src/: Dir.Remove not empty
2023/06/16 17:24:29 ERROR : data/model/b3e3f085-665c-42a6-8134-c9f60d49abfe/darknet_yolov4/src-auto-label/: Dir.Remove not empty
2023/06/17 18:41:02 ERROR : docker/registry/docker/registry/v2/repositories/ocr_over_speed/_uploads/7e6d5f6e-3359-40b1-95eb-0147ac6b0b83/: Dir.Remove not empty
2023/06/17 18:41:02 ERROR : docker/registry/docker/registry/v2/repositories/ocr_over_speed/_uploads/7e6d5f6e-3359-40b1-95eb-0147ac6b0b83/hashstates/: Dir.Remove not empty
2023/06/17 18:41:02 ERROR : docker/registry/docker/registry/v2/repositories/ocr_over_speed/_uploads/7e6d5f6e-3359-40b1-95eb-0147ac6b0b83/hashstates/sha256/: Dir.Remove not empty