High memory usage when rclone copy command running

What is the problem you are having with rclone?

When the rclone copy command is running my VPS use all the memory!

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: debian 9.13 (64 bit)
  • os/kernel: 4.9.0-16-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy cryptmount: cryptmount_wp: --log-file /home/xtend/logs/upload.log -v --fast-list --drive-chunk-size 2048M --transfers 2 --max-transfer 720G

The rclone config contents with secrets removed.

normal config nothing special!

A log from the command with the -vv flag

the command running fine so no need log, because memory usage the problem

root 15692 62.5 92.5 13875260 7573100 ? Sl 01:00 257:39 /usr/bin/rclone copy .................

So rclone copy command is eating 90% or more of my VPS memory.
In htop command I see rclone copy running 4-5-6 different PIDs.

Fetching profile over HTTP from http://localhost:5573/debug/pprof/heap
Saved profile in /root/pprof/pprof.rclone.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
File: rclone
Type: inuse_space
Time: Dec 12, 2021 at 8:43am (CET)
Showing nodes accounting for 10351.97kB, 100% of 10351.97kB total
flat flat% sum% cum cum%
1536.61kB 14.84% 14.84% 1536.61kB 14.84% github.com/aws/aws-sdk-go/aws/endpoints.init
1024.04kB 9.89% 24.74% 1024.04kB 9.89% encoding/json.(*decodeState).literalStore
583.01kB 5.63% 30.37% 583.01kB 5.63% bazil.org/fuse.allocMessage
528.17kB 5.10% 35.47% 528.17kB 5.10% regexp.(*bitState).reset
526.13kB 5.08% 40.55% 526.13kB 5.08% github.com/spf13/pflag.(*FlagSet).AddFlag
515.19kB 4.98% 45.53% 515.19kB 4.98% github.com/rclone/rclone/cmd/serve/restic.init
514.38kB 4.97% 50.50% 514.38kB 4.97% bytes.makeSlice
513.31kB 4.96% 55.46% 513.31kB 4.96% regexp.onePassCopy
513.31kB 4.96% 60.41% 513.31kB 4.96% regexp/syntax.(*compiler).inst (inline)
512.50kB 4.95% 65.37% 512.50kB 4.95% encoding/pem.Decode
512.50kB 4.95% 70.32% 512.50kB 4.95% runtime.allocm
512.22kB 4.95% 75.26% 512.22kB 4.95% google.golang.org/protobuf/internal/filedesc.newRawFile
512.20kB 4.95% 80.21% 512.20kB 4.95% runtime.malg
512.17kB 4.95% 85.16% 1024.39kB 9.90% google.golang.org/protobuf/internal/impl.legacyLoadMessageInfo
512.10kB 4.95% 90.11% 512.10kB 4.95% github.com/prometheus/client_golang/prometheus.(*Registry).Register
512.08kB 4.95% 95.05% 512.08kB 4.95% github.com/spf13/pflag.(*FlagSet).VarPF
512.05kB 4.95% 100% 512.05kB 4.95% regexp/syntax.(*parser).newRegexp (inline)

Because you told it to.

Best to use defaults if you aren't sure what something does.

That settings makes it us 2G per file transferring.

Oh....Jeeeeesus what a noob mistake. Thank you! You are right! Now its use only about 200-300MB not 8GB :smiley:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.