Is there a flag that can limit ram usage better than --transfers?

Is there a flag that can limit ram usage better than --transfers?
Usage case rclone-v1.63 on the free remote server e2.micro1 from oracle.
Has 1gb ram total
--transfers 4 will always break
--transfers 3 will rarely break and
--transfers 2 appears to never break...

The command is using --tpslimit and --transfers and default on all else. I was previously using --drive-chunk-size but that just ate up ram I didn't have to spare...

Is there a flag that does a good job of limiting total chunks? or some other alternate way I could set --transfers 4 instead of --transfers 2 but still keep ram usage below ~500-600ish?

In theory --max-size chunks is a flag that doesn't exist? but is strictly determined by --transfers via multiplication? or some such?

Alternatively maybe there's already a good way to do this, but if there is, I don't know it.

It would help if you provide more details e.g. what remotes you are using.

googledrive and box

Not sure they effect much (other than optional flags to increase chunk size that I wasn't using)? since the problem is, as far as O know, running out of ram, because using 600megabytes of ram breaks the computer

This is what I would try:

set environment variable:

export GOGC=20

and add these flags to sync command:

--use-mmap
--checkers=2
--max-backlog=10
--buffer-size=0
 rclone -v copy --tpslimit 512 --transfers 4 --checkers 2 "source" "destination" --use-mmap --max-backlog=10 --buffer-size=0

I cranked --transfers up from 2 to 4, because I wanted to see this fail (or not fail) and none of the changes you suggested would've been needed at --transfers 2

I'm trying to prove a negative, so I'll either reply soon that it didn't help, or I'll reply in the distant future saying it did.

Edit: I had to crank --checkers to 4, because with --checkers below --transfers --transfers wasn't being fully filled. In other words, --checkers 2 would've fixed the problem entirely, which wouldn't've tested all the other changes you recommended at all.

So far so good. Using about 500mb of ram out of 600mb allowed.

Edit: So far so good. Seems to do it for me. No clue what your fix is actually doing, but it works :slight_smile:

1 Like

I think it ran out of ram once, in the 24hours it's been running.

But instead of the whole system becoming weirdly unstable, it resumed normal operation on it's own immediately.

I think maybe instead of running out of ram this time, it gave a similar error message, but simply realized it was at it's limit and did something sensible, instead of not realizing anything and doing something unsensible. Still seems like a good set of flags for a system with 1gb ram (400MiB or so of which is utilized by the OS)

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.