Out of Memory after moving to Rpi 8Gb and SSD

Help and Support
1

What is the problem you are having with rclone?

Earlier I was on Rpi 2GB, SATA external HD, rclone v1.56.0

I upgraded all this on the same day:

  • Rpi 8GB
  • SSD external HD
  • rclone v1.59.2

And since then getting out of memory errors on a regular basis. Just sometimes, 1 out of 5 times it completes without any error.

I have tried all possible --flags combinations that I could find on forums.
Also installed Go to monitor Memory. And there is always enough free memory available.

It is always crashing around 2.7-2.8 GB mark:
runtime: out of memory: cannot allocate 50331648-byte block (2800549888 in use)

Any help would be really appreciated. Thanks

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

  • os/version: raspbian 11.5
  • os/kernel: 5.15.61-v7l+ (armv7l)
  • os/type: linux
  • os/arch: arm
  • go/version: go1.18.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Using: Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync "$SNAPSHOT" "$CLOUD" --rc --cache-db-purge --cache-chunk-no-memory --transfers 4 --check-first --delete-before --log-level INFO 2>&1 |

The rclone config contents with secrets removed.

type = dropbox
token = {"access_token":"

A log from the command with the -vv flag

2022/10/14 08:41:53 INFO  : MBP2020.sparsebundle/bands/2f5: Copied (replaced existing)
runtime: out of memory: cannot allocate 50331648-byte block (2800549888 in use)
fatal error: out of memory

goroutine 2194 [running]:
runtime.throw({0x173c6b0, 0xd})
	runtime/panic.go:992 +0x5c fp=0x4dedc3c sp=0x4dedc28 pc=0x4fc70
runtime.(*mcache).allocLarge(0xb6f16538, 0x3000000, 0x1)
	runtime/mcache.go:215 +0x1f8 fp=0x4dedc64 sp=0x4dedc3c pc=0x274bc
runtime.mallocgc(0x3000000, 0x1487ac8, 0x1)
	runtime/malloc.go:1096 +0x6b0 fp=0x4dedca4 sp=0x4dedc64 pc=0x1cf54
runtime.makeslice(0x1487ac8, 0x3000000, 0x3000000)
	runtime/slice.go:103 +0xac fp=0x4dedcb8 sp=0x4dedca4 pc=0x6ac98
github.com/rclone/rclone/backend/dropbox.(*Object).uploadChunked(0x5405ac0, {0x1baa1f0, 0x49953e0}, {0x1b9fb4c, 0x51e2580}, 0x4e42c00, 0x8000000)
	github.com/rclone/rclone/backend/dropbox/dropbox.go:1641 +0x280 fp=0x4dedd40 sp=0x4dedcb8 pc=0x9d9414
github.com/rclone/rclone/backend/dropbox.(*Object).Update(0x5405ac0, {0x1baa1f0, 0x49953e0}, {0x1b9fb4c, 0x51e2580}, {0xa6b8eb10, 0x5258740}, {0x48abbc8, 0x1, 0x1})
	github.com/rclone/rclone/backend/dropbox/dropbox.go:1791 +0x4fc fp=0x4dede10 sp=0x4dedd40 pc=0x9dac70
github.com/rclone/rclone/fs/operations.Copy({0x1baa1f0, 0x49953e0}, {0x1bb2504, 0x489efa0}, {0x1bb2540, 0x5405ac0}, {0x5257620, 0x25}, {0x1bb2b94, 0x5258740})
	github.com/rclone/rclone/fs/operations/operations.go:498 +0x1914 fp=0x4dedf6c sp=0x4dede10 pc=0x6d9c58
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0x494cc80, {0x1baa1f0, 0x49953e0}, 0x49952f0, {0x1bb2504, 0x489efa0}, 0x0, 0x494cd18)
	github.com/rclone/rclone/fs/sync/sync.go:416 +0x19c fp=0x4dedfc8 sp=0x4dedf6c pc=0x1197d54
github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers.func1()
	github.com/rclone/rclone/fs/sync/sync.go:443 +0x60 fp=0x4dedfec sp=0x4dedfc8 pc=0x1198320
runtime.goexit()
	runtime/asm_arm.s:831 +0x4 fp=0x4dedfec sp=0x4dedfec pc=0x89584
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:443 +0x44

goroutine 1 [semacquire, 24 minutes]:
sync.runtime_Semacquire(0x494cd20)
	runtime/sema.go:56 +0x34
sync.(*WaitGroup).Wait(0x494cd18)
	sync/waitgroup.go:136 +0x94
github.com/rclone/rclone/fs/sync.(*syncCopyMove).stopTransfers(0x494cc80)
	github.com/rclone/rclone/fs/sync/sync.go:451 +0x78
github.com/rclone/rclone/fs/sync.(*syncCopyMove).run(0x494cc80)
	github.com/rclone/rclone/fs/sync/sync.go:888 +0x3ac
github.com/rclone/rclone/fs/sync.runSyncCopyMove({0x1baa210, 0x4846088}, {0x1bb2504, 0x489efa0}, {0x1bb2b58, 0x4c6ba40}, 0x1, 0x0, 0x0, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:1109 +0x238
github.com/rclone/rclone/fs/sync.Sync({0x1baa210, 0x4846088}, {0x1bb2504, 0x489efa0}, {0x1bb2b58, 0x4c6ba40}, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:1115 +0x78
github.com/rclone/rclone/cmd/sync.glob..func1.1()
	github.com/rclone/rclone/cmd/sync/sync.go:67 +0x64
github.com/rclone/rclone/cmd.Run(0x1, 0x1, 0x2810f00, 0x4e9cea0)
	github.com/rclone/rclone/cmd/cmd.go:255 +0xfc
github.com/rclone/rclone/cmd/sync.glob..func1(0x2810f00, {0x4e1e050, 0x2, 0xa})
	github.com/rclone/rclone/cmd/sync/sync.go:65 +0xc8
github.com/spf13/cobra.(*Command).execute(0x2810f00, {0x4957f90, 0xa, 0xa})
	github.com/spf13/cobra@v1.4.0/command.go:860 +0x688
github.com/spf13/cobra.(*Command).ExecuteC(0x280aee8)
	github.com/spf13/cobra@v1.4.0/command.go:974 +0x414
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.4.0/command.go:902
github.com/rclone/rclone/cmd.Main()
	github.com/rclone/rclone/cmd/cmd.go:559 +0x98
main.main()
	github.com/rclone/rclone/rclone.go:14 +0x14
2

hello and welcome to the forum,

have you tried running v1.56.0 on the upgraded setup, does it crash?

most likely,the machine is running out of memory due to the use of --check-first
"This means that all the info on the objects to transfer is held in memory before the transfers start."

--cache-db-purge --cache-chunk-no-memory
those is for the deprecated cache remote, and based on your config, that is not being used

is there a reason you are using --check-first --delete-before

3

Hi Jojo,
Thanks for the quick response.

I was using --check-first because I thought that was crashing Rclone.
And --delete-before because Dropbox used to reset the connection and cause upload errors. And if there were any errors, Rclone would not Delete at the end.

I will try now without these 2 and confirm.

Thanks again.

4

yes, that is by design.

so if you are getting errors, use a debug log and we can take a look at it.

fyi, i run rclone on a 128MiB openwrt router

5

Hi again - sorry what is the command to downgrade to Rclone v1.56.0? I am not an expert.

6

rclone is a portable executable, does not need to be installed.

download that version of rclone from
https://downloads.rclone.org/v1.56.0

7

Hi
Unfortunately, keep getting same error.

With rclone v 1.59.2

rclone sync "$SNAPSHOT" "$CLOUD" --log-level INFO 2>&1 | tee -a "$LOG"

Previous rclone was installed using

sudo apt install rclone

rclone v1.53.3-DEV

  • os/arch: linux/arm
  • go version: go1.15.5

With previous Rclone also same error:

2022/10/14 14:49:14 INFO  : MBP2019.sparsebundle/bands/224: Copied (new)
runtime: out of memory: cannot allocate 50331648-byte block (2658336768 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x1565df5, 0xd)
	runtime/panic.go:1116 +0x5c
runtime.largeAlloc(0x3000000, 0x3c00101, 0x9b2f88)
	runtime/malloc.go:1179 +0x174
runtime.mallocgc.func1()
	runtime/malloc.go:1071 +0x38
runtime.systemstack(0x152)
	runtime/asm_arm.s:347 +0x84
runtime.mstart()
	runtime/proc.go:1116
8

Okay - looks like I might have found the problem.

It was always erroring after running for approx 30 mins.

I had a peek in to the install script:

https://rclone.org/install.sh

The script installs the regular linux-arm version for Raspberry Pi.

https://downloads.rclone.org/rclone-current-linux-arm.zip

Whereas there is a special version for RPI - linux-armv-71.

https://downloads.rclone.org/rclone-current-linux-arm-v7.zip

I manually unzipped and copied that to bin and re-ran the sync. It ran for over 2 hours and completed the uploaded error-free.

Also compared the filesize and they are different for both rclone files.

It is scheduled to run overnight again. Hopefully it should work all good again.

Thanks again for your help and for suggesting changing the version.

9

What output do you see from this command:

echo OS="$(uname)" OS_type="$(uname -m)"

Is the output a correct description of your system? or could it be wrong due to the way you upgraded your RPI?

10

Getting correct putput.

OS=Linux OS_type=armv7l
11

My yesterday's hunch was incorrect.
Since last night upload keeps on crashing within 5 minutes.

Here is the copy of logs.
Please help.

12

sorry, a bit confused?
--- which pi are you testing on?
--- which version of rclone?
--- which operating system and 32 or 64 bit?

and the pastebin
--- not a full debug log, missing the top lines.
--- what was the exact command run?

are you running a 32bit OS on a 64bit raspberry pi4?

13

Hi I am using

  • Raspberry Pi 8GB memory version
  • rclone v1.59.2
  • OS: Raspbian 32 bit
  • below is the link to full Debug log
rclone v1.59.2
- os/version: raspbian 11.5
- os/kernel: 5.15.61-v7l+ (armv7l)
- os/type: linux
- os/arch: arm
- go/version: go1.18.6
- go/linking: static
- go/tags: none

Debug log:

14

not sure what the exact issue is

fwiw, i never ran a 32bit OS on my pi4, only ubuntu server 64bit.

15

I've been always running a 32-bit OS on my Pi4 and never had this rclone out of memory issue.
It's only since last week it has started :frowning:

16

so does not happen with v1.56.0 and does happen with v1.59.2?

17

Hmm - it does happen on v1.56.0 as well.
So Raspberry Pi 8 gb with 32-bit OS is an issue then?

18

with a 32bit os, the max a single process could access should be limited to 4GiB.
but that is still a lot of memory for rclone

i do not know, someone more experienced than myself will need to look at those debug logs.

in the mean time, i would try --dropbox-batch-mode off

19

Thanks!

@ncw Seems like there is missing a case option in install.sh, to make it look something like:

  armv7l)
    OS_type='arm-v7'
    ;;
  arm*)
    OS_type='arm'
    ;;

The issue was discovered by @faridv in this post above.

20

@ncw I had a look at the "out of memory" stack trace in this log:

I cannot immediately spot the issue (without having to spend hours), do you have any ideas?

Perhaps @asdffdsa has a hunch with this proposed workaround: