Sync to crypt remote kills machine (eats all RAM)

Suspected Bug
1

So I finally started using crypt on my Jottacloud remote as it wasn’t supposed to croak when syncing big files.
Unfortunately this is exactly what happens. rclone kills the whole system while syncing one of my files (100G).
Machine has 32 GB of RAM and about half of that free when sync starts. Takes less than a minute for rclone to eat all RAM, all swap and then crash the machine.

sync command is:
rclone sync --log-file sync_jc.log --log-level INFO --transfers 6 -P /mnt/HDD/Storage/Backup jc-crypt:/Backup

Running FreeNAS and using rclone 1.46.

2

Can you run the same command with debug (-vv) for the one file that causes the problem and share the log?

3

Not much in there:

2019/04/10 18:20:27 DEBUG : rclone: Version “v1.46” starting with parameters ["/root/.config/rclone/rclone" “sync” “–log-file” “sync_jc_.log” “-vv” “–modify-window” “3601s” “–transfers” “6” “-P” “/mnt/HDD/Storage/Backup/User/Image” “jc-crypt:/Backup/User/Image”]
2019/04/10 18:20:27 DEBUG : Using config file from “/root/.config/rclone/rclone.conf”
2019/04/10 18:20:28 INFO : Encrypted drive ‘jc-crypt:/Backup/User/Image’: Waiting for checks to finish
2019/04/10 18:20:28 DEBUG : 360EC7344F42351C-00-00.mrimg: Size and modification time the same (differ by -915.394315ms, within tolerance 1h0m1s)
2019/04/10 18:20:28 DEBUG : 3DB92339B00E699A-00-00.mrimg: Size and modification time the same (differ by -436.602728ms, within tolerance 1h0m1s)
2019/04/10 18:20:28 DEBUG : 3DB92339B00E699A-00-00.mrimg: Unchanged skipping
2019/04/10 18:20:28 DEBUG : 360EC7344F42351C-00-00.mrimg: Unchanged skipping
2019/04/10 18:20:28 DEBUG : 1BE4988484679F60-00-00.mrimg: Size and modification time the same (differ by -681.135679ms, within tolerance 1h0m1s)
2019/04/10 18:20:28 DEBUG : 1BE4988484679F60-00-00.mrimg: Unchanged skipping
2019/04/10 18:20:28 INFO : Encrypted drive ‘jc-crypt:/Backup/User/Image’: Waiting for transfers to finish

4

Actually, syncing to a non-crypted JC remote behaves the exact same way.

5

Since the files never even start transferring I assume it’s the checksum function that’s killing the RAM.

6

Could be but I wasn’t sure how it did that with a file bigger than the default.

https://rclone.org/jottacloud/#advanced-options

I’d agree with that error though as it sounds like the md5sum.

7

Feels like it’s completely ignoring the ‘–jottacloud-md5-memory-limit’ option and just caching everything to RAM.

8

Is your /tmp mounted on a ram disk (tmpfs) by any chance?

9

You can use the environment variable TMPDIR to make rclone use a different directory

10

Is there any way to set the temp variable in rclone directly?

11

There isn’t at the moment, you’ll have to do

export TMPDIR=/…
rclone …

12

That seems to have done the trick! Thanks so much for the help, was almost beginning to panic over here. :smiley:

<3 rclone

1 Like
13

So if I summarize for my benefit, the file was being copied to /tmp (TMPDIR) which was a memory mount point to do the md5sum, which in turn ate up all the local memory on the system.

14

That is correct. Jottacloud needs to make temporary copies of the file when uploading which it does in the system temporary directory. This is normally /tmp but you can override where it is with TMPDIR.

15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.