If more than 251 Elements in the Que to upload -> Fails with Batcher is shutting down

Suspected Bug
1

What is the problem you are having with rclone?

When rclone is mount with vfs-mode full is shut down. Next time it starts it looks into the cache directory. If it finds more than ~251 elements to upload, all uploads fail with "batcher is shutting down".

Run the command 'rclone version' and share the full output of the command.

rclone --version

rclone v1.62.2

  • os/version: ubuntu 22.10 (64 bit)
  • os/kernel: 5.19.0-41-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox as CRYPT

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone rcd \
        --rc-addr '0.0.0.0:5572' \
        --rc-enable-metrics \
        --rc-user 'admin' \
        --rc-pass 'password' \
        --rc-web-gui \
        --rc-web-gui-no-open-browser \
        --rc-web-gui-force-update \
        --rc-serve \
        --log-level 'INFO' \
        --log-file '/var/log/rclone/rcd.log' \
        --tpslimit 12 \
        --tpslimit-burst 12 \
        --cache-dir '/mnt/mergerfs/rclone/cache' #just temporary and problem is also when cache is not on a mergerfs.

/usr/bin/rclone rc options/set \
        --rc-user 'admin' --rc-pass 'password' \
        --json '{"main": {"UserAgent": "someuseragenthere", "Timeout": 3600000000000}, "mount": {"AllowOther": true}, "vfs": {"Umask": 2, "UID": 1000 , "GID": 1000, "PollInterval": 15000000000, "DirCacheTime": 31557600000000000, "CacheMode": 3, "CacheMaxAge": 31557600000000000 }, "log": {"File": "/var/log/rclone/mount.log"}}'


/usr/bin/rclone rc options/set \
        --rc-user 'admin' --rc-pass 'Dedake42' \
        --json '{"vfs": {"CacheMaxSize": 109951162777600}}'

rclone rc --rc-user admin --rc-pass password --json "{\"main\": {\"LogLevel\": \"DEBUG\"}}" options/set

#How to do -vv on API?


/usr/bin/rclone rc mount/mount \
        --rc-user 'admin' --rc-pass 'password' \
        fs=dropbox-media-crypt: mountPoint=/mnt/rclone/dropbox

The rclone config contents with secrets removed.

# cat /root/.config/rclone/rclone.conf
[dropbox-media]
type = dropbox
token = {"access_token":"removed","token_type":"bearer","refresh_token":"removed","expiry":"2023-05-11T16:06:40.116578064+02:00"}

#chunk_size = 128Mi
[dropbox-media-crypt]
type = crypt
remote = dropbox-media:/backend
password = removed
password2 = removed

A log from the command with the -vv flag

I don't know how to anonymize that so it's not the full log

It's finding files in cache to upload and they get uploaded properly in the backgroud till the startup sequence finds more than arround 251 elements to upload, than all uploads fail with error: 

2023/05/11 10:38:29 ERROR : fullPathto.file: Failed to copy: upload failed: batcher is shutting down
2

This sounds like a bug, but I'm going to need more log to diagnose.

Just add -vv to the rclone rcd command.

If you want to email me nick@craig-wood.com the log (or maybe a link to the log somewhere I can download it) I can take a look. Put a link to this forum article in the email so I've got some context - thanks!

3

email with logs sent

The only solution for now is, to move this files out of dropbox, so the full file is in a temporary folder and safely removed from rclone cache. When the que is empty i can restart rclone and upload the file to the cloud without problems.

4

Got the log - thank you.

Looking at it, I can see the problem. The backend created by the rc mount/mount command shuts down too soon. This is easy to fix though - can you give this a go please?

v1.63.0-beta.7042.ff48cce91.fix-rc-mount on branch fix-rc-mount (uploaded in 15-30 mins)

5

I need to get a maintenance slot first. I can test it during the next week. Thanks for the fast fix.

1 Like
6

I tried, but still the same issue with same error message. Logfile was sent to u by e-mail.

7

Thanks for the log.

I see what went wrong and I'm pretty sure this should fix it now :crossed_fingers: - please can you give it a test.

v1.63.0-beta.7045.459337b7f.fix-rc-mount on branch fix-rc-mount (uploaded in 15-30 mins)

1 Like
8

It looks woking. I'm still uploading, because there is a huge backlog. Once finished i will confirm. How can i compare this release to the source code of the stable release to see which differences u made in order to learn something?

9

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.63

The change is here vfs: fix backends being Shutdown too early when startup takes a long … · rclone/rclone@fc798d8 · GitHub

1 Like
10

It seems like the remote will not be mounted before all the files are uploaded. In the Debug Log i can see the progress of uploads in the que, but there are no new messages like "queed for upload in 5s" which happens in the startup phase, when the cache folder is read. So it looks like the cache folder has been processed already.

11

How many files? The files should not have to be uploaded, but they are checked against the remote.

So a lot of files usually means a lot of checking.

12

U were right. I had a grep filter on my tail -f logfile. The cache folder is still being processed. I set transfers to 0, to speed that up.
Now it's 10-15x faster with showing messages in the log saying:

": vfs cache: queuing for upload in 5s"

13

We've debated/talked/discussed that particular issue as it's complex.

If you start before you can validate the data is consistent, you'd have problems. Not sure a good solution was found that doesn't have some potential data impact.

I normally have only a handful of files as it's not the size of the file per se, it's the number of files that have to be checked that makes it not start.

14

I aggree with u 100%, that it is a good idea to wait, to be consistent with the data. And the number of pending files is only so high because of the upload bug. So in the daily usage it schouldn't be an issue. But it would be great to get some more information about how may files need to be proccessed to mount the remote. Easy percentage calculation: Total files in cache devided by already checked files = progress in %

15

I can confirm the fix working. Remote is finaly mounted:

INFO : vfs cache: cleaned: objects 24737 (was 24737) in use 23732, to upload 23732

16

Great!

Still a few to upload though!!!

17

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.