Rclone combine: upload failed: batcher is shutting down

Help and Support
1

What is the problem you are having with rclone?

I have several mounts that I normally control individually and in the course of the release of "combine" I wanted to combine them. However, uploads do not work. According to the log "upload failed: batcher is shutting down ".

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-105-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox (Combine/Crypt)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone rcd \
  --config=/home/<obfuscated>/.config/rclone/rclone.conf \
  --rc-addr localhost:5572 \
  --rc-user <obfuscated> \
  --rc-pass <obfuscated> \
  --rc-enable-metrics \
  --rc-web-gui \
  --rc-web-gui-no-open-browser \
  --rc-web-gui-update \
  --log-file /var/log/rclone/data.log \
  --tpslimit 15 \
  --tpslimit-burst 15 \
  --transfers 16 \
  --checkers 16 \
  --use-mmap \
  --timeout 15s \
  --buffer-size 256M \
  --dropbox-batch-mode sync

/usr/bin/rclone rc --rc-user <obfuscated> --rc-pass <obfuscated> mount/mount fs=dbx_fbx_storage_combined: mountPoint=/mnt/cloud/Data vfsOpt='{"CacheMode": 3, "CacheMaxAge": 60000000000, "WriteBack": 120000000000, "ChunkSize": "512M"}' mountOpt='{"AllowNonEmpty": true, "AllowOther": true,"AttrTimeout": 600000000000,"DirCacheTime": 86400000000000}'

The rclone config contents with secrets removed.

[dbx_fbx_storage_combined]
type = combine
upstreams = Data1=dbx_fbx_storage_a_crypt: Data2=dbx_fbx_storage_b_crypt: Data3=dbx_fbx_storage_c_crypt: Data4=dbx_fbx_storage_d_crypt: Data5=dbx_fbx_storage_e_crypt: Data6=dbx_fbx_storage_f_crypt: Data7=dbx_fbx_storage_g_crypt: Data8=dbx_fbx_storage_h_crypt: Data9=dbx_fbx_storage_i_crypt:

[dbx_fbx_storage_a_base]
type = dropbox
client_id = <obfuscated>
client_secret = <obfuscated>
token = {"access_token":"<obfuscated>","token_type":"bearer","refresh_token":"<obfuscated>","expiry":"<obfuscated>"}

[dbx_fbx_storage_a_crypt]
type = crypt
remote = dbx_fbx_storage_a_base:
password = <obfuscated>
password2 = <obfuscated>
2

hello and welcome to the forum,

please post the full rclone debug log.
or at least, the top 30 lines and the error message.

3

I think this is fixed already :crossed_fingers:

Can you try the beta for v1.59.1 please?

https://beta.rclone.org/branch/v1.59-stable/v1.59.1-beta.6365.60d59e260.v1.59-stable/

4

I have now tested the whole thing again with the latest rclone beta (rclone v1.60.0-beta.6376.3a8e52de7). The problem still exists. As it seems, it does not always occur immediately.

Here are the log excerpts from the moment the problem occurs.

Upload started:

2022/08/03 02:12:36 DEBUG : folder/file.ext: vfs cache: starting upload
2022/08/03 02:12:36 DEBUG : fln2heqsplt3f49ilnk5saf3r1o4ji4uneaahlg884evapem7i1g/091h2o1mnetq0uqsn519ie8j5k/67o06krr64gk7d75b9hcrs4eur86pfeapeh13eu8bp0jqhi6vvgo987r696i9m2v0smt78j754a1c: Uploading chunk 1/35

Upload is completed:

2022/08/03 02:14:30 DEBUG : fln2heqsplt3f49ilnk5saf3r1o4ji4uneaahlg884evapem7i1g/091h2o1mnetq0uqsn519ie8j5k/67o06krr64gk7d75b9hcrs4eur86pfeapeh13eu8bp0jqhi6vvgo987r696i9m2v0smt78j754a1c: Uploading chunk 35/35

map[bytes:2353393758 eta:<nil> group:global_stats name:folder/file.ext percentage:100 size:2352819294 speed:1.3579913544309968e+07 speedAvg:1.5862150256488271e+07]

2022/08/03 02:15:33 ERROR : folder/file.ext: Failed to copy: upload failed: batcher is shutting down

In fact, that's all that was displayed despite Verbose (-vv).

5

Sorry, it turned out to be a different but related problem with the backend shutdown code.

Can you give this a go?

v1.60.0-beta.6379.d27b53de0.fix-combine-pin on branch fix-combine-pin (uploaded in 15-30 mins)

The problem will happen after the backend cache expires which is 5 minutes from startup by default.

6

All right, thank you very much. I'll try it out tonight!

1 Like
7

Installed the version and rclone has been running for 15 minutes without error. Probably that was the solution.

8

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.60 - I'll also put the patch in v1.59.1

1 Like
9

Thank you for your quick help!

1 Like
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.