Rclone compress backend crashes when user rate limit is expired and the .json metadata can't be saved

What is the problem you are having with rclone?

I'm trying to upload a single filesystem snapshot to a google drive remote which is 800GB in size. This obviously exceeds the daily user rate limit. The initial file does upload sucessfully, however is seems when the compress backend attempts to save it's metadata, even when low-level retries is set to 99999, this fails with rclone crashing

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0-DEV

  • os/version: centos 8 (64 bit)
  • os/kernel: 4.18.0-358.el8.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.12
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Several backends
Google drive
Crypt
Chunker 4Tb chunk
Compress gzip 5

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rcat

nice zfs send -c -w -v $newsnap 2>/root/zfssend-$volume.log | nice rclone --low-level-retries=99999999 --retries-sleep 30m --config=/etc/rclone.conf --log-level=DEBUG --log-file=/root/rclonezfs-$volume.log rcat server-backup-compress:zfs/$server/$newsnap-$frequency

The rclone config contents with secrets removed.

[server-backup]
type = drive
client_id = xxx
client_secret = xxx
scope = drive
auth_owner_only = false
use_trash = false
skip_gdocs = false
skip_checksum_gphotos = true
shared_with_me = false
trashed_only = false
allow_import_name_change = false
alternate_export = false
acknowledge_abuse = true
server_side_across_configs = true
stop_on_upload_limit = false
token = xxx
team_drive = xxxxx

[server-backup-crypt]
type = crypt
remote = server-backup:
password = xxx
password2 = xxx
server_side_across_configs = true
filename_encryption = standard
directory_name_encryption = true

[server-backup-compress]
type = compress
remote = server-backup-chunk:

[server-backup-chunk]
type = chunker
remote = server-backup-crypt:
chunk_size = 4Ti
hash_type = md5quick
name_format = *.splt.###
fail_hard = true

A log from the command with the -vv flag

2022/02/13 23:30:15 DEBUG : vcm513adva5gj07l0gm8bs7kg0vbvog5bee7fjskjl6t5gsiav88pt3k3uu3p6sebkojv97990n2q8e9n5snqhib83l0tjsd531lb0r5hk4u2dk6p1krgr1mdu7ppeqbj2makhd7p5kj2d89h2rugvam30: Sending chunk 873488973824 length 8388608
2022/02/13 23:30:16 DEBUG : vcm513adva5gj07l0gm8bs7kg0vbvog5bee7fjskjl6t5gsiav88pt3k3uu3p6sebkojv97990n2q8e9n5snqhib83l0tjsd531lb0r5hk4u2dk6p1krgr1mdu7ppeqbj2makhd7p5kj2d89h2rugvam30: Sending chunk 873497362432 length 8388608
2022/02/13 23:30:17 DEBUG : vcm513adva5gj07l0gm8bs7kg0vbvog5bee7fjskjl6t5gsiav88pt3k3uu3p6sebkojv97990n2q8e9n5snqhib83l0tjsd531lb0r5hk4u2dk6p1krgr1mdu7ppeqbj2makhd7p5kj2d89h2rugvam30: Sending chunk 873505751040 length 4282030
2022/02/13 23:30:19 DEBUG : init_vms@zfs-auto-snap_frequent-2022-02-13-1915-yearly.__________8.gz: md5 = b43320ad5256a418080e64446fb8eb6c OK
2022/02/13 23:30:20 INFO  : init_vms@zfs-auto-snap_frequent-2022-02-13-1915-yearly.__________8.gz.splt.001_9hxa9v: Moved (server-side) to: init_vms@zfs-auto-snap_frequent-2022-02-13-1915-yearly.__________8.gz
2022/02/13 23:30:21 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded)
2022/02/13 23:30:21 DEBUG : pacer: Rate limited, increasing sleep to 1.711088647s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x55c121d80dd9]

goroutine 1 [running]:
panic(0x55c1231acbc0, 0x55c12412bcc0)
        /usr/lib/golang/src/runtime/panic.go:1065 +0x565 fp=0xc000cc32f0 sp=0xc000cc3228 pc=0x55c1214b4aa5
runtime.panicmem()
        /usr/lib/golang/src/runtime/panic.go:212 +0x5d fp=0xc000cc3310 sp=0xc000cc32f0 pc=0x55c1214b275d
runtime.sigpanic()
        /usr/lib/golang/src/runtime/signal_unix.go:734 +0x185 fp=0xc000cc3348 sp=0xc000cc3310 pc=0x55c1214ccd25
github.com/rclone/rclone/backend/compress.(*Fs).putMetadata(0xc001fccc80, 0x55c123551e08, 0xc0000540e0, 0xc001ce2a20, 0x55c1235626b8, 0xc0000e00f0, 0xc0020ea400, 0x1, 0x1, 0xc0004db5e8, ...)
        /builddir/build/BUILD/rclone-1.57.0/backend/compress/compress.go:629 +0x279 fp=0xc000cc3428 sp=0xc000cc3348 pc=0x55c121d80dd9
github.com/rclone/rclone/backend/compress.(*Fs).putWithCustomFunctions(0xc001fccc80, 0x55c123551e08, 0xc0000540e0, 0x55c12351ea58, 0xc0020ee798, 0x55c1235626b8, 0xc0000e00f0, 0xc0020ea400, 0x1, 0x1, ...)
        /builddir/build/BUILD/rclone-1.57.0/backend/compress/compress.go:658 +0x1a5 fp=0xc000cc3510 sp=0xc000cc3428 pc=0x55c121d81165
github.com/rclone/rclone/backend/compress.(*Fs).PutStream(0xc001fccc80, 0x55c123551e08, 0xc0000540e0, 0x7fecac4918f8, 0xc0020ee798, 0x55c1235626b8, 0xc0000e00f0, 0xc0020ea400, 0x1, 0x1, ...)
        /builddir/build/BUILD/rclone-1.57.0/backend/compress/compress.go:707 +0x27e fp=0xc000cc3610 sp=0xc000cc3510 pc=0x55c121d81b9e
github.com/rclone/rclone/fs.PutStreamer.PutStream-fm(0x55c123551e08, 0xc0000540e0, 0x7fecac4918f8, 0xc0000c8060, 0x55c1235626b8, 0xc0000e00f0, 0xc0020ea400, 0x1, 0x1, 0x0, ...)
        /builddir/build/BUILD/rclone-1.57.0/fs/features.go:521 +0xb7 fp=0xc000cc3690 sp=0xc000cc3610 pc=0x55c121874b77
github.com/rclone/rclone/fs/operations.Rcat(0x55c123551e08, 0xc0000540e0, 0x55c12356cc58, 0xc001fccc80, 0x7ffd7ad40188, 0x36, 0x55c123538f60, 0xc000010010, 0xc07a74ae2db611ea, 0xa0ff0adb, ...)
        /builddir/build/BUILD/rclone-1.57.0/fs/operations/operations.go:1347 +0xb4e fp=0xc000cc3998 sp=0xc000cc3690 pc=0x55c121c8532e
github.com/rclone/rclone/fs/operations.RcatSize(0x55c123551e08, 0xc0000540e0, 0x55c12356cc58, 0xc001fccc80, 0x7ffd7ad40188, 0x36, 0x55c123538f60, 0xc000010010, 0xffffffffffffffff, 0xc07a74ae2db611ea, ...)
        /builddir/build/BUILD/rclone-1.57.0/fs/operations/operations.go:1665 +0x67f fp=0xc000cc3a90 sp=0xc000cc3998 pc=0x55c121c8861f
github.com/rclone/rclone/cmd/rcat.glob..func1.1(0x55c123551e00, 0xc0000540e0)
        /builddir/build/BUILD/rclone-1.57.0/cmd/rcat/rcat.go:68 +0x107 fp=0xc000cc3b48 sp=0xc000cc3a90 pc=0x55c122940a67
github.com/rclone/rclone/cmd.Run(0xc0002b0000, 0x55c1241497a0, 0xc000cc3d40)
        /builddir/build/BUILD/rclone-1.57.0/cmd/cmd.go:255 +0xfb fp=0xc000cc3d08 sp=0xc000cc3b48 pc=0x55c12287095b
github.com/rclone/rclone/cmd/rcat.glob..func1(0x55c1241497a0, 0xc0002bc070, 0x1, 0x7)
        /builddir/build/BUILD/rclone-1.57.0/cmd/rcat/rcat.go:67 +0x12e fp=0xc000cc3d78 sp=0xc000cc3d08 pc=0x55c12294090e
github.com/spf13/cobra.(*Command).execute(0x55c1241497a0, 0xc000067380, 0x7, 0xc, 0x55c1241497a0, 0xc000067380)
        /builddir/build/BUILD/rclone-1.57.0/vendor/github.com/spf13/cobra/command.go:860 +0x2c2 fp=0xc000cc3e38 sp=0xc000cc3d78 pc=0x55c1228604e2
github.com/spf13/cobra.(*Command).ExecuteC(0x55c124140920, 0x0, 0x200000003, 0xc000000180)
        /builddir/build/BUILD/rclone-1.57.0/vendor/github.com/spf13/cobra/command.go:974 +0x375 fp=0xc000cc3f18 sp=0xc000cc3e38 pc=0x55c1228612b5
github.com/spf13/cobra.(*Command).Execute(...)
        /builddir/build/BUILD/rclone-1.57.0/vendor/github.com/spf13/cobra/command.go:902
github.com/rclone/rclone/cmd.Main()
        /builddir/build/BUILD/rclone-1.57.0/cmd/cmd.go:555 +0xad fp=0xc000cc3f78 sp=0xc000cc3f18 pc=0x55c122872ecd
main.main()
        /builddir/build/BUILD/rclone-1.57.0/rclone.go:14 +0x25 fp=0xc000cc3f88 sp=0xc000cc3f78 pc=0x55c122a1c125
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:225 +0x263 fp=0xc000cc3fe0 sp=0xc000cc3f88 pc=0x55c1214b7a23
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000cc3fe8 sp=0xc000cc3fe0 pc=0x55c1214f0881

goroutine 2 [force gc (idle), 2 minutes]:
runtime.gopark(0x55c1234dbca0, 0x55c1241e39c0, 0x1411, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a8fb0 sp=0xc0000a8f90 pc=0x55c1214b7e66
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.forcegchelper()
        /usr/lib/golang/src/runtime/proc.go:276 +0xc5 fp=0xc0000a8fe0 sp=0xc0000a8fb0 pc=0x55c1214b7cc5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a8fe8 sp=0xc0000a8fe0 pc=0x55c1214f0881
created by runtime.init.7
        /usr/lib/golang/src/runtime/proc.go:264 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x55c1234dbca0, 0x55c1241e40c0, 0x140c, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a97a8 sp=0xc0000a9788 pc=0x55c1214b7e66
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.bgsweep(0xc0000d2000)
        /usr/lib/golang/src/runtime/mgcsweep.go:182 +0x145 fp=0xc0000a97d8 sp=0xc0000a97a8 pc=0x55c1214a1665
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a97e0 sp=0xc0000a97d8 pc=0x55c1214f0881
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:217 +0x5e

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x55c1234dbca0, 0x55c1241e6080, 0x140d, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a9f78 sp=0xc0000a9f58 pc=0x55c1214b7e66
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.bgscavenge(0xc0000d2000)
        /usr/lib/golang/src/runtime/mgcscavenge.go:314 +0x2a5 fp=0xc0000a9fd8 sp=0xc0000a9f78 pc=0x55c12149f7e5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a9fe0 sp=0xc0000a9fd8 pc=0x55c1214f0881
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:218 +0x85

goroutine 5 [finalizer wait, 724 minutes]:
runtime.gopark(0x55c1234dbca0, 0x55c124216548, 0xffff1410, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a8758 sp=0xc0000a8738 pc=0x55c1214b7e66
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.runfinq()
        /usr/lib/golang/src/runtime/mfinal.go:175 +0xab fp=0xc0000a87e0 sp=0xc0000a8758 pc=0x55c12149664b
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a87e8 sp=0xc0000a87e0 pc=0x55c1214f0881
created by runtime.createfing
        /usr/lib/golang/src/runtime/mfinal.go:156 +0x66

goroutine 7 [select]:
runtime.gopark(0x55c1234dbcf0, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000aa600 sp=0xc0000aa5e0 pc=0x55c1214b7e66
runtime.selectgo(0xc0000aa798, 0xc0000aa77c, 0x0, 0x0, 0x3, 0x55c121482801, 0x1, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0xef7 fp=0xc0000aa738 sp=0xc0000aa600 pc=0x55c1214c9297
go.opencensus.io/stats/view.(*worker).start(0xc0001bf800)
        /builddir/build/BUILD/rclone-1.57.0/vendor/go.opencensus.io/stats/view/worker.go:276 +0xcf fp=0xc0000aa7d8 sp=0xc0000aa738 pc=0x55c121e6130f
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000aa7e0 sp=0xc0000aa7d8 pc=0x55c1214f0881
created by go.opencensus.io/stats/view.init.0
        /builddir/build/BUILD/rclone-1.57.0/vendor/go.opencensus.io/stats/view/worker.go:34 +0x6a

goroutine 34 [GC worker (idle), 724 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000120260, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a4760 sp=0xc0000a4740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000a47e0 sp=0xc0000a4760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a47e8 sp=0xc0000a47e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 19 [GC worker (idle), 724 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000120280, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013c760 sp=0xc00013c740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013c7e0 sp=0xc00013c760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013c7e8 sp=0xc00013c7e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 20 [GC worker (idle), 724 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc0001202a0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013cf60 sp=0xc00013cf40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013cfe0 sp=0xc00013cf60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013cfe8 sp=0xc00013cfe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 35 [GC worker (idle), 724 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc0001202c0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a4f60 sp=0xc0000a4f40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000a4fe0 sp=0xc0000a4f60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a4fe8 sp=0xc0000a4fe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 50 [GC worker (idle), 724 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000436a40, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000138760 sp=0xc000138740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0001387e0 sp=0xc000138760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0001387e8 sp=0xc0001387e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 21 [GC worker (idle), 27 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000436a60, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013d760 sp=0xc00013d740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013d7e0 sp=0xc00013d760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013d7e8 sp=0xc00013d7e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000602000, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000aaf60 sp=0xc0000aaf40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000aafe0 sp=0xc0000aaf60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000aafe8 sp=0xc0000aafe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 51 [GC worker (idle), 19 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc0001202e0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000138f60 sp=0xc000138f40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000138fe0 sp=0xc000138f60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000138fe8 sp=0xc000138fe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000120300, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000ab760 sp=0xc0000ab740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000ab7e0 sp=0xc0000ab760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000ab7e8 sp=0xc0000ab7e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000602020, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013df60 sp=0xc00013df40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013dfe0 sp=0xc00013df60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013dfe8 sp=0xc00013dfe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 10 [GC worker (idle), 19 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000602040, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000abf60 sp=0xc0000abf40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000abfe0 sp=0xc0000abf60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000abfe8 sp=0xc0000abfe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 23 [GC worker (idle), 27 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000602060, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013e760 sp=0xc00013e740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013e7e0 sp=0xc00013e760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013e7e8 sp=0xc00013e7e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 11 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000602080, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000596760 sp=0xc000596740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005967e0 sp=0xc000596760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005967e8 sp=0xc0005967e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc0006020a0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013ef60 sp=0xc00013ef40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013efe0 sp=0xc00013ef60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013efe8 sp=0xc00013efe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 12 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc0006020c0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000596f60 sp=0xc000596f40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000596fe0 sp=0xc000596f60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000596fe8 sp=0xc000596fe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 25 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc0006020e0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013f760 sp=0xc00013f740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013f7e0 sp=0xc00013f760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013f7e8 sp=0xc00013f7e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 13 [GC worker (idle), 19 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000602100, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000597760 sp=0xc000597740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005977e0 sp=0xc000597760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005977e8 sp=0xc0005977e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 26 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000602120, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00013ff60 sp=0xc00013ff40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00013ffe0 sp=0xc00013ff60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00013ffe8 sp=0xc00013ffe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 14 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000602140, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000597f60 sp=0xc000597f40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000597fe0 sp=0xc000597f60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000597fe8 sp=0xc000597fe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 27 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000602160, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000592760 sp=0xc000592740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005927e0 sp=0xc000592760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005927e8 sp=0xc0005927e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 15 [GC worker (idle), 21 minutes]:
runtime.gopark(0x55c1234dbb38, 0xc000602180, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000598760 sp=0xc000598740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005987e0 sp=0xc000598760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005987e8 sp=0xc0005987e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 52 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000120320, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000139760 sp=0xc000139740 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0001397e0 sp=0xc000139760 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0001397e8 sp=0xc0001397e0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 16 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000120340, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000598f60 sp=0xc000598f40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000598fe0 sp=0xc000598f60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000598fe8 sp=0xc000598fe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 53 [GC worker (idle)]:
runtime.gopark(0x55c1234dbb38, 0xc000120360, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000139f60 sp=0xc000139f40 pc=0x55c1214b7e66
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000139fe0 sp=0xc000139f60 pc=0x55c12149a445
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000139fe8 sp=0xc000139fe0 pc=0x55c1214f0881
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 86 [runnable]:
runtime.gopark(0x55c1234dbcf0, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000593dc8 sp=0xc000593da8 pc=0x55c1214b7e66
runtime.selectgo(0xc000593f98, 0xc000593f60, 0x0, 0x0, 0x2, 0x55c1241e6901, 0x3b9d4397, 0xc000593f01)
        /usr/lib/golang/src/runtime/select.go:327 +0xef7 fp=0xc000593f00 sp=0xc000593dc8 pc=0x55c1214c9297
github.com/rclone/rclone/fs/accounting.(*Account).averageLoop(0xc000b34d00)
        /builddir/build/BUILD/rclone-1.57.0/fs/accounting/accounting.go:212 +0x10b fp=0xc000593fd8 sp=0xc000593f00 pc=0x55c121a8520b
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000593fe0 sp=0xc000593fd8 pc=0x55c1214f0881
created by github.com/rclone/rclone/fs/accounting.newAccountSizeName
        /builddir/build/BUILD/rclone-1.57.0/fs/accounting/accounting.go:110 +0x449

goroutine 85 [select]:
runtime.gopark(0x55c1234dbcf0, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a75a0 sp=0xc0000a7580 pc=0x55c1214b7e66
runtime.selectgo(0xc0000a7788, 0xc0000a7748, 0x0, 0x0, 0x2, 0x55c1241e6901, 0x3b9d4193, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0xef7 fp=0xc0000a76d8 sp=0xc0000a75a0 pc=0x55c1214c9297
github.com/rclone/rclone/fs/accounting.(*StatsInfo).averageLoop(0xc00035c1a0)
        /builddir/build/BUILD/rclone-1.57.0/fs/accounting/stats.go:310 +0x165 fp=0xc0000a77d8 sp=0xc0000a76d8 pc=0x55c121a8b5c5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a77e0 sp=0xc0000a77d8 pc=0x55c1214f0881
created by github.com/rclone/rclone/fs/accounting.(*StatsInfo).startAverageLoop.func1
        /builddir/build/BUILD/rclone-1.57.0/fs/accounting/stats.go:339 +0x67

goroutine 88 [chan receive, 724 minutes]:
runtime.gopark(0x55c1234dba98, 0xc0000d72b8, 0xc0002b170e, 0x2)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000a6650 sp=0xc0000a6630 pc=0x55c1214b7e66
runtime.chanrecv(0xc0000d7260, 0xc0000a6780, 0xc000056201, 0xc0000562b0)
        /usr/lib/golang/src/runtime/chan.go:576 +0x37a fp=0xc0000a66e0 sp=0xc0000a6650 pc=0x55c121482e1a
runtime.chanrecv2(0xc0000d7260, 0xc0000a6780, 0xc0002bd500)
        /usr/lib/golang/src/runtime/chan.go:444 +0x2b fp=0xc0000a6710 sp=0xc0000a66e0 pc=0x55c121482a8b
github.com/buengese/sgzip.(*Writer).Write.func1(0xc000742d80)
        /builddir/build/BUILD/rclone-1.57.0/vendor/github.com/buengese/sgzip/gzip.go:362 +0x6f fp=0xc0000a67d8 sp=0xc0000a6710 pc=0x55c121d4078f
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a67e0 sp=0xc0000a67d8 pc=0x55c1214f0881
created by github.com/buengese/sgzip.(*Writer).Write
        /builddir/build/BUILD/rclone-1.57.0/vendor/github.com/buengese/sgzip/gzip.go:358 +0x497

goroutine 942079 [select]:
runtime.gopark(0x55c1234dbcf0, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0000bbda0 sp=0xc0000bbd80 pc=0x55c1214b7e66
runtime.selectgo(0xc0000bbf98, 0xc0000bbf38, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
        /usr/lib/golang/src/runtime/select.go:327 +0xef7 fp=0xc0000bbed8 sp=0xc0000bbda0 pc=0x55c1214c9297
net/http.(*persistConn).writeLoop(0xc00065a480)
        /usr/lib/golang/src/net/http/transport.go:2382 +0xf9 fp=0xc0000bbfd8 sp=0xc0000bbed8 pc=0x55c12183c599
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000bbfe0 sp=0xc0000bbfd8 pc=0x55c1214f0881
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:1744 +0xc9c

goroutine 942078 [IO wait]:
runtime.gopark(0x55c1234dbc78, 0x7fecd45e1618, 0xc000951b02, 0x5)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc0007b1640 sp=0xc0007b1620 pc=0x55c1214b7e66
runtime.netpollblock(0x7fecd45e15f0, 0x72, 0xc000128800)
        /usr/lib/golang/src/runtime/netpoll.go:447 +0xb8 fp=0xc0007b1678 sp=0xc0007b1640 pc=0x55c1214afdd8
internal/poll.runtime_pollWait(0x7fecd45e15f0, 0x72, 0xffffffffffffffff)
        /usr/lib/golang/src/runtime/netpoll.go:227 +0x65 fp=0xc0007b16a0 sp=0xc0007b1678 pc=0x55c1214e9b85
internal/poll.(*pollDesc).wait(0xc000629898, 0x72, 0x1600, 0x1645, 0xffffffffffffffff)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47 fp=0xc0007b16d0 sp=0xc0007b16a0 pc=0x55c12156f047
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000629880, 0xc000128800, 0x1645, 0x1645, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8 fp=0xc0007b1740 sp=0xc0007b16d0 pc=0x55c121570158
net.(*netFD).Read(0xc000629880, 0xc000128800, 0x1645, 0x1645, 0xc0003c4518, 0xc0007b1868, 0x55c12174a218)
        /usr/lib/golang/src/net/fd_posix.go:55 +0x51 fp=0xc0007b17a0 sp=0xc0007b1740 pc=0x55c1215e8411
net.(*conn).Read(0xc00012edb0, 0xc000128800, 0x1645, 0x1645, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/net.go:183 +0x93 fp=0xc0007b1808 sp=0xc0007b17a0 pc=0x55c1215fee73
github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc000c01980, 0xc000128800, 0x1645, 0x1645, 0x55c12148a1c9, 0xc001c659f8, 0x18)
        /builddir/build/BUILD/rclone-1.57.0/fs/fshttp/dialer.go:104 +0x57 fp=0xc0007b1868 sp=0xc0007b1808 pc=0x55c121ab8577
crypto/tls.(*atLeastReader).Read(0xc001c659f8, 0xc000128800, 0x1645, 0x1645, 0xc0007b19f8, 0xc000680000, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:776 +0x65 fp=0xc0007b18d0 sp=0xc0007b1868 pc=0x55c121755805
bytes.(*Buffer).ReadFrom(0xc0005a0278, 0x55c123519c18, 0xc001c659f8, 0x55c121487f07, 0x55c1231d5400, 0x55c123366b80)
        /usr/lib/golang/src/bytes/buffer.go:204 +0xbe fp=0xc0007b1948 sp=0xc0007b18d0 pc=0x55c1215a235e
crypto/tls.(*Conn).readFromUntil(0xc0005a0000, 0x7fecac4d05d8, 0xc000c01980, 0x5, 0xc000c01980, 0x203000)
        /usr/lib/golang/src/crypto/tls/conn.go:798 +0xf3 fp=0xc0007b1998 sp=0xc0007b1948 pc=0x55c121755a73
crypto/tls.(*Conn).readRecordOrCCS(0xc0005a0000, 0x0, 0x0, 0x55c1214b7f1e)
        /usr/lib/golang/src/crypto/tls/conn.go:605 +0x125 fp=0xc0007b1c10 sp=0xc0007b1998 pc=0x55c1217528a5
crypto/tls.(*Conn).readRecord(...)
        /usr/lib/golang/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0005a0000, 0xc0021e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:1276 +0x173 fp=0xc0007b1c70 sp=0xc0007b1c10 pc=0x55c121758cf3
net/http.(*persistConn).Read(0xc00065a480, 0xc0021e4000, 0x1000, 0x1000, 0xc0007a2f00, 0xc0007b1d40, 0x55c121481c95)
        /usr/lib/golang/src/net/http/transport.go:1922 +0x77 fp=0xc0007b1ce0 sp=0xc0007b1c70 pc=0x55c121839cb7
bufio.(*Reader).fill(0xc0046be480)
        /usr/lib/golang/src/bufio/bufio.go:101 +0x108 fp=0xc0007b1d30 sp=0xc0007b1ce0 pc=0x55c12161d508
bufio.(*Reader).Peek(0xc0046be480, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
        /usr/lib/golang/src/bufio/bufio.go:139 +0x51 fp=0xc0007b1d50 sp=0xc0007b1d30 pc=0x55c12161d691
net/http.(*persistConn).readLoop(0xc00065a480)
        /usr/lib/golang/src/net/http/transport.go:2083 +0x1aa fp=0xc0007b1fd8 sp=0xc0007b1d50 pc=0x55c12183a9ca
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0007b1fe0 sp=0xc0007b1fd8 pc=0x55c1214f0881
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:1743 +0xc77

That's a tough one.

I'd say it's safe to open a bug report on github for it.

I don't see an easy way to deal with a 'disk full' on an operation like that as you have to wait until your quota resets to fix it.

1 Like

I would be ok if it would keep retrying the backend until it succeeded until it was successful. It's odd that even with an extremely high low-level-retries it only retries 1/1 times. I also tried with just 9999 retries incase it was overflowing the variable. Is there someway to generate the metadata manually? I've uploaded this file a few times now and it would be a shame to have to do it again unless it's needed for bug testing.

GitBug here: Compress backend crashes if metadata json can't be uploaded due to user rate limit on Google Drive, likely other remotes. · Issue #5994 · rclone/rclone · GitHub

It appears that 5 9's does overflow,, this issue is not a problem with 9999. However I want rclone to retry indefinitely.
setting min low level retry to 30m is an option it seems.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.