Does My Backend Need to Pass All Tests to Be Added to Rclone?

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

Hi everyone,

I've been working on adding a new backend to Rclone. The backend builds successfully on all platforms, and when I run the workflow on GitHub, all checks pass and show green.

However, when I run internal tests using:

go test -v -remote TestDrive:

some tests fail. I'm not sure if my backend needs to pass all tests to be accepted into Rclone or if partial test success is acceptable.

Can anyone clarify if full test success is required or suggest how to debug these test failures?

Thanks!

         
--- FAIL: TestRmdirsWithFilter (2.56s)
=== RUN   TestCopyURL
    run.go:180: Remote "FileLu root 'rclone-test-nudimum3tere'", Local "Local file system at /tmp/rclone3157771689", Modify Window "1s"
--- PASS: TestCopyURL (13.48s)
=== RUN   TestCopyURLToWriter
--- PASS: TestCopyURLToWriter (0.00s)
=== RUN   TestMoveFile
    run.go:180: Remote "FileLu root 'rclone-test-nudimum3tere'", Local "Local file system at /tmp/rclone3157771689", Modify Window "1s"
    fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3
    fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3
    fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3
    fstest.go:299:
                Error Trace:    /root/master/fstest/fstest.go:299
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:942
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       listing wrong, want
                                  sub/file2 (14) got
                                  file2 (14)
    fstest.go:185:
                Error Trace:    /root/master/fstest/fstest.go:185
                                                        /root/master/fstest/fstest.go:302
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:942
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       Unexpected file "file2"
    fstest.go:197: Not found "sub/file2"
    fstest.go:200:
                Error Trace:    /root/master/fstest/fstest.go:200
                                                        /root/master/fstest/fstest.go:304
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:942
                Error:          Not equal:
                                expected: 0
                                actual  : 1
                Test:           TestMoveFile
                Messages:       1 objects not found
    fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3
    fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3
    fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3
    fstest.go:299:
                Error Trace:    /root/master/fstest/fstest.go:299
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:950
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       listing wrong, want
                                  sub/file2 (14) got
                                  file2 (14), file2 (14)
    fstest.go:185:
                Error Trace:    /root/master/fstest/fstest.go:185
                                                        /root/master/fstest/fstest.go:302
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:950
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       Unexpected file "file2"
    fstest.go:185:
                Error Trace:    /root/master/fstest/fstest.go:185
                                                        /root/master/fstest/fstest.go:302
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:950
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       Unexpected file "file2"
    fstest.go:197: Not found "sub/file2"
    fstest.go:200:
                Error Trace:    /root/master/fstest/fstest.go:200
                                                        /root/master/fstest/fstest.go:304
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:950
                Error:          Not equal:
                                expected: 0
                                actual  : 1
                Test:           TestMoveFile
                Messages:       1 objects not found
    fstest.go:292: Sleeping for 1s for list eventual consistency: 1/3
    fstest.go:292: Sleeping for 2s for list eventual consistency: 2/3
    fstest.go:292: Sleeping for 4s for list eventual consistency: 3/3
    fstest.go:299:
                Error Trace:    /root/master/fstest/fstest.go:299
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:955
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       listing wrong, want
                                  sub/file2 (14) got
                                  file2 (14), file2 (14)
    fstest.go:185:
                Error Trace:    /root/master/fstest/fstest.go:185
                                                        /root/master/fstest/fstest.go:302
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:955
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       Unexpected file "file2"
    fstest.go:185:
                Error Trace:    /root/master/fstest/fstest.go:185
                                                        /root/master/fstest/fstest.go:302
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:955
                Error:          Should be true
                Test:           TestMoveFile
                Messages:       Unexpected file "file2"
    fstest.go:197: Not found "sub/file2"
    fstest.go:200:
                Error Trace:    /root/master/fstest/fstest.go:200
                                                        /root/master/fstest/fstest.go:304
                                                        /root/master/fstest/fstest.go:332
                                                        /root/master/fstest/fstest.go:344
                                                        /root/master/fstest/run.go:340
                                                        /root/master/fs/operations/operations_test.go:955
                Error:          Not equal:
                                expected: 0
                                actual  : 1
                Test:           TestMoveFile
                Messages:       1 objects not found
--- FAIL: TestMoveFile (34.90s)
=== RUN   TestMoveFileWithIgnoreExisting
    run.go:180: Remote "FileLu root 'rclone-test-nudimum3tere'", Local "Local file system at /tmp/rclone3157771689", Modify Window "1s"
panic: test timed out after 10m0s
running tests:
        TestMoveFileWithIgnoreExisting (2s)

goroutine 5117 [running]:
testing.(*M).startAlarm.func1()
        /usr/lib/golang/src/testing/testing.go:2259 +0x3b9
created by time.goFunc
        /usr/lib/golang/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0004aa4e0, {0x23f96c5?, 0x53e27c?}, 0x24f2268)
        /usr/lib/golang/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0xc00007ec00?)
        /usr/lib/golang/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc0004aa4e0, 0xc00065fbd0)
        /usr/lib/golang/src/testing/testing.go:1595 +0xff
testing.runTests(0xc00030ba40?, {0x3a1d1e0, 0x6c, 0x6c}, {0x1?, 0xedf34deb4?, 0x3b41c00?})
        /usr/lib/golang/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc00030ba40)
        /usr/lib/golang/src/testing/testing.go:1925 +0x636
github.com/rclone/rclone/fstest.TestMain(0x478b9a?)
        /root/master/fstest/run.go:69 +0xa5
github.com/rclone/rclone/fs/operations_test.TestMain(...)
        /root/master/fs/operations/operations_test.go:61
main.main()
        _testmain.go:265 +0x1c7

goroutine 13 [chan receive, 10 minutes]:
github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1()
        /root/master/fs/accounting/accounting_unix.go:24 +0x27
created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1
        /root/master/fs/accounting/accounting_unix.go:21 +0xab

goroutine 12 [syscall, 10 minutes]:
os/signal.signal_recv()
        /usr/lib/golang/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/lib/golang/src/os/signal/signal.go:151 +0x1f

goroutine 8 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00034ac80)
        /root/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /root/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 5069 [semacquire]:
sync.runtime_Semacquire(0xc000397788?)
        /usr/lib/golang/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0x27a0c00?)
        /usr/lib/golang/src/sync/waitgroup.go:116 +0x48
github.com/rclone/rclone/fs/walk.walk({0x27a0c00?, 0x3b73580}, {0x27b3118?, 0xc0000526c0}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0xc0004fe840, 0x24f2f30)
        /root/master/fs/walk/walk.go:451 +0x398
github.com/rclone/rclone/fs/walk.walkListDirSorted(...)
        /root/master/fs/walk/walk.go:347
github.com/rclone/rclone/fs/walk.Walk({0x27a0c00, 0x3b73580}, {0x27b3118, 0xc0000526c0}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0xc0003978b8?)
        /root/master/fs/walk/walk.go:75 +0x2a6
github.com/rclone/rclone/fs/walk.listRwalk({0x27a0c00?, 0x3b73580}, {0x27b3118, 0xc0000526c0}, {0x0, 0x0}, 0x58?, 0x0?, 0x3, 0xc000613a58)
        /root/master/fs/walk/walk.go:169 +0x13a
github.com/rclone/rclone/fs/walk.ListR({0x27a0c00, 0x3b73580}, {0x27b3118, 0xc0000526c0}, {0x0, 0x0}, 0x1, 0xffffffffffffffff, 0x3, 0xc000613a58)
        /root/master/fs/walk/walk.go:159 +0x236
github.com/rclone/rclone/fs/walk.GetAll({0x27a0c00, 0x3b73580}, {0x27b3118, 0xc0000526c0}, {0x0, 0x0}, 0x30?, 0x0?)
        /root/master/fs/walk/walk.go:631 +0x105
github.com/rclone/rclone/fstest.CheckListingWithRoot(0xc000897040, {0x27b3118, 0xc0000526c0}, {0x0, 0x0}, {0xc00075a440?, 0x1, 0x1}, {0x0, 0x0, ...}, ...)
        /root/master/fstest/fstest.go:274 +0x3d6
github.com/rclone/rclone/fstest.CheckListingWithPrecision(...)
        /root/master/fstest/fstest.go:332
github.com/rclone/rclone/fstest.CheckItemsWithPrecision(...)
        /root/master/fstest/fstest.go:344
github.com/rclone/rclone/fstest.(*Run).CheckRemoteItems(...)
        /root/master/fstest/run.go:340
github.com/rclone/rclone/fs/operations_test.TestMoveFileWithIgnoreExisting(0x0?)
        /root/master/fs/operations/operations_test.go:970 +0x319
testing.tRunner(0xc000897040, 0x24f2268)
        /usr/lib/golang/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
        /usr/lib/golang/src/testing/testing.go:1648 +0x3ad

goroutine 338 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 329
        /root/master/fs/march/march.go:209 +0x355

goroutine 549 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 524
        /root/master/fs/march/march.go:209 +0x355

goroutine 588 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 579
        /root/master/fs/march/march.go:209 +0x355

goroutine 437 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 428
        /root/master/fs/march/march.go:209 +0x355

goroutine 163 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 154
        /root/master/fs/march/march.go:209 +0x355

goroutine 201 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 192
        /root/master/fs/march/march.go:209 +0x355

goroutine 5131 [select]:
net/http.(*http2clientStream).writeRequest(0xc000655680, 0xc0009d0900)
        /usr/lib/golang/src/net/http/h2_bundle.go:8588 +0xac7
net/http.(*http2clientStream).doRequest(0x129d1c5?, 0xc000371748?)
        /usr/lib/golang/src/net/http/h2_bundle.go:8450 +0x18
created by net/http.(*http2ClientConn).RoundTrip in goroutine 5130
        /usr/lib/golang/src/net/http/h2_bundle.go:8356 +0x308

goroutine 22 [IO wait]:
internal/poll.runtime_pollWait(0x7f805ea1b7c0, 0x72)
        /usr/lib/golang/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc00034aa80?, 0xc00050d000?, 0x0)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00034aa80, {0xc00050d000, 0x1000, 0x1000})
        /usr/lib/golang/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc00034aa80, {0xc00050d000?, 0x41f028?, 0xc00050d005?})
        /usr/lib/golang/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00003f8d0, {0xc00050d000?, 0xc00050d005?, 0x1a?})
        /usr/lib/golang/src/net/net.go:179 +0x45
github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc0002865b8, {0xc00050d000?, 0xc0006138d8?, 0x18?})
        /root/master/fs/fshttp/dialer.go:111 +0x2c
crypto/tls.(*atLeastReader).Read(0xc0006138d8, {0xc00050d000?, 0xc0006138d8?, 0x0?})
        /usr/lib/golang/src/crypto/tls/conn.go:805 +0x3b
bytes.(*Buffer).ReadFrom(0xc0004409a8, {0x277ce20, 0xc0006138d8})
        /usr/lib/golang/src/bytes/buffer.go:211 +0x98
crypto/tls.(*Conn).readFromUntil(0xc000440700, {0x7f801416c598?, 0xc0002865b8}, 0x1000?)
        /usr/lib/golang/src/crypto/tls/conn.go:827 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc000440700, 0x0)
        /usr/lib/golang/src/crypto/tls/conn.go:625 +0x250
crypto/tls.(*Conn).readRecord(...)
        /usr/lib/golang/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc000440700, {0xc00019f000, 0x1000, 0x7c36a5?})
        /usr/lib/golang/src/crypto/tls/conn.go:1369 +0x158
bufio.(*Reader).Read(0xc00093ade0, {0xc00093e2e0, 0x9, 0x39dda90?})
        /usr/lib/golang/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x277b7e0, 0xc00093ade0}, {0xc00093e2e0, 0x9, 0x9}, 0x9)
        /usr/lib/golang/src/io/io.go:335 +0x90
io.ReadFull(...)
        /usr/lib/golang/src/io/io.go:354
net/http.http2readFrameHeader({0xc00093e2e0, 0x9, 0xc0004fe510?}, {0x277b7e0?, 0xc00093ade0?})
        /usr/lib/golang/src/net/http/h2_bundle.go:1635 +0x65
net/http.(*http2Framer).ReadFrame(0xc00093e2a0)
        /usr/lib/golang/src/net/http/h2_bundle.go:1902 +0x85
net/http.(*http2clientConnReadLoop).run(0xc000395f98)
        /usr/lib/golang/src/net/http/h2_bundle.go:9378 +0x11f
net/http.(*http2ClientConn).readLoop(0xc0004e4300)
        /usr/lib/golang/src/net/http/h2_bundle.go:9273 +0x65
created by net/http.(*http2Transport).newClientConn in goroutine 21
        /usr/lib/golang/src/net/http/h2_bundle.go:7945 +0xcbe

goroutine 1916 [chan receive, 6 minutes]:
github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1()
        /root/master/fs/accounting/accounting_unix.go:24 +0x27
created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 1915
        /root/master/fs/accounting/accounting_unix.go:21 +0xab

goroutine 1168 [chan receive (nil chan), 8 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 1159
        /root/master/fs/march/march.go:209 +0x355

goroutine 1109 [chan receive (nil chan), 8 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 1100
        /root/master/fs/march/march.go:209 +0x355

goroutine 718 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 709
        /root/master/fs/march/march.go:209 +0x355

goroutine 5124 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 246 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 221
        /root/master/fs/march/march.go:209 +0x355

goroutine 640 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 631
        /root/master/fs/march/march.go:209 +0x355

goroutine 490 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 481
        /root/master/fs/march/march.go:209 +0x355

goroutine 278 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 269
        /root/master/fs/march/march.go:209 +0x355

goroutine 5129 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 377 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 352
        /root/master/fs/march/march.go:209 +0x355

goroutine 1135 [chan receive (nil chan), 8 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 1126
        /root/master/fs/march/march.go:209 +0x355

goroutine 1056 [chan receive (nil chan), 8 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 1047
        /root/master/fs/march/march.go:209 +0x355

goroutine 402 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 361
        /root/master/fs/march/march.go:209 +0x355

goroutine 5127 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 5128 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 787 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 778
        /root/master/fs/march/march.go:209 +0x355

goroutine 817 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 808
        /root/master/fs/march/march.go:209 +0x355

goroutine 5126 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 5123 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 980 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 971
        /root/master/fs/march/march.go:209 +0x355

goroutine 940 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 931
        /root/master/fs/march/march.go:209 +0x355

goroutine 5125 [select]:
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:396 +0x176
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 1024 [chan receive (nil chan), 9 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2()
        /root/master/fs/march/march.go:211 +0x3c
created by github.com/rclone/rclone/fs/march.(*March).Run in goroutine 1015
        /root/master/fs/march/march.go:209 +0x355

goroutine 5130 [select]:
net/http.(*http2ClientConn).RoundTrip(0xc0004e4300, 0xc0009d0900)
        /usr/lib/golang/src/net/http/h2_bundle.go:8420 +0x44f
net/http.(*http2Transport).RoundTripOpt(0xc000534090, 0xc0009d0900, {0xc0?})
        /usr/lib/golang/src/net/http/h2_bundle.go:7688 +0x1b1
net/http.(*http2Transport).RoundTrip(...)
        /usr/lib/golang/src/net/http/h2_bundle.go:7637
net/http.http2noDialH2RoundTripper.RoundTrip({0xc00048e140?}, 0xc0009d0900?)
        /usr/lib/golang/src/net/http/h2_bundle.go:10242 +0x16
net/http.(*Transport).roundTrip(0xc00048e140, 0xc0009d0900)
        /usr/lib/golang/src/net/http/transport.go:549 +0x39e
net/http.(*Transport).RoundTrip(...)
        /usr/lib/golang/src/net/http/roundtrip.go:17
github.com/rclone/rclone/fs/fshttp.(*Transport).RoundTrip(0xc0003e0070, 0xc0009d0900)
        /root/master/fs/fshttp/http.go:352 +0x348
net/http.send(0xc0009d0900, {0x277c7c0, 0xc0003e0070}, {0xc00050a001?, 0x1f?, 0x0?})
        /usr/lib/golang/src/net/http/client.go:260 +0x606
net/http.(*Client).send(0xc00040e420, 0xc0009d0900, {0xc0006e2830?, 0x6fb905?, 0x0?})
        /usr/lib/golang/src/net/http/client.go:181 +0x98
net/http.(*Client).do(0xc00040e420, 0xc0009d0900)
        /usr/lib/golang/src/net/http/client.go:724 +0x912
net/http.(*Client).Do(...)
        /usr/lib/golang/src/net/http/client.go:590
github.com/rclone/rclone/backend/filelu.(*Fs).List(0xc0000526c0, {0x27a0c00, 0x3b73580}, {0x0, 0x0})
        /root/master/backend/filelu/filelu.go:798 +0x5bd
github.com/rclone/rclone/fs/list.DirSorted({0x27a0c00?, 0x3b73580}, {0x27b3118, 0xc0000526c0}, 0x1, {0x0, 0x0})
        /root/master/fs/list/list.go:24 +0x72
github.com/rclone/rclone/fs/walk.walk.func2()
        /root/master/fs/walk/walk.go:401 +0x1e7
created by github.com/rclone/rclone/fs/walk.walk in goroutine 5069
        /root/master/fs/walk/walk.go:393 +0x19f

goroutine 4564 [chan receive, 1 minutes]:
github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1()
        /root/master/fs/accounting/accounting_unix.go:24 +0x27
created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 4563
        /root/master/fs/accounting/accounting_unix.go:21 +0xab
exit status 2
FAIL    github.com/rclone/rclone/fs/operations  600.036s