Rclone sync objects from IBM COS to Azure Blob

Help and Support
1

What is the problem you are having with rclone?

rclone takes more than 5 hours of time to sync the objects between IBM COS to Azure blob. rclone job eventually stops with no error.

Tried various permutations and combinations.

With 90 days data:
Total objects: 447104
Total size: 367.440 GBytes (394535570884 Bytes)

rclone sync remote_prd:prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level INFO --max-age 90d --delete-excluded --use-server-modtime > 90Sync.txt &

rclone sync remote_prd:bkt-prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level ERROR --max-age 90d --delete-excluded --use-server-modtime --azureblob-chunk-size 64M --bwlimit 80M --transfers 1 --progress --max-transfer 750G --azureblob-memory-pool-use-mmap --azureblob-disable-checksum > 90Sync.txt &

rclone sync remote_prd:bkt-prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level ERROR --max-age 90d --delete-excluded --use-server-modtime --azureblob-memory-pool-use-mmap --azureblob-disable-checksum

Soft delete is enabled for IBM COS bucket configuration.
Works well for sync <= 20GB.

What is your rclone version (output from rclone version)

# rclone version
rclone v1.53.1
- os/arch: linux/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

# uname -r
3.10.0-1160.6.1.el7.x86_64

Which cloud storage system are you using? (eg Google Drive)

IBM COS to Azureblob. Even getting the size of IBM COS bucket takes more time.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# Tried various options...none works well. Could someone help on getting optimal configuration to sync # between IBM COS and azureblob
rclone sync remote_prd:prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level INFO --max-age 90d --delete-excluded --use-server-modtime > 90Sync.txt &

rclone sync remote_prd:bkt-prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level ERROR --max-age 90d --delete-excluded --use-server-modtime --azureblob-chunk-size 64M --bwlimit 80M --transfers 1 --progress --max-transfer 750G --azureblob-memory-pool-use-mmap --azureblob-disable-checksum > 90Sync.txt &

rclone sync remote_prd:bkt-prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level ERROR --max-age 90d --delete-excluded --use-server-modtime --azureblob-memory-pool-use-mmap --azureblob-disable-checksum

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

# Command Used
rclone sync remote_prd:bkt-prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level DEBUG --max-age 90d --delete-excluded --use-server-modtime  --fast-list --azureblob-disable-checksum  --bwlimit 80M --progress > 90Sync.txt &

#log trimmed for error
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x1ab8594, 0x16)
	runtime/panic.go:1116 +0x72
runtime.sysMap(0xc15c000000, 0x4000000, 0x28224b8)
	runtime/mem_linux.go:169 +0xc6
runtime.(*mheap).sysAlloc(0x2806c20, 0x400000, 0x42c037, 0x2806c28)
	runtime/malloc.go:727 +0x1e5
runtime.(*mheap).grow(0x2806c20, 0x200, 0x0)
	runtime/mheap.go:1344 +0x85
runtime.(*mheap).allocSpan(0x2806c20, 0x200, 0x430100, 0x28224c8, 0x200000001)
	runtime/mheap.go:1160 +0x6b6
runtime.(*mheap).alloc.func1()
	runtime/mheap.go:907 +0x65
runtime.(*mheap).alloc(0x2806c20, 0x200, 0xc0100c0101, 0xc0100ce000)
	runtime/mheap.go:901 +0x85
runtime.largeAlloc(0x400000, 0xc000600101, 0x0)
	runtime/malloc.go:1177 +0x92
runtime.mallocgc.func1()
	runtime/malloc.go:1071 +0x46
runtime.systemstack(0x0)
	runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	runtime/proc.go:1116

goroutine 69 [running]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:330 fp=0xc005b741e0 sp=0xc005b741d8 pc=0x46de80
runtime.mallocgc(0x400000, 0x1741060, 0xc00f8f1301, 0x0)
	runtime/malloc.go:1070 +0x938 fp=0xc005b74280 sp=0xc005b741e0 pc=0x40df18
runtime.makeslice(0x1741060, 0x400000, 0x400000, 0xc00f8f1200)
	runtime/slice.go:98 +0x6c fp=0xc005b742b0 sp=0xc005b74280 pc=0x44eeac
github.com/Azure/azure-storage-blob-go/azblob.copyFromReader.func1(0xc0106e4448, 0x3)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:48 +0x45 fp=0xc005b742e8 sp=0xc005b742b0 pc=0xa4de45
sync.(*Pool).Get(0xc0106e4448, 0x0, 0x0)
	sync/pool.go:148 +0xaa fp=0xc005b74330 sp=0xc005b742e8 pc=0x47c0ca
github.com/Azure/azure-storage-blob-go/azblob.(*copier).sendChunk(0xc0106e4300, 0x1b50a08, 0xc0106e4300)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:133 +0x72 fp=0xc005b743d8 sp=0xc005b74330 pc=0x9c4332
github.com/Azure/azure-storage-blob-go/azblob.copyFromReader(0x1d84300, 0xc0036812c0, 0x1d4b8e0, 0xc010736a80, 0x1d63a80, 0xc01064ca00, 0x400000, 0x4, 0x1abbd83, 0x18, ...)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:62 +0x2d4 fp=0xc005b74468 sp=0xc005b743d8 pc=0x9c3fb4
github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob(0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc010736a80, 0xc00031600a, 0x5, 0x0, 0x0, 0x0, 0xc000316012, ...)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/highlevel.go:384 +0x11c fp=0xc005b746e0 sp=0xc005b74468 pc=0x9c539c
github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1(0xc000094298, 0x4cf0de0ef1cb4c2b, 0x7fec7e3f0c6247c3)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1494 +0x65b fp=0xc005b757a0 sp=0xc005b746e0 pc=0xaaebfb
github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc01088ccc0, 0xc005b758a0, 0xc005b758a0, 0x40e2b8)
	github.com/rclone/rclone/fs/fs.go:1477 +0x3c fp=0xc005b75850 sp=0xc005b757a0 pc=0x79bc3c
github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0000ba660, 0xc01088ccc0, 0x1, 0xc00f695650, 0x1abbd83)
	github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x90 fp=0xc005b758b0 sp=0xc005b75850 pc=0x784070
github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...)
	github.com/rclone/rclone/lib/pacer/pacer.go:225
github.com/rclone/rclone/backend/azureblob.(*Object).Update(0xc00f695650, 0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc010736a80, 0x1d94640, 0xc001145c00, 0xc0108a3560, 0x1, 0x1, ...)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1487 +0x52d fp=0xc005b75bb0 sp=0xc005b758b0 pc=0xaaaacd
github.com/rclone/rclone/backend/azureblob.(*Fs).Put(0xc0005d87e0, 0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc010736a80, 0x1d94640, 0xc001145c00, 0xc0108a3560, 0x1, 0x1, ...)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:858 +0x11e fp=0xc005b75c38 sp=0xc005b75bb0 pc=0xaa5dfe
github.com/rclone/rclone/fs/operations.Copy(0x1d84300, 0xc000182000, 0x1d99da0, 0xc0005d87e0, 0x0, 0x0, 0xc0024b4180, 0x55, 0x1d9b8a0, 0xc001145c00, ...)
	github.com/rclone/rclone/fs/operations/operations.go:439 +0x182f fp=0xc005b75ee8 sp=0xc005b75c38 pc=0xb03f0f
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc00077c000, 0x1d84300, 0xc000182000, 0xc000768190, 0x1d99da0, 0xc0005d87e0, 0x4b, 0xc00077c0d8)
	github.com/rclone/rclone/fs/sync/sync.go:374 +0x22c fp=0xc005b75fa0 sp=0xc005b75ee8 pc=0x14dc6ac
runtime.goexit()
	runtime/asm_amd64.s:1374 +0x1 fp=0xc005b75fa8 sp=0xc005b75fa0 pc=0x46fac1
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:401 +0xd2

goroutine 1 [semacquire, 51 minutes]:
sync.runtime_Semacquire(0xc0007620e8)
	runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0007620e0)
	sync/waitgroup.go:130 +0x65
github.com/rclone/rclone/fs/march.(*March).Run(0xc00031e7e0, 0xc00031e7e0, 0x1d905c0)
	github.com/rclone/rclone/fs/march/march.go:204 +0x317
github.com/rclone/rclone/fs/sync.(*syncCopyMove).run(0xc00077c000, 0xc0000b6020, 0x1d99da0)
	github.com/rclone/rclone/fs/sync/sync.go:822 +0x215
github.com/rclone/rclone/fs/sync.runSyncCopyMove(0x1d84340, 0xc0000b6020, 0x1d99da0, 0xc0005d87e0, 0x1d9b820, 0xc0001e6000, 0xc000000003, 0xc000762080, 0xc000770060)
	github.com/rclone/rclone/fs/sync/sync.go:1055 +0x27c
github.com/rclone/rclone/fs/sync.Sync(...)
	github.com/rclone/rclone/fs/sync/sync.go:1060
github.com/rclone/rclone/cmd/sync.glob..func1.1(0xc00075e280, 0xc00076c12b)
	github.com/rclone/rclone/cmd/sync/sync.go:55 +0x8f
github.com/rclone/rclone/cmd.Run(0xc000350101, 0x27dde20, 0xc0005b9d40)
	github.com/rclone/rclone/cmd/cmd.go:246 +0xcb
github.com/rclone/rclone/cmd/sync.glob..func1(0x27dde20, 0xc000352380, 0x2, 0xe)
	github.com/rclone/rclone/cmd/sync/sync.go:53 +0x12a
github.com/spf13/cobra.(*Command).execute(0x27dde20, 0xc0003522a0, 0xe, 0xe, 0x27dde20, 0xc0003522a0)
	github.com/spf13/cobra@v1.0.0/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0x27d23c0, 0x60096fb7, 0x27ee140, 0xc000094058)
	github.com/spf13/cobra@v1.0.0/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.0.0/command.go:887
github.com/rclone/rclone/cmd.Main()
	github.com/rclone/rclone/cmd/cmd.go:518 +0x9a
main.main()
	github.com/rclone/rclone/rclone.go:14 +0x25

goroutine 5 [select, 1 minutes]:
go.opencensus.io/stats/view.(*worker).start(0xc000184580)
	go.opencensus.io@v0.22.4/stats/view/worker.go:276 +0x105
created by go.opencensus.io/stats/view.init.0
	go.opencensus.io@v0.22.4/stats/view/worker.go:34 +0x68

goroutine 49 [syscall, 51 minutes]:
os/signal.signal_recv(0x0)
	runtime/sigqueue.go:147 +0x9d
os/signal.loop()
	os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
	os/signal/signal.go:150 +0x45

goroutine 50 [chan receive, 51 minutes]:
github.com/rclone/rclone/fs/accounting.startSignalHandler.func1(0xc00007b0e0)
	github.com/rclone/rclone/fs/accounting/accounting_unix.go:24 +0x38
created by github.com/rclone/rclone/fs/accounting.startSignalHandler
	github.com/rclone/rclone/fs/accounting/accounting_unix.go:21 +0xab

goroutine 2489 [select]:
github.com/rclone/rclone/fs/march.(*March).Run.func1.1(0xc016f80000, 0x1c086, 0x1d76d, 0xc00031e7e0, 0xc0007620e0, 0xc000206480)
	github.com/rclone/rclone/fs/march/march.go:173 +0x1b9
created by github.com/rclone/rclone/fs/march.(*March).Run.func1
	github.com/rclone/rclone/fs/march/march.go:169 +0x327

goroutine 57 [runnable]:
time.Now(0x1899100, 0xc01097c000, 0x1)
	time/time.go:1066 +0xb2
github.com/rclone/rclone/fs/accounting.(*StatsInfo).totalDuration(0xc000198000, 0x49ff32)
	github.com/rclone/rclone/fs/accounting/stats.go:171 +0xd1
github.com/rclone/rclone/fs/accounting.(*StatsInfo).String(0xc000198000, 0xc, 0xc000198000)
	github.com/rclone/rclone/fs/accounting/stats.go:232 +0x110
github.com/rclone/rclone/cmd.printProgress(0x0, 0x0)
	github.com/rclone/rclone/cmd/progress.go:79 +0xd5
github.com/rclone/rclone/cmd.startProgress.func2(0xc000762080, 0xc000770060, 0x1b524e0)
	github.com/rclone/rclone/cmd/progress.go:50 +0xaa
created by github.com/rclone/rclone/cmd.startProgress
	github.com/rclone/rclone/cmd/progress.go:40 +0xbf

goroutine 58 [select, 8 minutes]:
github.com/rclone/rclone/fs/sync.(*pipe).GetMax(0xc000768140, 0x1d84300, 0xc000182000, 0x0, 0x1d9b8a0, 0xc002a48bd0, 0x1d9b800, 0xc002a48bd0, 0x0)
	github.com/rclone/rclone/fs/sync/pipe.go:124 +0xe5
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc00077c000, 0xc000768140, 0xc000768190, 0x0, 0xc00077c0c0)
	github.com/rclone/rclone/fs/sync/sync.go:290 +0xb6
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startCheckers
	github.com/rclone/rclone/fs/sync/sync.go:385 +0xbb

goroutine 59 [select, 8 minutes]:
github.com/rclone/rclone/fs/sync.(*pipe).GetMax(0xc000768140, 0x1d84300, 0xc000182000, 0xc, 0x1d9b8a0, 0xc002a48e70, 0x1d9b800, 0xc002a48e70, 0x0)
	github.com/rclone/rclone/fs/sync/pipe.go:124 +0xe5
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc00077c000, 0xc000768140, 0xc000768190, 0xc, 0xc00077c0c0)
	github.com/rclone/rclone/fs/sync/sync.go:290 +0xb6
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startCheckers
	github.com/rclone/rclone/fs/sync/sync.go:385 +0xbb

goroutine 60 [select, 8 minutes]:
github.com/rclone/rclone/fs/sync.(*pipe).GetMax(0xc000768140, 0x1d84300, 0xc000182000, 0x19, 0x1d9b8a0, 0xc002a49110, 0x1d9b800, 0xc002a49110, 0x0)
	github.com/rclone/rclone/fs/sync/pipe.go:124 +0xe5
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc00077c000, 0xc000768140, 0xc000768190, 0x19, 0xc00077c0c0)
	github.com/rclone/rclone/fs/sync/sync.go:290 +0xb6
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startCheckers
	github.com/rclone/rclone/fs/sync/sync.go:385 +0xbb

goroutine 61 [select, 8 minutes]:
github.com/rclone/rclone/fs/sync.(*pipe).GetMax(0xc000768140, 0x1d84300, 0xc000182000, 0x25, 0x1d9b8a0, 0xc002a49730, 0x1d9b800, 0xc002a49730, 0x0)
	github.com/rclone/rclone/fs/sync/pipe.go:124 +0xe5
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc00077c000, 0xc000768140, 0xc000768190, 0x25, 0xc00077c0c0)
	github.com/rclone/rclone/fs/sync/sync.go:290 +0xb6
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startCheckers
	github.com/rclone/rclone/fs/sync/sync.go:385 +0xbb

goroutine 62 [select, 8 minutes]:
github.com/rclone/rclone/fs/sync.(*pipe).GetMax(0xc000768140, 0x1d84300, 0xc000182000, 0x32, 0x1d9b8a0, 0xc002a48d90, 0x1d9b800, 0xc002a48d90, 0x0)
	github.com/rclone/rclone/fs/sync/pipe.go:124 +0xe5
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc00077c000, 0xc000768140, 0xc000768190, 0x32, 0xc00077c0c0)
	github.com/rclone/rclone/fs/sync/sync.go:290 +0xb6
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startCheckers
	github.com/rclone/rclone/fs/sync/sync.go:385 +0xbb

goroutine 63 [select, 8 minutes]:
github.com/rclone/rclone/fs/sync.(*pipe).GetMax(0xc000768140, 0x1d84300, 0xc000182000, 0x3e, 0x1d9b8a0, 0xc002a49570, 0x1d9b800, 0xc002a49570, 0x0)
	github.com/rclone/rclone/fs/sync/pipe.go:124 +0xe5
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairChecker(0xc00077c000, 0xc000768140, 0xc000768190, 0x3e, 0xc00077c0c0)
	github.com/rclone/rclone/fs/sync/sync.go:290 +0xb6
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startCheckers
	github.com/rclone/rclone/fs/sync/sync.go:385 +0xbb
	github.com/rclone/rclone/fs/fs.go:1477 +0x3c
github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0000ba660, 0xc01088cea0, 0x1, 0xc00f74db90, 0x1abbd83)
	github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x90
github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...)
	github.com/rclone/rclone/lib/pacer/pacer.go:225
github.com/rclone/rclone/backend/azureblob.(*Object).Update(0xc00f74db90, 0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc010736b60, 0x1d94640, 0xc0011c1dc0, 0xc0108056f0, 0x1, 0x1, ...)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1487 +0x52d
github.com/rclone/rclone/backend/azureblob.(*Fs).Put(0xc0005d87e0, 0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc010736b60, 0x1d94640, 0xc0011c1dc0, 0xc0108056f0, 0x1, 0x1, ...)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:858 +0x11e
github.com/rclone/rclone/fs/operations.Copy(0x1d84300, 0xc000182000, 0x1d99da0, 0xc0005d87e0, 0x0, 0x0, 0xc003989b80, 0x31, 0x1d9b8a0, 0xc0011c1dc0, ...)
	github.com/rclone/rclone/fs/operations/operations.go:439 +0x182f
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc00077c000, 0x1d84300, 0xc000182000, 0xc000768190, 0x1d99da0, 0xc0005d87e0, 0x0, 0xc00077c0d8)
	github.com/rclone/rclone/fs/sync/sync.go:374 +0x22c
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:401 +0xd2

goroutine 67 [runnable]:
github.com/rclone/rclone/fs/accounting.(*transferMap).del(0xc00007e300, 0xc002383920, 0x59)
	github.com/rclone/rclone/fs/accounting/transfermap.go:36
github.com/rclone/rclone/fs/accounting.(*StatsInfo).DoneTransferring(0xc000198000, 0xc002383920, 0x59, 0xd65757274203a01)
	github.com/rclone/rclone/fs/accounting/stats.go:574 +0x46
github.com/rclone/rclone/fs/accounting.(*Transfer).Done(0xc01080a510, 0x0, 0x0)
	github.com/rclone/rclone/fs/accounting/transfer.go:120 +0x146
github.com/rclone/rclone/fs/operations.Copy.func1(0xc01080a510, 0xc00d8c7f48)
	github.com/rclone/rclone/fs/operations/operations.go:348 +0x45
github.com/rclone/rclone/fs/operations.Copy(0x1d84300, 0xc000182000, 0x1d99da0, 0xc0005d87e0, 0x0, 0xc0107383f0, 0xc002383920, 0x59, 0x1d9b8a0, 0xc001031570, ...)
	github.com/rclone/rclone/fs/operations/operations.go:492 +0x77b
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc00077c000, 0x1d84300, 0xc000182000, 0xc000768190, 0x1d99da0, 0xc0005d87e0, 0x19, 0xc00077c0d8)
	github.com/rclone/rclone/fs/sync/sync.go:374 +0x22c
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:401 +0xd2

goroutine 68 [runnable]:
github.com/Azure/azure-storage-blob-go/azblob.(*copier).close(0xc00dcbfb00, 0xc000096030, 0x1d4a500)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:204 +0x195
github.com/Azure/azure-storage-blob-go/azblob.copyFromReader(0x1d84300, 0xc0036967c0, 0x1d4b8e0, 0xc01079c0e0, 0x1d63a80, 0xc0103e2c80, 0x400000, 0x4, 0x1abbd83, 0x18, ...)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:72 +0x336
github.com/Azure/azure-storage-blob-go/azblob.UploadStreamToBlockBlob(0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc01079c0e0, 0xc00031600a, 0x5, 0x0, 0x0, 0x0, 0xc000316012, ...)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/highlevel.go:384 +0x11c
github.com/rclone/rclone/backend/azureblob.(*Object).Update.func1(0xc000094298, 0x0, 0x0)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1494 +0x65b
github.com/rclone/rclone/fs.pacerInvoker(0x1, 0x1, 0xc0107910e0, 0xc008a178a0, 0xc008a178a0, 0x40e2b8)
	github.com/rclone/rclone/fs/fs.go:1477 +0x3c
github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc0000ba660, 0xc0107910e0, 0x1, 0xc00f74d9d0, 0x1abbd83)
	github.com/rclone/rclone/lib/pacer/pacer.go:197 +0x90
github.com/rclone/rclone/lib/pacer.(*Pacer).CallNoRetry(...)
	github.com/rclone/rclone/lib/pacer/pacer.go:225
github.com/rclone/rclone/backend/azureblob.(*Object).Update(0xc00f74d9d0, 0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc01079c0e0, 0x1d94640, 0xc0010315e0, 0xc0105eda40, 0x1, 0x1, ...)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:1487 +0x52d
github.com/rclone/rclone/backend/azureblob.(*Fs).Put(0xc0005d87e0, 0x1d84300, 0xc000182000, 0x1d4b8e0, 0xc01079c0e0, 0x1d94640, 0xc0010315e0, 0xc0105eda40, 0x1, 0x1, ...)
	github.com/rclone/rclone/backend/azureblob/azureblob.go:858 +0x11e
github.com/rclone/rclone/fs/operations.Copy(0x1d84300, 0xc000182000, 0x1d99da0, 0xc0005d87e0, 0x0, 0x0, 0xc0023fd140, 0x5a, 0x1d9b8a0, 0xc0010315e0, ...)
	github.com/rclone/rclone/fs/operations/operations.go:439 +0x182f
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc00077c000, 0x1d84300, 0xc000182000, 0xc000768190, 0x1d99da0, 0xc0005d87e0, 0x32, 0xc00077c0d8)
	github.com/rclone/rclone/fs/sync/sync.go:374 +0x22c
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
	github.com/rclone/rclone/fs/sync/sync.go:401 +0xd2

goroutine 70 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc0022358f0, 0x0, 0x0, 0xc0022358f0)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc0022358f0, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc002870e40, 0x4d, 0xc002870e40, 0x4d, 0x7ffffffb, 0x0, 0x100, 0x0, 0x0, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 71 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc00224ca10, 0x0, 0x0, 0xc00224ca10)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc00224ca10, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc0028e96e0, 0x24, 0xc0028e96e0, 0x24, 0x7ffffffc, 0x0, 0x100, 0x0, 0x0, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 72 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc00224c7e0, 0x0, 0x0, 0xc00224c7e0)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc00224c7e0, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc002df9320, 0x24, 0xc002df9320, 0x24, 0x7ffffffc, 0x0, 0x100, 0x0, 0x0, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 73 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc00224c8c0, 0x0, 0x0, 0xc00224c8c0)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc00224c8c0, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc0028e9560, 0x24, 0xc0028e9560, 0x24, 0x7ffffffc, 0x0, 0x100, 0xc00f74d180, 0x2, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 74 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc002235a40, 0x0, 0x0, 0xc002235a40)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc002235a40, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc0028712c0, 0x4d, 0xc0028712c0, 0x4d, 0x7ffffffb, 0x0, 0x100, 0x0, 0x0, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 75 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc002235d50, 0x0, 0x0, 0xc002235d50)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc002235d50, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc002871a40, 0x4d, 0xc002871a40, 0x4d, 0x7ffffffb, 0x0, 0x100, 0xc003635880, 0x1, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 76 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc002235b90, 0x0, 0x0, 0xc002235b90)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc002235b90, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc0028715c0, 0x4d, 0xc0028715c0, 0x4d, 0x7ffffffb, 0x0, 0x100, 0x0, 0x0, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 77 [select]:
github.com/rclone/rclone/fs/sync.(*pipe).Put(0xc000768190, 0x1d84300, 0xc000182000, 0x1d9b8a0, 0xc002235c00, 0x0, 0x0, 0xc002235c00)
	github.com/rclone/rclone/fs/sync/pipe.go:105 +0x1ec
github.com/rclone/rclone/fs/sync.(*syncCopyMove).SrcOnly(0xc00077c000, 0x7f106aa896c0, 0xc002235c00, 0x0)
	github.com/rclone/rclone/fs/sync/sync.go:955 +0x325
github.com/rclone/rclone/fs/march.(*March).processJob(0xc00031e7e0, 0xc002871740, 0x4d, 0xc002871740, 0x4d, 0x7ffffffb, 0x0, 0x100, 0x0, 0x0, ...)
	github.com/rclone/rclone/fs/march/march.go:432 +0x35c
github.com/rclone/rclone/fs/march.(*March).Run.func1(0xc0007620d0, 0xc00031e7e0, 0xc000206480, 0xc0007620c0, 0xc0007603b0, 0xc0007620c8, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:157 +0x217
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:147 +0x1d9

goroutine 78 [chan receive, 51 minutes]:
github.com/rclone/rclone/fs/march.(*March).Run.func2(0xc00031e7e0, 0xc000206480, 0xc0007620e0)
	github.com/rclone/rclone/fs/march/march.go:199 +0x4e
created by github.com/rclone/rclone/fs/march.(*March).Run
	github.com/rclone/rclone/fs/march/march.go:197 +0x306

goroutine 11820 [IO wait]:
internal/poll.runtime_pollWait(0x7f106ad587e8, 0x72, 0x1d50fa0)
	runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc005b16c18, 0x72, 0xc00819c000, 0x1fb0, 0x1fb0)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005b16c00, 0xc00819c000, 0x1fb0, 0x1fb0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc005b16c00, 0xc00819c000, 0x1fb0, 0x1fb0, 0x1fa3, 0x3f3d1bdf08001042, 0x41028c02882bdb6b)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0002081b8, 0xc00819c000, 0x1fb0, 0x1fb0, 0x0, 0x0, 0x0)
	net/net.go:182 +0x8e
github.com/rclone/rclone/fs/fshttp.(*timeoutConn).readOrWrite(0xc007f6e040, 0xc0083d5580, 0xc00819c000, 0x1fb0, 0x1fb0, 0x203001, 0x67a95b, 0xc007f684e0)
	github.com/rclone/rclone/fs/fshttp/http.go:75 +0x48
github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc007f6e040, 0xc00819c000, 0x1fb0, 0x1fb0, 0xc007f68528, 0xd, 0x10)
	github.com/rclone/rclone/fs/fshttp/http.go:87 +0x8a
crypto/tls.(*atLeastReader).Read(0xc006971dc0, 0xc00819c000, 0x1fb0, 0x1fb0, 0x2d0, 0x1fa3, 0xc0083d5668)
	crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc007f68600, 0x1d4a2a0, 0xc006971dc0, 0x40b665, 0x181ee40, 0x19898e0)
	bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc007f68380, 0x7f106acd85e8, 0xc007f6e040, 0x5, 0xc007f6e040, 0x2c0)
	crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc007f68380, 0x0, 0x0, 0xc0106aaa80)
	crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
	crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc007f68380, 0xc0081f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	crypto/tls/conn.go:1252 +0x15f
net/http.(*persistConn).Read(0xc0074925a0, 0xc0081f8000, 0x1000, 0x1000, 0xc0083d5eb0, 0x4680e0, 0xc0083d5eb0)
	net/http/transport.go:1887 +0x77
bufio.(*Reader).fill(0xc001df3c20)
	bufio/bufio.go:101 +0x105
bufio.(*Reader).Peek(0xc001df3c20, 0x1, 0x2, 0x0, 0x0, 0x0, 0xc0106aaa20)
	bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0074925a0)
	net/http/transport.go:2040 +0x1a8
created by net/http.(*Transport).dialConn
	net/http/transport.go:1708 +0xcb7

goroutine 11873 [IO wait]:
internal/poll.runtime_pollWait(0x7f106ad589a8, 0x72, 0x1d50fa0)
	runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc005b16998, 0x72, 0xc007fc8000, 0x1fb0, 0x1fb0)
	internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc005b16980, 0xc007fc8000, 0x1fb0, 0x1fb0, 0x0, 0x0, 0x0)
	internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc005b16980, 0xc007fc8000, 0x1fb0, 0x1fb0, 0x1fa3, 0x8825d4bc8f19bf6b, 0xf7c751576db69a2f)
	net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0002081b0, 0xc007fc8000, 0x1fb0, 0x1fb0, 0x0, 0x0, 0x0)
	net/net.go:182 +0x8e
github.com/rclone/rclone/fs/fshttp.(*timeoutConn).readOrWrite(0xc007f59fc0, 0xc005b67580, 0xc007fc8000, 0x1fb0, 0x1fb0, 0x203002, 0x7f106aabe7d8, 0xbf)
	github.com/rclone/rclone/fs/fshttp/http.go:75 +0x48
github.com/rclone/rclone/fs/fshttp.(*timeoutConn).Read(0xc007f59fc0, 0xc007fc8000, 0x1fb0, 0x1fb0, 0xc007f68100, 0xd, 0x10)
	github.com/rclone/rclone/fs/fshttp/http.go:87 +0x8a
crypto/tls.(*atLeastReader).Read(0xc00b2f57e0, 0xc007fc8000, 0x1fb0, 0x1fb0, 0x16c, 0x1fa3, 0xc005b67668)
	crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc007f68280, 0x1d4a2a0, 0xc00b2f57e0, 0x40b665, 0x181ee40, 0x19898e0)
	bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc007f68000, 0x7f106acd85e8, 0xc007f59fc0, 0x5, 0xc007f59fc0, 0x15c)
	crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc007f68000, 0x0, 0x0, 0xc0108127e0)
	crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
	crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc007f68000, 0xc00811d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	crypto/tls/conn.go:1252 +0x15f
net/http.(*persistConn).Read(0xc0075386c0, 0xc00811d000, 0x1000, 0x1000, 0xc005b67eb0, 0x4680e0, 0xc005b67eb0)
	net/http/transport.go:1887 +0x77
bufio.(*Reader).fill(0xc000e97bc0)
	bufio/bufio.go:101 +0x105
bufio.(*Reader).Peek(0xc000e97bc0, 0x1, 0x2, 0x0, 0x0, 0x0, 0xc010812780)
	bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0075386c0)
	net/http/transport.go:2040 +0x1a8
created by net/http.(*Transport).dialConn
	net/http/transport.go:1708 +0xcb7

goroutine 72579 [chan receive]:
github.com/Azure/azure-storage-blob-go/azblob.(*copier).writer(0xc0106e4480)
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:165 +0xb4
created by github.com/Azure/azure-storage-blob-go/azblob.copyFromReader
	github.com/Azure/azure-storage-blob-go@v0.10.0/azblob/chunkwriting.go:56 +0x2b1

goroutine 11874 [select]:
net/http.(*persistConn).writeLoop(0xc0075386c0)
	net/http/transport.go:2340 +0x11c
created by net/http.(*Transport).dialConn
	net/http/transport.go:1709 +0xcdc

goroutine 5230 [select]:
net/http.(*persistConn).readLoop(0xc007538000)
	net/http/transport.go:2161 +0x9cc
created by net/http.(*Transport).dialConn
	net/http/transport.go:1708 +0xcb7
2

With no log file, there is zero way to figure out what is going on.

3

@Animosity022, updated with log

4

I'm a bit confused.

Did it error below or no error?

That means you ran out of memory on the server are you running on. You'd need to add memory, reduce transfers, reduce checkers, reduce buffer-size, etc. What are the specs on what you are running on?

5

Trying to transfer 90 days data with max-age filter.

Total objects: 454461
Total size: 377.267 GBytes (405087759260 Bytes)

System Configuration

#  lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 62
Model name:            Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
Stepping:              4
CPU MHz:               2699.999
BogoMIPS:              5399.99
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase tsc_adjust smep arat
root@apvrs79341:/root
#  cat /proc/meminfo
MemTotal:        3880276 kB
MemFree:         2958588 kB
MemAvailable:    3272240 kB
Buffers:           21104 kB
Cached:           471964 kB
SwapCached:         9596 kB
Active:           278016 kB
Inactive:         347976 kB
Active(anon):      30756 kB
Inactive(anon):   102404 kB
Active(file):     247260 kB
Inactive(file):   245572 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       2097148 kB
SwapFree:        1989160 kB
Dirty:                40 kB
Writeback:             0 kB
AnonPages:        130988 kB
Mapped:            39984 kB
Shmem:               216 kB
Slab:              88288 kB
SReclaimable:      51040 kB
SUnreclaim:        37248 kB
KernelStack:        7200 kB
PageTables:         9740 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4037284 kB
Committed_AS:     993088 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      162160 kB
VmallocChunk:   34359341052 kB
Percpu:              848 kB
HardwareCorrupted:     0 kB
AnonHugePages:      6144 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       81856 kB
DirectMap2M:     4112384 kB

Command Used

rclone sync remote_prd:bkt-prd/ azure_prd:customfiles/prd --log-file=rclone_detail_prd.log --log-level DEBUG --max-age 90d --delete-excluded --use-server-modtime  --fast-list --azureblob-disable-checksum  --bwlimit 80M --progress > 90Sync.txt &

Is there any way to derive optimal rclone command.?

6

So the error was that you ran out of memory?

You can reduce:

      --checkers int                         Number of checkers to run in parallel. (default 8)
     --transfers int                        Number of file transfers to run in parallel. (default 4)
--buffer-size SizeSuffix               In memory buffer size when reading files for each --transfer. (default 16M)

You can try using

      --use-mmap                             Use mmap allocator (see docs).

A little over 3GB memory machine is pretty tiny and I don't know what else you are running on it.

The goal would be to start small and increased based on whatever is running on your system.

7

might want to remove
https://rclone.org/docs/#fast-list
" It will use more memory. Rclone has to load the whole listing into memory."

8

I think this might be the azure SDK memory leak we've been having problems with

Try the beta in this post

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.