File silently missing after sync (fails with IDrive, works locally)

Help and Support
1

What is the problem you are having with rclone?

File appears to upload using rclone sync, but is not present afterwards. Repeatable. Works if the remote is a local drive, but fails in IDrive S3 cloud.
Is this an IDrive issue or my config?

Using encryption so unfortunately not so very readable but I reduced to two levels of sub-dirs.
Additional info: Afterwards,
folder r066bcq5jspunrvodla8eeb9p0 exists but is empty
folder ujd6heie8l38rofqk7fo0hg41s does NOT exist

Side notes (not the main problem):

  • File is ~200MB. Was surprised it still uploaded in chunks (supposedly can upload 5GB in S3?!) Therefore my attempt to 'force' via command line options.
  • Often, there are also notifications in the logs about a "operation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: ... api error NoSuchKey: The specified key does not exist." I am posting a clean log file first (file is still missing after sync ie problem is still there) and then a second log with this error, in case it helps !

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

  • os/version: Microsoft Windows 10 Home 22H2 (64 bit)
  • os/kernel: 10.0.19045.5131 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.23.1
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

IDrive S3 storage (fails), local drive (works)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync "Z:\Captures" "compressed:Backups/Captures" --multi-thread-cutoff=500M --s3-chunk-size=20M -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[compressed]
type = compress
remote = trezor:

[idrive-remote]
type = s3
provider = IDrive
env_auth = false
access_key_id = XXX
secret_access_key = XXX
acl = private
endpoint = w3x8.ca.idrivee2-26.com
server_side_encryption = aws:kms

[trezor]
type = crypt
remote = idrive-remote:bucket
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX
server_side_across_configs = false
no_data_encryption = false

A log from the command that you were trying to run with the -vv flag

2024/11/20 06:51:01 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "sync" "Z:\\Captures" "compressed:Backups/Captures" "--multi-thread-cutoff=500M" "--s3-chunk-size=20M" "-vv"]
2024/11/20 06:51:01 DEBUG : Creating backend with remote "Z:\\Captures"
2024/11/20 06:51:01 DEBUG : Using config file from "c:\\rclone-v1.68.1-windows-amd64\\rclone.conf"
2024/11/20 06:51:01 DEBUG : fs cache: renaming cache item "Z:\\Captures" to be canonical "//?/Z:/Captures"
2024/11/20 06:51:01 DEBUG : Creating backend with remote "compressed:Backups/Captures"
2024/11/20 06:51:01 DEBUG : Creating backend with remote "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s"
2024/11/20 06:51:01 DEBUG : idrive-remote: detected overridden config - adding "{Ygqdw}" suffix to name
2024/11/20 06:51:01 DEBUG : fs cache: renaming cache item "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s" to be canonical "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s"
2024/11/20 06:51:01 DEBUG : fs cache: switching user supplied name "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s" for canonical name "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s"
2024/11/20 06:51:01 DEBUG : Creating backend with remote "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0"
2024/11/20 06:51:01 DEBUG : idrive-remote: detected overridden config - adding "{Ygqdw}" suffix to name
2024/11/20 06:51:01 DEBUG : fs cache: renaming cache item "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0" to be canonical "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0"
2024/11/20 06:51:01 DEBUG : fs cache: switching user supplied name "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0" for canonical name "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0"
2024/11/20 06:51:01 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: Need to transfer - File not found at Destination
2024/11/20 06:51:01 DEBUG : Compressed: compressed:Backups/Captures: Waiting for checks to finish
2024/11/20 06:51:01 DEBUG : Compressed: compressed:Backups/Captures: Waiting for transfers to finish
2024/11/20 06:51:01 NOTICE: S3 bucket bucket path e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0: Streaming uploads using chunk size 20Mi will have maximum file size of 195.312Gi
2024/11/20 06:51:01 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: open chunk writer: started multipart upload: 32767.a0d125b2-ebb5-4504-9c84-a98b9f363868
2024/11/20 06:51:02 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 0 size 20Mi offset 0/off
2024/11/20 06:51:02 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 1 size 20Mi offset 20Mi/off
2024/11/20 06:51:02 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 2 size 20Mi offset 40Mi/off
2024/11/20 06:51:02 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 3 size 20Mi offset 60Mi/off
2024/11/20 06:51:11 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 1 with 20971520 bytes and etag "326e0f1607a279c71e97c4a1ac7cb4b3"
2024/11/20 06:51:11 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 4 size 20Mi offset 80Mi/off
2024/11/20 06:51:12 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 2 with 20971520 bytes and etag "169cf5e4f91fa73d2d4e771717ea823e"
2024/11/20 06:51:12 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 5 size 20Mi offset 100Mi/off
2024/11/20 06:51:15 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 3 with 20971520 bytes and etag "638c88e27f11623503eba9d58b8304d6"
2024/11/20 06:51:15 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 6 size 20Mi offset 120Mi/off
2024/11/20 06:51:17 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 4 with 20971520 bytes and etag "c04d9186ce9c15b6be79e65982c6eaa0"
2024/11/20 06:51:17 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 7 size 20Mi offset 140Mi/off
2024/11/20 06:51:23 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 5 with 20971520 bytes and etag "0d786d3615c4d0a2119cb00361a357a7"
2024/11/20 06:51:23 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 8 size 20Mi offset 160Mi/off
2024/11/20 06:51:23 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 6 with 20971520 bytes and etag "3fee09096a399d9d1ebef691fbbc3cc1"
2024/11/20 06:51:23 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 9 size 13.974Mi offset 180Mi/off
2024/11/20 06:51:25 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 7 with 20971520 bytes and etag "5290b625cf3634f1d35e70e0d1d7c896"
2024/11/20 06:51:27 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 8 with 20971520 bytes and etag "4ed390a3e6dbf7d27aa5c7cf63a9470a"
2024/11/20 06:51:30 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 10 with 14652603 bytes and etag "d75d8293da438e114ebb8c738622f5b2"
2024/11/20 06:51:31 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 9 with 20971520 bytes and etag "afb7afdfe047ad6903d919799377c9c0"
2024/11/20 06:51:31 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload "32767.a0d125b2-ebb5-4504-9c84-a98b9f363868" finished
2024/11/20 06:51:31 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: md5 = cbeb1d754e49b8fe1d6665d077ac73e2 OK
2024/11/20 06:51:31 INFO  : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: Copied (new)
2024/11/20 06:51:31 DEBUG : Waiting for deletions to finish
2024/11/20 06:51:31 INFO  :
Transferred:      193.926 MiB / 193.926 MiB, 100%, 5.708 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        29.6s

2024/11/20 06:51:31 DEBUG : 15 go routines active

And here is a second log file, with some "S3: AbortMultipartUpload" errors. The problem (the file is not present after upload) happens in either case. Posting this just in case.

2024/11/20 07:07:05 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "sync" "Z:\\Captures" "compressed:Backups/Captures/" "--multi-thread-cutoff=500M" "--s3-chunk-size=20M" "-vv"]
2024/11/20 07:07:05 DEBUG : Creating backend with remote "Z:\\Captures"
2024/11/20 07:07:05 DEBUG : Using config file from "c:\\rclone-v1.68.1-windows-amd64\\rclone.conf"
2024/11/20 07:07:05 DEBUG : fs cache: renaming cache item "Z:\\Captures" to be canonical "//?/Z:/Captures"
2024/11/20 07:07:05 DEBUG : Creating backend with remote "compressed:Backups/Captures/"
2024/11/20 07:07:05 DEBUG : Creating backend with remote "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s"
2024/11/20 07:07:05 DEBUG : idrive-remote: detected overridden config - adding "{Ygqdw}" suffix to name
2024/11/20 07:07:05 DEBUG : fs cache: renaming cache item "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s" to be canonical "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s"
2024/11/20 07:07:05 DEBUG : fs cache: switching user supplied name "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s" for canonical name "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/ujd6heie8l38rofqk7fo0hg41s"
2024/11/20 07:07:05 DEBUG : Creating backend with remote "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0"
2024/11/20 07:07:05 DEBUG : idrive-remote: detected overridden config - adding "{Ygqdw}" suffix to name
2024/11/20 07:07:05 DEBUG : fs cache: renaming cache item "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0" to be canonical "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0"
2024/11/20 07:07:05 DEBUG : fs cache: switching user supplied name "idrive-remote:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0" for canonical name "idrive-remote{Ygqdw}:bucket/e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0"
2024/11/20 07:07:05 DEBUG : fs cache: renaming cache item "compressed:Backups/Captures/" to be canonical "compressed:Backups/Captures"
2024/11/20 07:07:05 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: Need to transfer - File not found at Destination
2024/11/20 07:07:05 DEBUG : Compressed: compressed:Backups/Captures: Waiting for checks to finish
2024/11/20 07:07:05 DEBUG : Compressed: compressed:Backups/Captures: Waiting for transfers to finish
2024/11/20 07:07:06 NOTICE: S3 bucket bucket path e8kg0lug98i2q4rl6df4uv2sd0/r066bcq5jspunrvodla8eeb9p0: Streaming uploads using chunk size 20Mi will have maximum file size of 195.312Gi
2024/11/20 07:07:06 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: open chunk writer: started multipart upload: 32767.ac72cda6-9c91-4cb8-81ce-115e0b65539b
2024/11/20 07:07:06 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 0 size 20Mi offset 0/off
2024/11/20 07:07:06 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 1 size 20Mi offset 20Mi/off
2024/11/20 07:07:06 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: Cancelling multipart upload
2024/11/20 07:07:06 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: Failed to cancel multipart upload: failed to abort multipart upload "32767.ac72cda6-9c91-4cb8-81ce-115e0b65539b": operation error S3: AbortMultipartUpload, https response error StatusCode: 404, RequestID: 1809ABCCE27E60DF, HostID: c1a90093-2a25-4d71-907d-fe8a0138bef3, api error NoSuchKey: The specified key does not exist.
2024/11/20 07:07:06 ERROR : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: Failed to copy: failed to upload chunk 1 with 20971520 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: 1809ABCCD62227E8, HostID: c1a90093-2a25-4d71-907d-fe8a0138bef3, api error NoSuchKey: The specified key does not exist.
2024/11/20 07:07:06 ERROR : Compressed: compressed:Backups/Captures: not deleting files as there were IO errors
2024/11/20 07:07:06 ERROR : Compressed: compressed:Backups/Captures: not deleting directories as there were IO errors
2024/11/20 07:07:06 ERROR : Attempt 1/3 failed with 1 errors and: failed to upload chunk 1 with 20971520 bytes: operation error S3: UploadPart, https response error StatusCode: 404, RequestID: 1809ABCCD62227E8, HostID: c1a90093-2a25-4d71-907d-fe8a0138bef3, api error NoSuchKey: The specified key does not exist.
2024/11/20 07:07:06 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: Need to transfer - File not found at Destination
2024/11/20 07:07:06 DEBUG : Compressed: compressed:Backups/Captures: Waiting for checks to finish
2024/11/20 07:07:06 DEBUG : Compressed: compressed:Backups/Captures: Waiting for transfers to finish
2024/11/20 07:07:06 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: open chunk writer: started multipart upload: 32767.6d18bab5-1dff-4e50-82f2-bd2265f76412
2024/11/20 07:07:06 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 0 size 20Mi offset 0/off
2024/11/20 07:07:07 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 1 size 20Mi offset 20Mi/off
2024/11/20 07:07:07 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 2 size 20Mi offset 40Mi/off
2024/11/20 07:07:07 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 3 size 20Mi offset 60Mi/off
2024/11/20 07:07:19 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 2 with 20971520 bytes and etag "2b40d47b11b7f878d446c5c0a499905e"
2024/11/20 07:07:20 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 4 size 20Mi offset 80Mi/off
2024/11/20 07:07:20 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 1 with 20971520 bytes and etag "f1f469856815ba577ba7e702a83c381e"
2024/11/20 07:07:20 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 5 size 20Mi offset 100Mi/off
2024/11/20 07:07:20 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 4 with 20971520 bytes and etag "4203e054f3ee30a4a97754464ae0f413"
2024/11/20 07:07:20 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 6 size 20Mi offset 120Mi/off
2024/11/20 07:07:22 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 3 with 20971520 bytes and etag "f1ef83f76c25293455d359814ac7c916"
2024/11/20 07:07:22 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 7 size 20Mi offset 140Mi/off
2024/11/20 07:07:29 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 7 with 20971520 bytes and etag "31cbbf11b24725adff7861ed1a5c29d6"
2024/11/20 07:07:29 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 8 size 20Mi offset 160Mi/off
2024/11/20 07:07:31 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 6 with 20971520 bytes and etag "409608d4d7e70560d13c51584e158523"
2024/11/20 07:07:31 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4.9YQkDAAAAAA.gz: multipart upload: starting chunk 9 size 13.974Mi offset 180Mi/off
2024/11/20 07:07:31 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 5 with 20971520 bytes and etag "a792e9980383a85336572776db5c25dd"
2024/11/20 07:07:34 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 8 with 20971520 bytes and etag "a349a80aadf9cb1bdf1feb9a002b1250"
2024/11/20 07:07:35 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 10 with 14652603 bytes and etag "5829195660ecbe2a65d2341deb231717"
2024/11/20 07:07:36 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload wrote chunk 9 with 20971520 bytes and etag "442d273a921a73fd6e62b675f9c229e4"
2024/11/20 07:07:37 DEBUG : bd550gmhlcilhmdqt1ksgd86k334u9abn32vsg511m32j9dlr5q39b4rijphch1uq49rpu1b14ujotjnr1a5t54i1bq4e355hk0dnng: multipart upload "32767.6d18bab5-1dff-4e50-82f2-bd2265f76412" finished
2024/11/20 07:07:37 DEBUG : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: md5 = cbeb1d754e49b8fe1d6665d077ac73e2 OK
2024/11/20 07:07:37 INFO  : 20181204_091940 (2024_10_20 14_16_05 UTC).mp4: Copied (new)
2024/11/20 07:07:37 DEBUG : Waiting for deletions to finish
2024/11/20 07:07:37 ERROR : Attempt 2/3 succeeded
2024/11/20 07:07:37 INFO  :
Transferred:      233.926 MiB / 233.926 MiB, 100%, 6.205 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        31.4s

2024/11/20 07:07:37 DEBUG : 17 go routines active
2

First thing I would try if all works without compress remote.

3

I does WORK without compression, locally and on IDrive.

But... Can this be a "solution" ? I mean, for one a lot of other files in my backup set benefit from compression. And, I would think rclone's compression feature should have endless "field" testing by now.
Any suggestions. Should this be a bug ?

4

welcome to the forum,

the remote is experimental, not even alpha or beta. based on forum posts, it is not used much.

what is the exact command that works?

5

Thx. So originally I was using my compressed: remote, which was configured as follows (from the above configuration)

[compressed]
type = compress
remote = trezor:

Now I simply by-passed and used the same command line on the trezor: remote

rclone sync "Z:\Captures" "trezor:Backups/Captures" --multi-thread-cutoff=500M --s3-chunk-size=20M -vv

(It's probably not smart because I have compressed and uncompressed files in the trezor: remote, but was just for testing). Executed the command twice and on the second run rclone found no need to copy the file again)

6

sure, rclone does not re-copy unchanged files.

if you want to narrow down the issue, to see if this is a bug, or if there is a workaround, then

  • test on a single file using, rclone copy --dump=headers
  • create a new bucket
  • test without crypt, something like
[compressed]
type = compress
remote = idrive-remote:newbucket
7

Well, yes, supposedly. But this was the whole problem here: rclone uploaded the file, but for some reason it didn't appear or vanished in the remote. So on second run it would have to copy it again.

Will try your other recommendations now.

8

The good update: Everything worked with a fresh bucket: With and without compression, with and without encryption.
The bad update: In the process I destroyed the original setup, so I can't run any tests for troubleshooting or improvements (if any). The logs above are the only clues left.

I am running a complete rclone sync again. I will append here if it happens again, but so far it hasn't. I am sorry I probably wasted your time. Thanks though.

10

After (accidentally) deleting a few thousand files in the original bucket, the same 'rclone sync' now ran successfully a few times...

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.