Copy to Scality S3 : corrupted on transfer: sizes differ xxx vs 0

Help and Support
1

What is the problem you are having with rclone?

Hi.
I'm having issues uploading files to Scality S3 storage using RClone. The source and destination sizes always differ. Destination size is always 0 as per RClone debug log, but when I browse the bucket, the files are available AND usable.
When it's a small file, despite the different sizes, RClone doesn't throw an error, but when it comes to large files, then I get an error from RClone.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.0

  • os/version: Microsoft Windows Server 2022 Standard 21H2 (64 bit)
  • os/kernel: 10.0.20348.2113 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Scality (on prem storage)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy README.txt <remote_name>:<bucket_name> -vv --s3-no-check-bucket --s3-no-head --s3-no-head-object --retries=1

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[<remote_name>]
type = s3
provider = Other
access_key_id = XXX
secret_access_key = XXX
endpoint = https://<endpoint-url>

Here is the IAM Policy applied to the user :

"Version":"2012-10-17"
    "Statement":[
        0:{
            "Sid":"<StatementID>"
                "Action":[
                    0:"s3:Put*"
                    1:"s3:List*"
                    2:"s3:AbortMultipartUpload"
                    3:"s3:Get*"
                ]
            "Effect":"Allow"
            "Resource":[
                0:"*"
            ]
        }
    ]

A log from the command that you were trying to run with the -vv flag

This is the log for a small file :

C:\Temp\rclone-v1.64.0-windows-amd64>rclone copy README.txt <remote_name>:<bucket_name> -vv --s3-no-check-bucket --s3-no-head --s3-no-head-object --retries=1
2023/12/05 10:22:26 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "README.txt" "<remote_name>:<bucket_name>" "-vv" "--s3-no-check-bucket" "--s3-no-head" "--s3-no-head-object" "--retries=1"]
2023/12/05 10:22:26 DEBUG : Creating backend with remote "README.txt"
2023/12/05 10:22:26 DEBUG : Using config file from "C:\\Users\\<user_profile>\\AppData\\Roaming\\rclone\\rclone.conf"
2023/12/05 10:22:26 DEBUG : fs cache: adding new entry for parent of "README.txt", "//?/C:/Temp/rclone-v1.64.0-windows-amd64"
2023/12/05 10:22:26 DEBUG : Creating backend with remote "<remote_name>:<bucket_name>"
2023/12/05 10:22:26 DEBUG : <remote_name>: detected overridden config - adding "{ECHgI}" suffix to name
2023/12/05 10:22:26 DEBUG : Resolving service "s3" region "us-east-1"
2023/12/05 10:22:26 DEBUG : fs cache: renaming cache item "<remote_name>:<bucket_name>" to be canonical "<remote_name>{ECHgI}:<bucket_name>"
2023/12/05 10:22:26 DEBUG : README.txt: Sizes differ (src 2086374 vs dst 0)
2023/12/05 10:22:26 DEBUG : README.txt: md5 = 124c4692e127d1c2b12ea35b8dc49c96 OK
2023/12/05 10:22:26 INFO  : README.txt: Copied (replaced existing)
2023/12/05 10:22:26 INFO  :
Transferred:        1.990 MiB / 1.990 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.2s

2023/12/05 10:22:26 DEBUG : 5 go routines active

This is the log for a large file :

C:\Temp\rclone-v1.64.0-windows-amd64>rclone copy SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO <remote_name>:<bucket_name>/ -vv --s3-no-check-bucket --s3-no-head --s3-no-head-object --retries=1
2023/12/05 10:16:06 DEBUG : rclone: Version "v1.64.0" starting with parameters ["rclone" "copy" "SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO" "<remote_name>:<bucket_name>/" "-vv" "--s3-no-check-bucket" "--s3-no-head" "--s3-no-head-object" "--retries=1"]
2023/12/05 10:16:06 DEBUG : Creating backend with remote "SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO"
2023/12/05 10:16:06 DEBUG : Using config file from "C:\\Users\\<user_profile>\\AppData\\Roaming\\rclone\\rclone.conf"
2023/12/05 10:16:06 DEBUG : fs cache: adding new entry for parent of "SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO", "//?/C:/Temp/rclone-v1.64.0-windows-amd64"
2023/12/05 10:16:06 DEBUG : Creating backend with remote "<remote_name>:<bucket_name>/"
2023/12/05 10:16:06 DEBUG : <remote_name>: detected overridden config - adding "{ECHgI}" suffix to name
2023/12/05 10:16:06 DEBUG : Resolving service "s3" region "us-east-1"
2023/12/05 10:16:06 DEBUG : fs cache: renaming cache item "<remote_name>:<bucket_name>/" to be canonical "<remote_name>{ECHgI}:<bucket_name>"
2023/12/05 10:16:06 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Sizes differ (src 5440065536 vs dst 0)
2023/12/05 10:17:06 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:       1m0.5s
Transferring:
 * SW_DVD9_Win_Pro_11_22H…DU_N_MLF_X23-50057.ISO: transferring

2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: open chunk writer: started multipart upload: 127bd4ebf31149908615ac9fcbb624dc
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Starting multi-thread copy with 1038 chunks of size 5Mi with 4 parallel streams
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: chunk 3/1038 (10485760-15728640) size 5Mi starting
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: chunk 4/1038 (15728640-20971520) size 5Mi starting
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: chunk 1/1038 (0-5242880) size 5Mi starting
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: chunk 2/1038 (5242880-10485760) size 5Mi starting
2023/12/05 10:17:47 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multipart upload wrote chunk 1 with 5242880 bytes and etag "ccc427170474919f4db77522e61b27f0"
...
2023/12/05 10:18:14 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: chunk 1036/1038 (5426380800-5431623680) size 5Mi finished
2023/12/05 10:18:14 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multipart upload wrote chunk 1038 with 3198976 bytes and etag "e549f6d1a41ae8d371cc1ff6d63e4cca"
2023/12/05 10:18:14 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: chunk 1038/1038 (5436866560-5440065536) size 3.051Mi finished
2023/12/05 10:18:15 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multipart upload "127bd4ebf31149908615ac9fcbb624dc" finished
2023/12/05 10:18:15 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Finished multi-thread copy with 1038 parts of size 5Mi
2023/12/05 10:18:15 ERROR : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: corrupted on transfer: sizes differ 5440065536 vs 0
2023/12/05 10:18:15 INFO  : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Removing failed copy
2023/12/05 10:18:15 INFO  : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Failed to remove failed copy: AccessDenied: Access Denied
        status code: 403, request id: 2442ac15d741aeb377f7, host id: 2442ac15d741aeb377f7
2023/12/05 10:18:15 ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: sizes differ 5440065536 vs 0
2023/12/05 10:18:15 INFO  :
Transferred:        5.066 GiB / 5.066 GiB, 100%, 148.471 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       2m9.5s

2023/12/05 10:18:15 DEBUG : 13 go routines active
2023/12/05 10:18:15 Failed to copy: corrupted on transfer: sizes differ 5440065536 vs 0
2

Would you mind to try the same with the latest rclone version? v1.65?

3

This is likely to do with the threshold for switching to multipart uploads

  --s3-upload-cutoff SizeSuffix         Cutoff for switching to chunked upload (default 200Mi)

This means that after the multipart transfer, rclone read the size of the file and saw it was 0, rather than the 5440065536 it was expecting.

This is probably because the server is still assembling the chunks, I'd guess. If you try this on AWS you'll always get 5440065536 as the size, so I'd count this as an S3 incompatibility in Scality's S3 implementation.

You've got options for workarounds though - set --s3-upload-cutoff 5G so file up to 5G will be uploaded as single parts. This can't be bigger that 5G unfortunately.

The other option would be to use --ignore-size which isn't ideal but would probably work.

Also setting --retries=2 will likely work - rclone will notice the file is fine on the second retry.

5

Thanks guys for your replies.

Unfortunately, I tried all of the options suggested, but I'm still having errors. And when I use the --ignore-size option, I get hash calculation errors (even for small files) :

C:\Temp\rclone-v1.65.0-windows-amd64>rclone copy SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO <remote_name>:<bucket_name>/test -vv --s3-no-check-bucket --s3-no-head --s3-no-head-object --ignore-size --retries=1
2023/12/07 15:52:18 DEBUG : rclone: Version "v1.65.0" starting with parameters ["rclone" "copy" "SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO" "<remote_name>:<bucket_name>/test" "-vv" "--s3-no-check-bucket" "--s3-no-head" "--s3-no-head-object" "--ignore-size" "--retries=1"]
2023/12/07 15:52:18 DEBUG : Creating backend with remote "SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO"
2023/12/07 15:52:18 DEBUG : Using config file from "C:\\Users\\<user_profile>\\AppData\\Roaming\\rclone\\rclone.conf"
2023/12/07 15:52:18 DEBUG : fs cache: adding new entry for parent of "SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO", "//?/C:/Temp/rclone-v1.65.0-windows-amd64"
2023/12/07 15:52:18 DEBUG : Creating backend with remote "<remote_name>:<bucket_name>/test"
2023/12/07 15:52:18 DEBUG : <remote_name>: detected overridden config - adding "{ECHgI}" suffix to name
2023/12/07 15:52:18 DEBUG : Resolving service "s3" region "us-east-1"
2023/12/07 15:52:18 DEBUG : fs cache: renaming cache item "<remote_name>:<bucket_name>/test" to be canonical "<remote_name>{ECHgI}:<bucket_name>/test"
2023/12/07 15:52:18 NOTICE: SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Failed to read metadata: object not found
2023/12/07 15:52:18 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Modification times differ by 1997h2m22.2904663s: 2023-09-15 11:49:56.5686723 +0200 CEST, 2023-12-07 15:52:18.8591386 +0100 CET m=+0.410353101
2023/12/07 15:52:18 ERROR : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Failed to calculate src hash: hash: failed to read: context canceled
2023/12/07 15:52:18 ERROR : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: Failed to calculate dst hash: object not found
2023/12/07 15:52:18 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: md5 differ
2023/12/07 15:52:18 DEBUG : SW_DVD9_Win_Pro_11_22H2.10_64BIT_English_Pro_Ent_EDU_N_MLF_X23-50057.ISO: multi-thread copy: disabling buffering because source is local disk

The files get copied and are usable, but for scripted transfers with log files, it's pretty anoying as these errors trigger an alert every time.

6

Is this file bigger than 5G?

Did you try that? I would have thought that would fix your problem.

You might need to set

  --retries-sleep Duration   Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)

To give the scality time to assemble the file.

7

Unfortunately, even with --retries=2 I still get the issue. I tried setting --retries-sleep=60s but still with no luck.
What seems strange is that the "dst" size is always 0, no matter what size is the source file. As if something prevented RClone to read the file size on the destination.
I modified the policy to allow "s3:*", but still the same issue. The only difference is that now, RClone is able to "remove failed copy".