Rclone throwing error "corrupted on transfer: sizes differ"

[navdeep]

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I have two workstations that are integrated to rclone.

1. From workstation1 I copy (using rcd api's) a file from local file system to AWS s3,
2. Once the file is copied/uploaded successfully, I receive a notification from s3 and from workstation 2 I copy the same file from AWS s3 to local file system.
   When I perform the second action, rclone is throwing an error saying "corrupted on transfer: sizes differ".
   I have set --local-no-check-updated=true already.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

• os/version: Microsoft Windows 10 Enterprise 21H2 (64 bit)
• os/kernel: 10.0.19044.2846 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.19.4
• go/linking: static
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

http://localhost:5572/operations/copyfile

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

Paste config here

A log from the command that you were trying to run with the -vv flag

Paste  log here

[kapitainsky]

First try the same with the latest rclone version. You are using quite old one.

[navdeep]

@kapitainsky
Thanks for the suggestion.
The issue doesn't seem to occur in the latest version.
Can you please point me to the github commit or feature request that was done between rclone version 1.61.1 and 1.66.0 to fix this issue ?

[kapitainsky]

@navdeep

Can you please point me to the github commit or feature request that was done between rclone version 1.61.1 and 1.66.0 to fix this issue ?

I am not following all commits and in addition 1.61.1 is from 2022... It was 12 releases ago:) so many things changed since that time.

If you want to identify specific commit probably the best way is to run git bisect.

[asdffdsa (jojothehumanmonkey)]

@navdeep

The issue doesn't seem to occur in the latest version.

now, using the latest version, are you forced to use --local-no-check-updated ?

[navdeep]

@asdffdsa
Even without --local-no-check-updated , it is working without issues.
Just wanted to know whether there was any specific fix done to address this issue, since it doesn't occur for all files, only for some specific file types.

[asdffdsa (jojothehumanmonkey)]

@navdeep

Even without --local-no-check-updated , it is working without issues.
Just wanted to know whether there was any specific fix done to address this issue, since it doesn't occur for all files, only for some specific file types.

sorry, not sure i understand that.
now, is rclone working without issues or not?

at this point, do you still run into corrupted on transfer: sizes differ ?

[navdeep]

@asdffdsa ,
No, I do not run into this issue anymore.
But was there a change/fix done to address this issue ? The answer to this will help us to make certain decisions to rclone integration

[asdffdsa (jojothehumanmonkey)]

from what little info you have provided, i do not know what the exact issue is, so i cannot pinpoint the change/fix.
no commands were posted, no logs were posted, no errors were posted, no config was posted.

[navdeep]

@asdffdsa
Command is already part of the post.
It is http://localhost:5572/operations/copyfile

No config, since it is rcd
command to run rclone : >rclone rcd --log-level INFO --rc-user=user --rc-pass=pass --s3-upload-concurrency=8 --s3-chunk-size=16M --s3-upload-cutoff=1G --s3-no-check-bucket

Failing scenario:

1. From workstation1 I copy (using rcd api's) a file from local file system to AWS s3,
2. Once the file is copied/uploaded successfully, I receive a notification from s3 and from workstation 2 I copy the same file from AWS s3 to local file system.
   When I perform the second action, rclone is throwing an error saying "corrupted on transfer: sizes differ".

part of the log failing

[asdffdsa (jojothehumanmonkey)]

ok, i tried to help you pinpoint the exact issue, but you are not provided the requested info, and simply re-posting old incomplete info.

[navdeep]

@ncw
Please help on this topic.
I just wanted to know about any change/feature request that was done post 1.61.1 version to the latest version to fix issues related to "file corrupt on transfer".
Already provided all the required info I have, please let me know if you need any more details that will help troubleshoot ?