FTPS : corrupted on transfer: sizes differ and MD5 hash differ

wanghuang · April 1, 2022, 9:47am

What is the problem you are having with rclone?

When I use Rclone-v1.58.0 to set FTPS to upload files, there will be an error of corrupted on transfer: sizes differ 1020264448 vs 1019228160. When I use the "--ignore-size" parameter, although the file can be uploaded to the server, the file MD5 is different from the local one. The file in the server itself is about 1MB less than the local file. Is there any solution.

Use FTP tools to upload files without loss

Run the command 'rclone version' and share the full output of the command.

- os/version: centos 8.4.2105 (64 bit)
- os/kernel: 4.18.0-305.19.1.el8_4.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.8
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy video.mp4 wanghuang:

The rclone config contents with secrets removed.

type = ftp
host = xxxxx
user = xxx
pass = xxxxxxx
explicit_tls = true
no_check_certificate = true

A log from the command with the `-vv` flag

[root@centos work]# rclone copy video.mp4 wanghuang: -vv
2022/04/01 05:39:18 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "video.mp4" "wanghuang:" "-vv"]
2022/04/01 05:39:18 DEBUG : Creating backend with remote "video.mp4"

2022/04/01 05:39:18 DEBUG : fs cache: adding new entry for parent of "video.mp4", "/work"

2022/04/01 05:39:19 DEBUG : video.mp4: Need to transfer - File not found at Destination
2022/04/01 05:39:19 ERROR : video.mp4: corrupted on transfer: sizes differ 14531413 vs 13381632
2022/04/01 05:39:19 INFO  : video.mp4: Removing failed copy
2022/04/01 05:39:19 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 14531413 vs 13381632
2022/04/01 05:39:19 DEBUG : video.mp4: Need to transfer - File not found at Destination
2022/04/01 05:39:20 ERROR : video.mp4: corrupted on transfer: sizes differ 14531413 vs 13643776
2022/04/01 05:39:20 INFO  : video.mp4: Removing failed copy
2022/04/01 05:39:20 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 14531413 vs 13643776
2022/04/01 05:39:20 DEBUG : video.mp4: Need to transfer - File not found at Destination
2022/04/01 05:39:20 ERROR : video.mp4: corrupted on transfer: sizes differ 14531413 vs 13643776
2022/04/01 05:39:20 INFO  : video.mp4: Removing failed copy
2022/04/01 05:39:20 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 14531413 vs 13643776
2022/04/01 05:39:20 INFO  : 
Transferred:   	   41.575 MiB / 41.575 MiB, 100%, 27.707 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         2.0s

2022/04/01 05:39:20 DEBUG : 3 go routines active
2022/04/01 05:39:20 Failed to copy: corrupted on transfer: sizes differ 14531413 vs 13643776

asdffdsa · April 1, 2022, 12:41pm

hello and welcome to the forum,

MD5 hash differ
FTP does not support hashes.

can you use SFTP, as rclone has better support for that, including MD5 hashes?

can you post the output of this command?
if possible, copy a single file so the debug log will be small
rclone copy video.mp4 wanghuang: --dump=headers --retries=1

wanghuang · April 2, 2022, 1:16am

Hello ！

I download the file from the server and do MD5 comparison with the original file.
I found that using --ignore-size when uploading files lost some data, when using " rclone copy video.mp4 wanghuang: " this command will compare the uploaded file with the original file. Sizes differ 1020264448 vs 1019228160. So my files have been unable to upload to the server.
I want to know corrupted on transfer: sizes differ 1020264448 vs 1019228160 , where is the problem?

我是按照链接配置的FTPS

rclone copy video.mp4 wanghuang: --dump=headers --retries=1

output content：

[root@centos8 work]# rclone copy video.mp4 wanghuang: --dump=headers --retries=1
2022/04/01 20:59:20 NOTICE: Automatically setting -vv as --dump is enabled
2022/04/01 20:59:20 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "video.mp4" "wanghuang:" "--dump=headers" "--retries=1"]
2022/04/01 20:59:20 DEBUG : Creating backend with remote "video.mp4"
2022/04/01 20:59:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/04/01 20:59:20 DEBUG : fs cache: adding new entry for parent of "video.mp4", "/work"
2022/04/01 20:59:20 DEBUG : Creating backend with remote "wanghuang:"
2022/04/01 20:59:20 DEBUG : ftp://xxxxxx: Connecting to FTP server
2022/04/01 20:59:20 DEBUG : FTP Rx: "220 pyftpdlib based ftpd ready."
2022/04/01 20:59:20 DEBUG : FTP Tx: "AUTH TLS"
2022/04/01 20:59:20 DEBUG : FTP Rx: "234 AUTH TLS successful."
2022/04/01 20:59:20 DEBUG : FTP Tx: "USER xxxx"
2022/04/01 20:59:21 DEBUG : FTP Rx: "331 Username ok, send password."
2022/04/01 20:59:21 DEBUG : FTP Tx: PASS *****
2022/04/01 20:59:21 DEBUG : FTP Rx: "230 Login successful."
2022/04/01 20:59:21 DEBUG : FTP Tx: "FEAT"
2022/04/01 20:59:21 DEBUG : FTP Rx: "211-Features supported:"
2022/04/01 20:59:21 DEBUG : FTP Rx: " AUTH SSL"
2022/04/01 20:59:21 DEBUG : FTP Rx: " AUTH TLS"
2022/04/01 20:59:21 DEBUG : FTP Rx: " EPRT"
2022/04/01 20:59:21 DEBUG : FTP Rx: " EPSV"
2022/04/01 20:59:21 DEBUG : FTP Rx: " MDTM"
2022/04/01 20:59:21 DEBUG : FTP Rx: " MFMT"
2022/04/01 20:59:21 DEBUG : FTP Rx: " MLST type*;perm*;size*;modify*;unique*;unix.mode;unix.uid;unix.gid;"
2022/04/01 20:59:21 DEBUG : FTP Rx: " PBSZ"
2022/04/01 20:59:21 DEBUG : FTP Rx: " PROT"
2022/04/01 20:59:21 DEBUG : FTP Rx: " REST STREAM"
2022/04/01 20:59:21 DEBUG : FTP Rx: " SIZE"
2022/04/01 20:59:21 DEBUG : FTP Rx: " TVFS"
2022/04/01 20:59:21 DEBUG : FTP Rx: " UTF8"
2022/04/01 20:59:21 DEBUG : FTP Rx: "211 End FEAT."
2022/04/01 20:59:21 DEBUG : FTP Tx: "TYPE I"
2022/04/01 20:59:21 DEBUG : FTP Rx: "200 Type set to: Binary."
2022/04/01 20:59:21 DEBUG : FTP Tx: "OPTS UTF8 ON"
2022/04/01 20:59:21 DEBUG : FTP Rx: "501 Invalid argument."
2022/04/01 20:59:21 DEBUG : FTP Tx: "PBSZ 0"
2022/04/01 20:59:21 DEBUG : FTP Rx: "200 PBSZ=0 successful."
2022/04/01 20:59:21 DEBUG : FTP Tx: "PROT P"
2022/04/01 20:59:21 DEBUG : FTP Rx: "200 Protection set to Private"
2022/04/01 20:59:21 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:59:21 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2010|)."
2022/04/01 20:59:21 DEBUG : FTP Tx: "MLSD ."
2022/04/01 20:59:21 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:59:21 DEBUG : FTP Rx: "type=file;size=462350656;perm=el;modify=20220401074514;unique=; pycharm-professional-2021.3.3.exe"
2022/04/01 20:59:21 DEBUG : FTP Rx: "type=file;size=13676544;perm=el;modify=20220402004935;unique=; video.mp4"
2022/04/01 20:59:21 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:59:21 DEBUG : video.mp4: Sizes differ (src 14531413 vs dst 13676544)
2022/04/01 20:59:21 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:59:21 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2627|)."
2022/04/01 20:59:21 DEBUG : FTP Tx: "STOR video.mp4"
2022/04/01 20:59:21 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:59:21 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:59:21 DEBUG : FTP Tx: "MFMT 20220401024021 video.mp4"
2022/04/01 20:59:21 DEBUG : FTP Rx: "213 Modify=20220401024021; /video.mp4."
2022/04/01 20:59:21 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:59:21 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2186|)."
2022/04/01 20:59:21 DEBUG : FTP Tx: "MLSD ."
2022/04/01 20:59:21 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:59:21 DEBUG : FTP Rx: "type=file;size=462350656;perm=el;modify=20220401074514;unique=; pycharm-professional-2021.3.3.exe"
2022/04/01 20:59:21 DEBUG : FTP Rx: "type=file;size=13152256;perm=el;modify=20220402005921;unique=; video.mp4"
2022/04/01 20:59:21 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:59:21 ERROR : video.mp4: corrupted on transfer: sizes differ 14531413 vs 13152256
2022/04/01 20:59:21 INFO  : video.mp4: Removing failed copy
2022/04/01 20:59:21 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:59:21 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2483|)."
2022/04/01 20:59:21 DEBUG : FTP Tx: "MLSD ."
2022/04/01 20:59:21 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:59:21 DEBUG : FTP Rx: "type=file;size=462350656;perm=el;modify=20220401074514;unique=; pycharm-professional-2021.3.3.exe"
2022/04/01 20:59:21 DEBUG : FTP Rx: "type=file;size=13152256;perm=el;modify=20220402005921;unique=; video.mp4"
2022/04/01 20:59:21 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:59:21 DEBUG : FTP Tx: "DELE video.mp4"
2022/04/01 20:59:21 DEBUG : FTP Rx: "250 File removed."
2022/04/01 20:59:21 ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: sizes differ 14531413 vs 13152256
2022/04/01 20:59:21 INFO  : 
Transferred:   	   13.858 MiB / 13.858 MiB, 100%, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.8s

2022/04/01 20:59:21 DEBUG : 3 go routines active
2022/04/01 20:59:21 Failed to copy: corrupted on transfer: sizes differ 14531413 vs 13152256

rclone copy video.mp4 wanghuang: --ignore-size  --dump=headers --retries=1

output content： file size is different

2022/04/01 20:49:35 NOTICE: Automatically setting -vv as --dump is enabled
2022/04/01 20:49:35 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "video.mp4" "wanghuang:" "--ignore-size" "--dump=headers" "--retries=1"]
2022/04/01 20:49:35 DEBUG : Creating backend with remote "video.mp4"
2022/04/01 20:49:35 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/04/01 20:49:35 DEBUG : fs cache: adding new entry for parent of "video.mp4", "/work"
2022/04/01 20:49:35 DEBUG : Creating backend with remote "wanghuang:"
2022/04/01 20:49:35 DEBUG : ftp://xxxxx: Connecting to FTP server
2022/04/01 20:49:35 DEBUG : FTP Rx: "220 pyftpdlib based ftpd ready."
2022/04/01 20:49:35 DEBUG : FTP Tx: "AUTH TLS"
2022/04/01 20:49:35 DEBUG : FTP Rx: "234 AUTH TLS successful."
2022/04/01 20:49:35 DEBUG : FTP Tx: "USER xxxx"
2022/04/01 20:49:35 DEBUG : FTP Rx: "331 Username ok, send password."
2022/04/01 20:49:35 DEBUG : FTP Tx: PASS *****
2022/04/01 20:49:35 DEBUG : FTP Rx: "230 Login successful."
2022/04/01 20:49:35 DEBUG : FTP Tx: "FEAT"
2022/04/01 20:49:35 DEBUG : FTP Rx: "211-Features supported:"
2022/04/01 20:49:35 DEBUG : FTP Rx: " AUTH SSL"
2022/04/01 20:49:35 DEBUG : FTP Rx: " AUTH TLS"
2022/04/01 20:49:35 DEBUG : FTP Rx: " EPRT"
2022/04/01 20:49:35 DEBUG : FTP Rx: " EPSV"
2022/04/01 20:49:35 DEBUG : FTP Rx: " MDTM"
2022/04/01 20:49:35 DEBUG : FTP Rx: " MFMT"
2022/04/01 20:49:35 DEBUG : FTP Rx: " MLST type*;perm*;size*;modify*;unique*;unix.mode;unix.uid;unix.gid;"
2022/04/01 20:49:35 DEBUG : FTP Rx: " PBSZ"
2022/04/01 20:49:35 DEBUG : FTP Rx: " PROT"
2022/04/01 20:49:35 DEBUG : FTP Rx: " REST STREAM"
2022/04/01 20:49:35 DEBUG : FTP Rx: " SIZE"
2022/04/01 20:49:35 DEBUG : FTP Rx: " TVFS"
2022/04/01 20:49:35 DEBUG : FTP Rx: " UTF8"
2022/04/01 20:49:35 DEBUG : FTP Rx: "211 End FEAT."
2022/04/01 20:49:35 DEBUG : FTP Tx: "TYPE I"
2022/04/01 20:49:35 DEBUG : FTP Rx: "200 Type set to: Binary."
2022/04/01 20:49:35 DEBUG : FTP Tx: "OPTS UTF8 ON"
2022/04/01 20:49:35 DEBUG : FTP Rx: "501 Invalid argument."
2022/04/01 20:49:35 DEBUG : FTP Tx: "PBSZ 0"
2022/04/01 20:49:35 DEBUG : FTP Rx: "200 PBSZ=0 successful."
2022/04/01 20:49:35 DEBUG : FTP Tx: "PROT P"
2022/04/01 20:49:35 DEBUG : FTP Rx: "200 Protection set to Private"
2022/04/01 20:49:35 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:49:35 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2531|)."
2022/04/01 20:49:35 DEBUG : FTP Tx: "MLSD ."
2022/04/01 20:49:35 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:49:35 DEBUG : FTP Rx: "type=file;size=462350656;perm=el;modify=20220401074514;unique=; pycharm-professional-2021.3.3.exe"
2022/04/01 20:49:35 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:49:35 DEBUG : video.mp4: Need to transfer - File not found at Destination
2022/04/01 20:49:35 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:49:35 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2210|)."
2022/04/01 20:49:35 DEBUG : FTP Tx: "STOR video.mp4"
2022/04/01 20:49:35 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:49:36 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:49:36 DEBUG : FTP Tx: "MFMT 20220401024021 video.mp4"
2022/04/01 20:49:36 DEBUG : FTP Rx: "213 Modify=20220401024021; /video.mp4."
2022/04/01 20:49:36 DEBUG : FTP Tx: "EPSV"
2022/04/01 20:49:36 DEBUG : FTP Rx: "229 Entering extended passive mode (|||2186|)."
2022/04/01 20:49:36 DEBUG : FTP Tx: "MLSD ."
2022/04/01 20:49:36 DEBUG : FTP Rx: "125 Data connection already open. Transfer starting."
2022/04/01 20:49:36 DEBUG : FTP Rx: "type=file;size=462350656;perm=el;modify=20220401074514;unique=; pycharm-professional-2021.3.3.exe"
2022/04/01 20:49:36 DEBUG : FTP Rx: "type=file;size=13676544;perm=el;modify=20220402004935;unique=; video.mp4"
2022/04/01 20:49:36 DEBUG : FTP Rx: "226 Transfer complete."
2022/04/01 20:49:36 INFO  : video.mp4: Copied (new)
2022/04/01 20:49:36 INFO  : 
Transferred:   	   13.858 MiB / 13.858 MiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.6s

2022/04/01 20:49:36 DEBUG : 3 go routines active

asdffdsa · April 2, 2022, 1:59pm

have you tested uploading using another ftp client, such as filezilla?

fwiw, ftp cannot verify file transfers.
sftp can verify file transfers and is supported by ultracc.

as a side note, i use pycharm to script rclone.

wanghuang · April 6, 2022, 7:17am

I tested other ftp client tools, and the files transferred using them have no loss after comparing with local files, and the file size is the same.

The company's own storage service, currently only has the ftp protocol.

ncw · April 6, 2022, 11:42am

Can you tell us what kind of FTP server you are connecting to? I'd like to reproduce this.

asdffdsa · April 6, 2022, 1:14pm

hi,

is this what you need, from the --dump log, i had the OP post?
220 pyftpdlib based ftpd ready

ncw · April 6, 2022, 1:28pm

Ah, so it is likely a custom ftp server that probably isn't available freely.

asdffdsa · April 6, 2022, 1:40pm

"pyftpdlib is a high-level library to easily write asynchronous portable FTP servers with Python."

and the documentation makes clear it is not fully compliant with ftp standards and never will be.

"Some of the features like ACCT or SMNT commands will never be implemented deliberately. Other features described in more recent RFCs like the TLS/SSL support for securing FTP (RFC-4217) are now implemented as a demo script, waiting to reach the proper level of stability to be then included in the standard code base."

ncw · April 6, 2022, 3:33pm

I think this problem is unlikely to be an rclone problem as it is very well tested with lots of different FTP servers, but then again there are lots of corner cases in the FTP protocol so maybe we are doing something wrong...

The FTPS would be the least used part here, so @wanghuang can you try the transfer without FTPS to see if it is successful like that?

system · May 6, 2022, 3:34pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.