FTP: Uploading with a lot of doubles in root folder

Suspected Bug
1

Hello.

Upload Sintel.mp4 to "hello" folder.

# rclone -vv --ignore-size copy Sintel.mp4 REMOTE:hello/
2021/05/27 23:18:30 DEBUG : Using config file from "/home/rclone.conf"
2021/05/27 23:18:30 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vv" "--ignore-size" "copy" "Sintel.mp4" "REMOTE:hello/"]
2021/05/27 23:18:30 DEBUG : Creating backend with remote "Sintel.mp4"
2021/05/27 23:18:30 DEBUG : fs cache: adding new entry for parent of "Sintel.mp4", "/home/Sintel"
2021/05/27 23:18:30 DEBUG : Creating backend with remote "REMOTE:hello/"
2021/05/27 23:18:30 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/27 23:18:30 DEBUG : Sintel.mp4: Need to transfer - File not found at Destination
2021/05/27 23:18:31 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/27 23:18:32 DEBUG : Sintel.mp4: Failed to remove: object not found
2021/05/27 23:18:32 DEBUG : Sintel.mp4: Received error: update stor: write tcp 172.18.0.18:54340->185.150.30.12:56211: write: broken pipe - low level retry 1/10
2021/05/27 23:19:30 INFO  :
Transferred:      123.328M / 123.328 MBytes, 100%, 2.058 MBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       1m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 524.500k/s, 0s

2021/05/27 23:20:30 INFO  :
Transferred:      123.328M / 123.328 MBytes, 100%, 1.028 MBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       2m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 10.914k/s, 0s

2021/05/27 23:21:30 INFO  :
Transferred:      123.328M / 123.328 MBytes, 100%, 701.946 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       3m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 232/s, 0s

2021/05/27 23:22:30 INFO  :
Transferred:      123.328M / 123.328 MBytes, 100%, 526.394 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       4m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 4/s, 0s

2021/05/27 23:23:30 INFO  :
Transferred:      123.328M / 123.328 MBytes, 100%, 421.084 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       5m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 0/s, 0s

2021/05/27 23:23:41 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connection failed, closing: write tcp 172.18.0.18:58286->185.150.30.13:21: i/o timeout
2021/05/27 23:23:41 DEBUG : Sintel.mp4: Received error: update getinfo: read tcp 172.18.0.18:59794->185.150.30.13:22389: i/o timeout - low level retry 2/10
2021/05/27 23:23:41 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/27 23:24:30 INFO  :
Transferred:      246.583M / 246.583 MBytes, 100%, 701.562 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       6m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 937.548k/s, 0s

2021/05/27 23:25:30 INFO  :
Transferred:      246.583M / 246.583 MBytes, 100%, 601.318 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       7m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 19.510k/s, 0s

2021/05/27 23:26:30 INFO  :
Transferred:      246.583M / 246.583 MBytes, 100%, 526.140 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       8m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 415/s, 0s

2021/05/27 23:27:30 INFO  :
Transferred:      246.583M / 246.583 MBytes, 100%, 467.670 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       9m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 8/s, 0s

2021/05/27 23:28:30 INFO  :
Transferred:      246.583M / 246.583 MBytes, 100%, 420.896 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      10m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 0/s, 0s

2021/05/27 23:28:56 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connection failed, closing: write tcp 172.18.0.18:46032->185.150.30.14:21: i/o timeout
2021/05/27 23:28:56 DEBUG : Sintel.mp4: Received error: update getinfo: read tcp 172.18.0.18:52200->185.150.30.14:58787: i/o timeout - low level retry 3/10
2021/05/27 23:28:56 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/27 23:29:02 ERROR : Sintel.mp4: Failed to copy: update getinfo: object not found
2021/05/27 23:29:02 ERROR : Attempt 1/3 failed with 1 errors and: update getinfo: object not found
2021/05/27 23:29:02 DEBUG : Sintel.mp4: Need to transfer - File not found at Destination
2021/05/27 23:29:03 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/27 23:29:03 DEBUG : Sintel.mp4: Failed to remove: object not found
2021/05/27 23:29:03 DEBUG : Sintel.mp4: Received error: update stor: write tcp 172.18.0.18:41888->185.150.30.12:56219: write: broken pipe - low level retry 1/10
2021/05/27 23:29:30 INFO  :
Transferred:      493.166M / 493.166 MBytes, 100%, 765.391 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      11m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 4.041M/s, 0s

2021/05/27 23:30:30 INFO  :
Transferred:      493.166M / 493.166 MBytes, 100%, 701.589 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      12m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 86.106k/s, 0s

2021/05/27 23:31:30 INFO  :
Transferred:      493.166M / 493.166 MBytes, 100%, 647.607 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      13m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 1.791k/s, 0s

2021/05/27 23:32:30 INFO  :
Transferred:      493.166M / 493.166 MBytes, 100%, 601.338 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      14m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 38/s, 0s

2021/05/27 23:33:30 INFO  :
Transferred:      493.166M / 493.166 MBytes, 100%, 561.239 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:      15m0.4s
Transferring:
 *                                    Sintel.mp4:100% /123.255M, 0/s, 0s

2021/05/27 23:34:18 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connection failed, closing: write tcp 172.18.0.18:47106->185.150.30.14:21: i/o timeout
2021/05/27 23:34:18 DEBUG : Sintel.mp4: Received error: update getinfo: read tcp 172.18.0.18:41366->185.150.30.14:58791: i/o timeout - low level retry 2/10
2021/05/27 23:34:18 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/27 23:34:27 ERROR : Sintel.mp4: Failed to copy: update getinfo: object not found
2021/05/27 23:34:27 ERROR : Attempt 2/3 failed with 1 errors and: update getinfo: object not found
2021/05/27 23:34:27 DEBUG : Sintel.mp4: Need to transfer - File not found at Destination
2021/05/27 23:34:30 INFO  :
Transferred:      704.417M / 739.675 MBytes, 95%, 751.605 kBytes/s, ETA 48s
Transferred:            0 / 1, 0%
Elapsed time:      16m0.4s
Transferring:
 *                                    Sintel.mp4: 71% /123.255M, 36.946M/s, 0s

2021/05/27 23:34:33 ERROR : Sintel.mp4: Failed to copy: update getinfo: object not found
2021/05/27 23:34:33 ERROR : Attempt 3/3 failed with 1 errors and: update getinfo: object not found
2021/05/27 23:34:33 INFO  :
Transferred:      739.675M / 739.675 MBytes, 100%, 786.923 kBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:      16m3.2s

2021/05/27 23:34:33 DEBUG : 7 go routines active
2021/05/27 23:34:33 Failed to copy: update getinfo: object not found

Files are successfully uploaded, but are duplicated many times in the root.

Hello folder is empty.

2

If you run the command of the 2nd time, then 3 identical files are created in the root and a new folder in /hello/hello.

3

With this FTP server, when you upload something can you no longer see it in the listing via FTP?

If that is the case then these problems

Will cause duplicates.

Those are the problems that need fixing!

Can you do a run like this and post the output please?

rclone -vv --retries 1 --low-level-retries 1 --dump bodies --ignore-size copy Sintel.mp4 REMOTE:hello/

Also please post your config file with secrets removed - thanks.

4

Thanks for the answer.

My /home/rclone.conf

[REMOTE]
type = ftp
user = 700eef6812685613d184
pass = XMf9Fs8pbeHOcknh96Y9N3COox110xbkda42hSUzEw
host = ftp.ftpwebsite.com

# rclone -vv --retries 1 --low-level-retries 1 --dump bodies --ignore-size copy Sintel.mp4 REMOTE:hello/
2021/05/28 16:30:28 DEBUG : Using config file from "/home/rclone.conf"
2021/05/28 16:30:28 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vv" "--retries" "1" "--low-level-retries" "1" "--dump" "bodies" "--ignore-size" "copy" "Sintel.mp4" "REMOTE:hello/"]
2021/05/28 16:30:28 DEBUG : Creating backend with remote "Sintel.mp4"
2021/05/28 16:30:28 DEBUG : fs cache: adding new entry for parent of "Sintel.mp4", "/home/Sintel"
2021/05/28 16:30:28 DEBUG : Creating backend with remote "REMOTE:hello/"
2021/05/28 16:30:28 DEBUG : ftp://ftp.ftpwebsite.com:21/hello: Connecting to FTP server
2021/05/28 16:30:28 DEBUG : FTP Rx: "220 IP modified"
2021/05/28 16:30:28 DEBUG : FTP Tx: "USER 700eef6812685613d184"
2021/05/28 16:30:28 DEBUG : FTP Rx: "331 Password required for 700eef6812685613d184."
2021/05/28 16:30:28 DEBUG : FTP Tx: PASS *****
2021/05/28 16:30:28 DEBUG : FTP Rx: "230 User 700eef6812685613d184 logged in from 51.9.22.220."
2021/05/28 16:30:28 DEBUG : FTP Tx: "FEAT"
2021/05/28 16:30:28 DEBUG : FTP Rx: "211-Extensions supported"
2021/05/28 16:30:28 DEBUG : FTP Rx: " EPRT"
2021/05/28 16:30:28 DEBUG : FTP Rx: " EPSV"
2021/05/28 16:30:28 DEBUG : FTP Rx: " PASV"
2021/05/28 16:30:28 DEBUG : FTP Rx: " SIZE"
2021/05/28 16:30:28 DEBUG : FTP Rx: " MDTM"
2021/05/28 16:30:28 DEBUG : FTP Rx: " UTF8"
2021/05/28 16:30:28 DEBUG : FTP Rx: " HASH"
2021/05/28 16:30:28 DEBUG : FTP Rx: " MLST Type*;Size*;Modify*;Perm;UNIX.mode;UNIX.owner;UNIX.group"
2021/05/28 16:30:28 DEBUG : FTP Rx: "211 end"
2021/05/28 16:30:28 DEBUG : FTP Tx: "TYPE I"
2021/05/28 16:30:28 DEBUG : FTP Rx: "200 type set."
2021/05/28 16:30:28 DEBUG : FTP Tx: "OPTS UTF8 ON"
2021/05/28 16:30:28 DEBUG : FTP Rx: "200 Using UTF-8."
2021/05/28 16:30:28 DEBUG : FTP Tx: "EPSV"
2021/05/28 16:30:28 DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||57709|)"
2021/05/28 16:30:28 DEBUG : FTP Tx: "MLSD hello"
2021/05/28 16:30:28 DEBUG : FTP Rx: "150 Opening Binary mode data connection."
2021/05/28 16:30:28 DEBUG : FTP Rx: "226 Transfer complete."
2021/05/28 16:30:28 DEBUG : FTP Tx: "EPSV"
2021/05/28 16:30:28 DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||57710|)"
2021/05/28 16:30:28 DEBUG : FTP Tx: "MLSD hello"
2021/05/28 16:30:28 DEBUG : FTP Rx: "150 Opening Binary mode data connection."
2021/05/28 16:30:28 DEBUG : FTP Rx: "226 Transfer complete."
2021/05/28 16:30:28 DEBUG : Sintel.mp4: Need to transfer - File not found at Destination
2021/05/28 16:30:28 DEBUG : FTP Tx: "EPSV"
2021/05/28 16:30:28 DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||57711|)"
2021/05/28 16:30:28 DEBUG : FTP Tx: "MLSD ."
2021/05/28 16:30:28 DEBUG : FTP Rx: "150 Opening Binary mode data connection."
2021/05/28 16:30:29 DEBUG : FTP Rx: "226 Transfer complete."
2021/05/28 16:30:29 DEBUG : FTP Tx: "EPSV"
2021/05/28 16:30:29 DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||57712|)"
2021/05/28 16:30:29 DEBUG : FTP Tx: "STOR hello/Sintel.mp4"
2021/05/28 16:30:29 DEBUG : FTP Rx: "150 Opening Binary mode data connection."
2021/05/28 16:30:34 DEBUG : FTP Rx: "226 Transfer complete."
2021/05/28 16:30:34 DEBUG : FTP Tx: "EPSV"
2021/05/28 16:30:34 DEBUG : FTP Rx: "229 Entering Extended Passive Mode (|||57713|)"
2021/05/28 16:30:34 DEBUG : FTP Tx: "MLSD hello"
2021/05/28 16:30:34 DEBUG : FTP Rx: "150 Opening Binary mode data connection."
2021/05/28 16:30:34 DEBUG : FTP Rx: "226 Transfer complete."
2021/05/28 16:30:34 ERROR : Sintel.mp4: Failed to copy: update getinfo: object not found
2021/05/28 16:30:34 ERROR : Attempt 1/1 failed with 1 errors and: update getinfo: object not found
2021/05/28 16:30:34 INFO  :
Transferred:      123.255M / 123.255 MBytes, 100%, 23.532 MBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         5.7s

2021/05/28 16:30:34 DEBUG : 3 go routines active
2021/05/28 16:30:34 Failed to copy: update getinfo: object not found

Folder "hello" - empty

5

perhaps try REMOTE:hello, not REMOTE:hello/

6

The same, the file is uploaded into the root.

7

Rclone quite clearly says store this file into the hello directory here.

If it arrives in the root then this sounds like a bug in the FTP server.

Does uploading to the root work OK?

Maybe you could do that then use rclone to move the files?

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.