Rclone FTPS (Explicit) + rclone touch / empty files + ProFTPD => Unable to build data connection: Operation not permitted (with 1.59.x)

icom · September 6, 2022, 2:35pm

What is the problem you are having with rclone?

I have created an Rclone FTPS connection to another server, but I cannot use the "rclone touch" command, nor can I upload an empty file (with copy, copyto, move, moveto).

I have reported the same issue back with Rclone version 1.54.0 here. Now trying out Rclone 1.59.0 or 1.59.1 it seems the error is back.
After it was fixed the last time, the error did not occur until (including) version 1.58.1.

When using 1.58.1 -> It works.
When using 1.59 / 1.59.1 -> It does not work.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

os/version: ubuntu 22.04 (64 bit)
os/kernel: 5.15.0-43-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18.5
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

FTP

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone touch my-ftp-dev23:/files/my-test-file --retries 1 -vv

The rclone config contents with secrets removed.

[my-ftp-dev23]
type = ftp
host = dev23.keymachine.de
user = test
pass = <my-secret>
explicit_tls = true

A log from the command with the `-vv` flag

2022/09/06 14:26:19 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "touch" "my-ftp-dev23:/files/my-test-file" "--retries" "1" "-vv"]
2022/09/06 14:26:19 DEBUG : Creating backend with remote "my-ftp-dev23:/files/"
2022/09/06 14:26:19 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/09/06 14:26:19 DEBUG : ftp://dev23.keymachine.de:21/files: Connecting to FTP server
2022/09/06 14:26:20 DEBUG : Touch time 2022-09-06 14:26:20.238751756 +0000 UTC m=+0.515825587
2022/09/06 14:26:20 DEBUG : ftp://dev23.keymachine.de:21/files: Touching (creating) "my-test-file"
2022/09/06 14:26:21 DEBUG : ftp://dev23.keymachine.de:21/files: Connecting to FTP server
2022/09/06 14:26:21 DEBUG : my-test-file: Removed after failed upload: 1 error occurred:
	* 425 Unable to build data connection: Die Operation ist nicht erlaubt

2022/09/06 14:26:21 ERROR : Attempt 1/1 failed with 1 errors and: failed to touch (create): update stor: 1 error occurred:
	* 425 Unable to build data connection: Die Operation ist nicht erlaubt

2022/09/06 14:26:21 DEBUG : 2 go routines active
2022/09/06 14:26:21 DEBUG : ftp://dev23.keymachine.de:21/files: closing 1 unused connections
2022/09/06 14:26:21 Failed to touch: failed to touch (create): update stor: 1 error occurred:
	* 425 Unable to build data connection: Die Operation ist nicht erlaubt

("Die Operation ist nicht erlaubt" means "Operation not permitted")

Thanks for your help again!
Alex

ncw · September 20, 2022, 12:10pm

I've just fixed this again!

Try the beta linked from here

github.com/rclone/rclone

FTP upload always delete file after upload when file size >40k

opened 06:01AM - 08 Sep 22 UTC

sriccio

Remote: FTP

#### The associated forum post URL from `https://forum.rclone.org` No post on f…orum.rclone.org (yet) #### What is the problem you are having with rclone? It looks like FTP uploads fails for files bigger than 40k. The file is transfered then immediately deleted (I guess this is because the resulting file size is getting random, see below). For smaller files like, for example around 40k it works correctly. Note: I first discovered the issue while using rclone in mount mode with the same config and as only some files were uploaded to the target server I begin troubleshooting the issue using `copy` method. Edit: I forgot to add that I've checked everything was working correctly uploading these files to the same servers and from the same origin box but using `lftp`. It showed no issue. #### What is your rclone version (output from `rclone version`) rclone v1.59.1 - os/version: ubuntu 22.04 (64 bit) - os/kernel: 5.15.0-47-generic (x86_64) - os/type: linux - os/arch: amd64 - go/version: go1.18.5 - go/linking: static - go/tags: none #### Which OS you are using and how many bits (e.g. Windows 7, 64 bit) Ubuntu Linux 64 bit (see above) #### Which cloud storage system are you using? (e.g. Google Drive) FTP Server (pure-ftpd) running on another Ubuntu box. But I also tried with another FTP server on another machine and the result is the same. #### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`) ```rclone copy test.40K web23:/ -vv --log-file bug.log``` and ```rclone copy test.41K web23:/ -vv --log-file bug.log``` For the records, the `rclone.conf` contains: ``` # cat ~/.config/rclone/rclone.conf [web23] type = ftp host = web23.somedomain.com user = user port = 21 pass = ThisIsSomeUberRedactedSecretPass explicit_tls = true concurrency = 3 no_check_certificate = true disable_epsv = false disable_mlsd = false ``` #### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`) Generating the files ``` dd if=/dev/urandom of=test.40K bs=40K count=1 dd if=/dev/urandom of=test.41K bs=41K count=1 ``` Result ``` -rw-r--r-- 1 root root 40960 Sep 8 07:37 test.40K -rw-r--r-- 1 root root 41984 Sep 8 07:52 test.41K ``` Log of `copy` of the 40k file ``` 2022/09/08 07:49:53 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "copy" "test.40K" "web23:/" "-vv" "--log-file" "bug.log"] 2022/09/08 07:49:53 DEBUG : Creating backend with remote "test.40K" 2022/09/08 07:49:53 DEBUG : Using config file from "/root/.config/rclone/rclone.conf" 2022/09/08 07:49:53 DEBUG : fs cache: adding new entry for parent of "test.40K", "/root/test" 2022/09/08 07:49:53 DEBUG : Creating backend with remote "web23:/" 2022/09/08 07:49:53 DEBUG : ftp://web23.somedomain.com:21: Connecting to FTP server 2022/09/08 07:49:53 DEBUG : test.40K: Need to transfer - File not found at Destination 2022/09/08 07:49:53 INFO : test.40K: Copied (new) 2022/09/08 07:49:53 INFO : Transferred: 40 KiB / 40 KiB, 100%, 0 B/s, ETA - Transferred: 1 / 1, 100% Elapsed time: 0.1s ``` FTP Server log of the 40k file upload (size matches) ``` Sep 8 07:36:33 web23 pure-ftpd[1867956]: (user@2a00:a500:0:93::dead:beef) [NOTICE] /home/user//test.40K uploaded (40960 bytes, 39217.43KB/sec) ``` Log for attempt of `copy` of the `41k` file ``` 2022/09/08 07:49:56 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "copy" "test.41K" "web23:/" "-vv" "--log-file" "bug.log"] 2022/09/08 07:49:56 DEBUG : Creating backend with remote "test.41K" 2022/09/08 07:49:56 DEBUG : Using config file from "/root/.config/rclone/rclone.conf" 2022/09/08 07:49:56 DEBUG : fs cache: adding new entry for parent of "test.41K", "/root/test" 2022/09/08 07:49:56 DEBUG : Creating backend with remote "web23:/" 2022/09/08 07:49:56 DEBUG : ftp://web23.somedomain.com:21: Connecting to FTP server 2022/09/08 07:49:56 DEBUG : test.41K: Need to transfer - File not found at Destination 2022/09/08 07:49:57 DEBUG : ftp://web23.somedomain.com:21: Connecting to FTP server 2022/09/08 07:49:57 DEBUG : test.41K: Removed after failed upload: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 4.17 Mbytes per second 2022/09/08 07:49:57 ERROR : test.41K: Failed to copy: update stor: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 4.17 Mbytes per second 2022/09/08 07:49:57 ERROR : Attempt 1/3 failed with 1 errors and: update stor: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 4.17 Mbytes per second 2022/09/08 07:49:57 DEBUG : test.41K: Need to transfer - File not found at Destination 2022/09/08 07:49:58 DEBUG : ftp://web23.somedomain.com:21: Connecting to FTP server 2022/09/08 07:49:58 DEBUG : test.41K: Removed after failed upload: 1 error occurred: * 451 Transfer aborted 0.002 seconds (measured here), 9.13 Mbytes per second 2022/09/08 07:49:58 ERROR : test.41K: Failed to copy: update stor: 1 error occurred: * 451 Transfer aborted 0.002 seconds (measured here), 9.13 Mbytes per second 2022/09/08 07:49:58 ERROR : Attempt 2/3 failed with 1 errors and: update stor: 1 error occurred: * 451 Transfer aborted 0.002 seconds (measured here), 9.13 Mbytes per second 2022/09/08 07:49:58 DEBUG : test.41K: Need to transfer - File not found at Destination 2022/09/08 07:49:59 DEBUG : ftp://web23.somedomain.com:21: Connecting to FTP server 2022/09/08 07:49:59 DEBUG : test.41K: Removed after failed upload: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 16.32 Mbytes per second 2022/09/08 07:49:59 ERROR : test.41K: Failed to copy: update stor: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 16.32 Mbytes per second 2022/09/08 07:49:59 ERROR : Attempt 3/3 failed with 1 errors and: update stor: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 16.32 Mbytes per second 2022/09/08 07:49:59 INFO : Transferred: 123 KiB / 123 KiB, 100%, 40.998 KiB/s, ETA 0s Errors: 1 (retrying may help) Elapsed time: 3.4s 2022/09/08 07:49:59 DEBUG : 4 go routines active 2022/09/08 07:49:59 DEBUG : ftp://web23.somedomain.com:21: closing 1 unused connections 2022/09/08 07:49:59 Failed to copy: update stor: 1 error occurred: * 451 Error during read from data connection Transfer aborted 0.002 seconds (measured here), 16.32 Mbytes per second ``` FTP Server log of the 41k file upload attempt (file size is starting to be random) ``` Sep 8 07:49:56 web23 pure-ftpd[1941648]: (user@2a00:a500:0:93::dead:beef) [NOTICE] /home/user//test.41K uploaded (7116 bytes, 4271.27KB/sec) Sep 8 07:49:57 web23 pure-ftpd[1941672]: (user@2a00:a500:0:93::dead:beef) [NOTICE] Deleted /home/user/test.41K Sep 8 07:49:57 web23 pure-ftpd[1941672]: (user@2a00:a500:0:93::dead:beef) [NOTICE] /home/user//test.41K uploaded (17790 bytes, 9345.63KB/sec) Sep 8 07:49:58 web23 pure-ftpd[1941723]: (user@2a00:a500:0:93::dead:beef) [NOTICE] Deleted /home/user//test.41K Sep 8 07:49:58 web23 pure-ftpd[1941723]: (user@2a00:a500:0:93::dead:beef) [NOTICE] /home/user//test.41K uploaded (32768 bytes, 16710.37KB/sec) Sep 8 07:49:59 web23 pure-ftpd[1941757]: (user@2a00:a500:0:93::dead:beef) [NOTICE] Deleted /home/user//test.41K ``` #### How to use GitHub * Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue. * Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue. * Subscribe to receive notifications on status change and new comments.

Hopefully that should fix the problems.

system · October 20, 2022, 12:10pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.