Using rclone to SFTP to AWS Transfer Family

What is the problem you are having with rclone?

I have an SFTP server setup with AWS Transfer Family. I am able to connect to it via FileZilla and from the CLI by using sftp. But I've tried to setup rclone to connect to this as SFTP, and I just cannot seem to get anywhere. It seems to be authenticating but just can't seem to get past that point.

Running version: 1.58.1
Trying to connect to: AWS EFS via SFTP via AWS Transfer

rclone ls gfrclone:/Events/01/ -v --log-file=rclog

The rclone config contents with secrets removed.

[gfrclone]
type = sftp
host = rclone.mydomain.com
user = rcloneupload
key_file = /home/username/.ssh/rcloneupload.pem
md5sum_command = none
sha1sum_command = none

A log from the command with the -vv flag =

2022/05/26 22:42:13 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "ls" "gfrclone:/Events/01/" "-vv" "--log-file=rclog"]
2022/05/26 22:42:13 DEBUG : Creating backend with remote "gfrclone:/Events/01/"
2022/05/26 22:42:13 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2022/05/26 22:42:14 DEBUG : sftp://rcloneupload@rclone.mydomain.com:22//Events/01/: New connection 192.168.3.207:36126->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/26 22:42:15 ERROR : : error listing: directory not found
2022/05/26 22:42:15 DEBUG : 11 go routines active
2022/05/26 22:42:15 Failed to ls with 2 errors: last error was: directory not found
2022/05/26 22:42:52 ERROR : : error listing: directory not found
2022/05/26 22:42:52 Failed to ls with 2 errors: last error was: directory not found

Try removing the leading / - that means go to the root directory in the SFTP backend which you probably don't want. So

"gfrclone:Events/01/"

That worked!
But now when I'm actually copying I get a lot of errors. Looks like 'connection lost'. Not sure if it's AWS Transfer that just isn't playing nice.

2022/05/27 08:36:28 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "copy" "/media/Events/01/915/" "gfrclone:test/" "--size-only" "-P" "--checkers"$
2022/05/27 08:36:28 DEBUG : Creating backend with remote "/media/Events/01/915/"
2022/05/27 08:36:28 DEBUG : Using config file from "/home/glossyadmin/.config/rclone/rclone.conf"
2022/05/27 08:36:28 DEBUG : Creating backend with remote "gfrclone:test/"
2022/05/27 08:36:28 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43448->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:30 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Using absolute root directory "/fs-02fc184c19aaa11ab/test"
2022/05/27 08:36:31 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43454->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:31 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43456->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:31 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43458->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:31 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43460->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:31 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43462->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0001.jpg: Sizes differ (src 1213542 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0003.jpg: Sizes differ (src 1636217 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0006.jpg: Sizes differ (src 1646563 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0004.jpg: Sizes differ (src 1824525 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0010.jpg: Sizes differ (src 1323857 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0012.jpg: Sizes differ (src 1163291 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0005.jpg: Sizes differ (src 1338585 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0013.jpg: Sizes differ (src 1129267 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0007.jpg: Sizes differ (src 1116453 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0015.jpg: Sizes differ (src 1122379 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0011.jpg: Sizes differ (src 1281356 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0008.jpg: Sizes differ (src 1421586 vs dst 0)
2022/05/27 08:36:31 DEBUG : 012/HiRes/1393nr_220515_915_012_0009.jpg: Sizes differ (src 1708270 vs dst 0)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0004.jpg: Sizes differ (src 48803 vs dst 32768)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0005.jpg: Sizes differ (src 46350 vs dst 32768)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0007.jpg: Sizes differ (src 46905 vs dst 32768)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0006.jpg: Sizes differ (src 56700 vs dst 32768)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0010.jpg: Sizes differ (src 53489 vs dst 32768)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0009.jpg: Sizes differ (src 53835 vs dst 32768)
2022/05/27 08:36:34 DEBUG : 008/Preview/1393nr_220514_915_008_0008.jpg: Sizes differ (src 52939 vs dst 32768)
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43464->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0001.jpg: Sizes identical
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0002.jpg: Sizes identical
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0009.jpg: Sizes identical
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0010.jpg: Sizes identical
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0009.jpg: Unchanged skipping
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0024.jpg: Sizes identical
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0024.jpg: Unchanged skipping
2022/05/27 08:36:34 DEBUG : 008/Thumbs/1393nr_220514_915_008_0023.jpg: Unchanged skipping
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Waiting for checks to finish
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Waiting for transfers to finish
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43466->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:34 INFO  : 043/Thumbs/1393nr_220514_915_043_0002.jpg: Copied (new)
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:34 DEBUG : 043/Preview/1393nr_220514_915_043_0002.jpg: Failed to remove: connection lost
2022/05/27 08:36:34 ERROR : 043/Preview/1393nr_220514_915_043_0002.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:34 ERROR : 033/HiRes/1393nr_220515_915_033_0001.jpg: Failed to copy: Update Create failed: connection lost
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:34 ERROR : 033/HiRes/1393nr_220515_915_033_0003.jpg: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: connection $
2022/05/27 08:36:34 DEBUG : 043/Preview/1393nr_220514_915_043_0001.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:34 ERROR : 043/Preview/1393nr_220514_915_043_0001.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:34 ERROR : 033/Preview/1393nr_220515_915_033_0001.jpg: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: connectio$
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43468->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:34 DEBUG : 033/HiRes/1393nr_220515_915_033_0002.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:34 ERROR : 033/HiRes/1393nr_220515_915_033_0002.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:34 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43470->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:35 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43472->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:35 INFO  : 033/Preview/1393nr_220515_915_033_0003.jpg: Copied (new)
2022/05/27 08:36:35 INFO  : 033/Thumbs/1393nr_220515_915_033_0001.jpg: Copied (new)
2022/05/27 08:36:36 INFO  : 033/Thumbs/1393nr_220515_915_033_0002.jpg: Copied (new)
2022/05/27 08:36:36 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:36 ERROR : 033/Thumbs/1393nr_220515_915_033_0003.jpg: Failed to copy: Update SetModTime failed: SetModTime stat failed: stat failed: connection lost
2022/05/27 08:36:36 INFO  : 043/Thumbs/1393nr_220514_915_043_0001.jpg: Copied (new)
2022/05/27 08:36:36 INFO  : 033/Thumbs/1393nr_220515_915_033_0004.jpg: Copied (new)
2022/05/27 08:36:36 DEBUG : 043/HiRes/1393nr_220514_915_043_0002.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:36 ERROR : 043/HiRes/1393nr_220514_915_043_0002.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:37 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: New connection 192.168.3.207:43474->52.45.205.95:22 to "SSH-2.0-AWS_SFTP_1.1"
2022/05/27 08:36:37 DEBUG : 043/HiRes/1393nr_220514_915_043_0001.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:37 ERROR : 043/HiRes/1393nr_220514_915_043_0001.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:37 DEBUG : 033/HiRes/1393nr_220515_915_033_0004.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:37 ERROR : 033/HiRes/1393nr_220515_915_033_0004.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:38 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:38 ERROR : 033/Preview/1393nr_220515_915_033_0002.jpg: Failed to copy: Update SetModTime failed: SetModTime stat failed: stat failed: connection lost
2022/05/27 08:36:38 INFO  : 033/Preview/1393nr_220515_915_033_0004.jpg: Copied (new)
2022/05/27 08:36:40 DEBUG : 030/Preview/1393nr_220514_915_030_0001.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:40 ERROR : 030/Preview/1393nr_220514_915_030_0001.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:40 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:40 ERROR : 030/Preview/1393nr_220514_915_030_0006.jpg: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: connectio$
2022/05/27 08:36:41 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:41 ERROR : 030/Preview/1393nr_220514_915_030_0009.jpg: Failed to copy: Put mkParentDir failed: mkdir dirExists failed: dirExists stat failed: connectio$
2022/05/27 08:36:41 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost
2022/05/27 08:36:41 ERROR : 030/Preview/1393nr_220514_915_030_0004.jpg: Failed to copy: Update Create failed: connection lost
2022/05/27 08:36:41 DEBUG : 030/Preview/1393nr_220514_915_030_0003.jpg: Removed after failed upload: connection lost
2022/05/27 08:36:41 ERROR : 030/Preview/1393nr_220514_915_030_0003.jpg: Failed to copy: Update ReadFrom failed: connection lost
2022/05/27 08:36:41 DEBUG : sftp://rcloneupload@rclone.glossyfinish.com:22/test/: Connection failed, closing: connection lost

I don't know anything about AWS transfer, but it is conceivable it has a number of connections limit - if it does, try turning down --checkers and or --transfers - each one will use a connection.