Failed to sync: 501 Not a directory

Help and Support
1

Hi there,

I've got an instance running on EC2 (Amazon Linux 2 AMI). On this instance I want to be pushing files out to an external FTP server.

rclone is able to connect to the external server fine but the transfer eventually fails due to 501 Not a Directory.

On my EC2 instance I have an S3 bucket mounted using S3FS so it appears as a local drive. I've attempted changing the source in rclone from this local mount of S3 as well as rclone's S3 implementation with the same outcome.

Here's the full dump of what I receive. I have redacted a couple of details:

[ec2-user@EC2 ~]$ rclone -vvv sync s3:/company-delivery/global/scheduler-2/ global:scheduler-2/
2021/02/18 08:58:12 DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "-vvv" "sync" "s3:/company-delivery/global/scheduler-2/" "global:scheduler-2/"]
2021/02/18 08:58:12 DEBUG : Using config file from "/home/ec2-user/.config/rclone/rclone.conf"
2021/02/18 08:58:12 DEBUG : Creating backend with remote "s3:/company-delivery/global/scheduler-2/"
2021/02/18 08:58:12 DEBUG : fs cache: renaming cache item "s3:/company-delivery/global/scheduler-2/" to be canonical "s3:company-delivery/global/scheduler-2"
2021/02/18 08:58:12 DEBUG : Creating backend with remote "global:scheduler-2/"
2021/02/18 08:58:12 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Connecting to FTP server
2021/02/18 08:58:13 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Waiting for checks to finish
2021/02/18 08:58:13 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Waiting for transfers to finish
2021/02/18 08:58:13 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Connecting to FTP server
2021/02/18 08:58:13 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Connecting to FTP server
2021/02/18 08:58:13 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Connecting to FTP server
2021/02/18 08:58:13 ERROR : em/sync/broadsign-manifest.json: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/sync" failed: 501 /scheduler-2/em: Not a directory.
2021/02/18 08:58:13 ERROR : em/sync/sports-q.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/sync" failed: 229 Entering Extended Passive Mode (|||50010|)
2021/02/18 08:58:13 ERROR : em/sync/sports-z.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/sync" failed: 227 Entering Passive Mode (xx,xx,xx,xxx,xxx,81)
2021/02/18 08:58:13 ERROR : roadside/sync/broadsign-manifest.json: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/roadside/sync" failed: 501 /scheduler-2/em: Not a directory.
2021/02/18 08:58:13 ERROR : em/sync/scheduler-2_global_em.js: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/sync" failed: 501 /scheduler-2/em: Not a directory.
2021/02/18 08:58:13 ERROR : em/sync/sports-m.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/sync" failed: 501 /scheduler-2/em: Not a directory.
2021/02/18 08:58:13 ERROR : em/sync/sports-e.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/sync" failed: 501 /scheduler-2/em: Not a directory.
2021/02/18 08:58:13 ERROR : roadside/sync/scheduler-2_global_roadside.js: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/roadside/sync" failed: 227 Entering Passive Mode (xx,xx,xx,xxx,xxx,86)
2021/02/18 08:58:13 ERROR : test/sync/broadsign-manifest.json: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/test/sync" failed: 501 /scheduler-2/roadside: Not a directory.
2021/02/18 08:58:14 ERROR : roadside/sync/sports-z.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/roadside/sync" failed: 229 Entering Extended Passive Mode (|||50009|)
2021/02/18 08:58:14 ERROR : roadside/sync/sports-g-greenpark.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/roadside/sync" failed: 227 Entering Passive Mode (xx,xx,xx,xxx,xxx,88)
2021/02/18 08:58:14 ERROR : roadside/sync/sports-q.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/roadside/sync" failed: 227 Entering Passive Mode (xx,xx,xx,xxx,xxx,80)
2021/02/18 08:58:14 ERROR : test/sync/scheduler-2_global_test.js: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/test/sync" failed: 227 Entering Passive Mode (xx,xx,xx,xxx,xxx,83)
2021/02/18 08:58:14 ERROR : em/onetime/index.html: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/em/onetime" failed: 501 /scheduler-2/roadside: Not a directory.
2021/02/18 08:58:14 ERROR : test/sync/sports-m.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/test/sync" failed: 501 /scheduler-2/roadside: Not a directory.
2021/02/18 08:58:14 ERROR : test/sync/sports-z.mp4: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/test/sync" failed: 227 Entering Passive Mode (xx,xx,xx,xxx,xxx,84)
2021/02/18 08:58:14 ERROR : roadside/onetime/index.html: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/roadside/onetime" failed: 501 /scheduler-2/test: Not a directory.
2021/02/18 08:58:14 ERROR : test/onetime/index.html: Failed to copy: Put mkParentDir failed: mkdir "scheduler-2/test/onetime" failed: 426 Data connection unexpectedly closed.
2021/02/18 08:58:14 ERROR : ftp://ftp.externalserver.com:21/scheduler-2: not deleting files as there were IO errors
2021/02/18 08:58:14 ERROR : ftp://ftp.externalserver.com:21/scheduler-2: not deleting directories as there were IO errors
2021/02/18 08:58:14 ERROR : Attempt 1/3 failed with 18 errors and: Put mkParentDir failed: mkdir "scheduler-2/test/onetime" failed: 426 Data connection unexpectedly closed.
2021/02/18 08:58:14 ERROR : : error reading destination directory: 426 Data connection unexpectedly closed.
2021/02/18 08:58:14 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Waiting for checks to finish
2021/02/18 08:58:14 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Waiting for transfers to finish
2021/02/18 08:58:14 ERROR : ftp://ftp.externalserver.com:21/scheduler-2: not deleting files as there were IO errors
2021/02/18 08:58:14 ERROR : ftp://ftp.externalserver.com:21/scheduler-2: not deleting directories as there were IO errors
2021/02/18 08:58:14 ERROR : Attempt 2/3 failed with 1 errors and: 426 Data connection unexpectedly closed.
2021/02/18 08:58:14 ERROR : : error reading destination directory: 501 /scheduler-2/test: Not a directory.
2021/02/18 08:58:14 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Waiting for checks to finish
2021/02/18 08:58:14 DEBUG : ftp://ftp.externalserver.com:21/scheduler-2: Waiting for transfers to finish
2021/02/18 08:58:14 ERROR : ftp://ftp.externalserver.com:21/scheduler-2: not deleting files as there were IO errors
2021/02/18 08:58:14 ERROR : ftp://ftp.externalserver.com:21/scheduler-2: not deleting directories as there were IO errors
2021/02/18 08:58:14 ERROR : Attempt 3/3 failed with 1 errors and: 501 /scheduler-2/test: Not a directory.
2021/02/18 08:58:14 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         1.8s

2021/02/18 08:58:14 DEBUG : 5 go routines active
2021/02/18 08:58:14 Failed to sync: 501 /scheduler-2/test: Not a directory.

I am able to connect to this FTP server using FileZilla to create directories no problem. In an attempt to troubleshoot this I did temporarily enable all inbound and outbound traffic to this VM within AWS however I received the same error.

Any help is greatly appreciated.

Cheers,
Jon

2

hello and welcome to the forum.

rclone can only connect to passive ftp servers.

rclone sync will delete files on the dest and the problem seems to be with the ftp, not s3, so when testing, try using rclone ls, rclone lsd, rclone mkdir using the ftp server.

about folder scheduler-2, is that accessed from the root folder, or is that a sub-folder of the home folder?
from the docs
"If the path does not begin with a / it is relative to the home directory of the user. An empty path remote: refers to the user's home directory."

was that test done on the ec2 instance?

when you posted, there was a template of questions, that help us help you....

3

Hi,

Thanks for your response.

Apologies for leaving some gaps in my post.

I believe that the destination server is passive.

I'm able to transfer files successfully using FileZilla on MacOS using the same credentials, just not with rclone.

Correct, I believe the issue is with the external FTP server. I ran rclone ls global:/ and it came back successfully with a full list of files.

The scheduler-2 folder is accessed from the root folder once connected to the FTP server. I've been using FileZilla on MacOS and the credentials used do have permission to create directories here.

I've attempted to sync files using global:/scheduler-2/ as well as global:scheduler-2/. I have also tried rclone copy as well as rclone sync.

Running rclone mkdir global:/scheduler-2 is successful and the directory is made on the FTP server. Once creating that directory the copy still fails as it's not able to create the subdirectories.

Cheers,
Jon

4

can you confirm that?

i am not a ftp expert but rclone seems to be having an issue, not being able to use passive mode.
Entering Passive Mode
Entering Extended Passive Mode

5

I'll confirm with the administrators of that server tomorrow but it does seem it's passive. I've connected to their server via the standard ftp command line on the same EC2 instance. I was able to toggle passive on and off as well as create directories.

I'm feeling it could be some kind of firewall configuration but I don't understand how standard ftp can create a directory but rclone can't.

ftp> passive
Passive mode off.
ftp> passive
Passive mode on.
ftp> mkdir test1
257 "/test1" directory created.
ftp> cd test1
250 Directory changed to /test1
6

can you post the rclone config file, redacting sensitive info.

7

Sure, here it is.

[global]
type = ftp
host = ftp.externalserver.com
user = [username]
pass = [password]

[s3]
type = s3
provider = AWS
env_auth = true
region = eu-west-1
location_constraint = eu-west-1
acl = bucket-owner-full-control
8

As you asked for that I had a thought. I've recreated global as SFTP in rclone config.

It connected and copied all files successfully!

Very strange, again, that standard ftp could do it in plain FTP yet rclone required SFTP.

Thanks a lot for your help!

9

yeah,
i was going to suggest sftp, more secure and rclone can checksum files.

10

It's rare that I use FTP anywhere that isn't FileZilla on my Mac so never really give a thought to it being secure SFTP and not actually just plain FTP.

Anyway, I hope my frustrating couple of days can be of use to others in the future experiencing the same issue!

Cheers,
Jon

11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.