I use rclone to move files from a FTP to S3. The FTP is very bulky with many directories and millions of files. Many files are push to the FTP server synchronously. The move rate is around 40 files per min that is so slow for the case.
When I use rclone copy i can reach 2500 files per min but I need to delete the copied files, too.
As I observed, the rclone delete has the same rate as rclone move. I could observe that the network bandwidth is full when i use rclone move or rclone copy. However, when i use rclone move, rclone stops for a while after getting files and dont send files right after getting them. Do rclone copy files from FTP to my local then send them to the S3 bucket? I saw that rclone copy has no stop, but rclone move stops for a while and this makes a lot of delay.
The rclone version is 1.65.2 on ubuntu 20.04
There are two remotes, one for the FTP and the other for S3 bucket.
I have tested the following flags but didn't achieve higher rate:
- --disable listR
Moreover, I have changed connections timeout for close, idle and shutdown.
Can anyone help me?
send all details you were asked for in the template including debug log from your transfer session. Then somebody can have a look and maybe suggest some improvements.
welcome to the forum,
for a deeper look including timestamps, use
fwiw, ftp is a terrible protocol, does that server support sftp/webdav or any other protocol?
Sorry, i should move the files from FTP. I have no option to change it.
at the end, what's your suggestion? if i cant enhance rclone, then i should develop a code for moving files.
You have not provided any details yet. Just a story. So impossible to suggest anything.
as compared to what? what other ftp copy tools have you tested and what were the results?
did you test
for move operation, need to figure out exactly where the slow down is.
so forget about S3 and uploading.
just move a single file from ftp to local using
can you ssh or remote into the ftp server?
might compress the source files into a tar and then move that
As I understand OP is saying that
rclone move is like 60x slower than
rclone copy. But no details what command is using so I am not sure what is going on.
thank you and thanks "kapitainsky" for your response and attention.
I figured out that the problem was because of very crowded ftp directory. I copied and deleted millions of files, then reactivated rclone move. Now, i have no problem and everything is smooth.
I found out that in a very bulky directory, rclone faces delay toward confirming delete.