on my experimental branch rclone will issue the MFMT/MDTM optional command to set file time as soon as ftp server adverises it.
the mainline doesn't support this feature yet.
looks like your server advertises it but prohibits afterwards, but i'm not sure.
could you run your command with -vv --log-file log --dump bodies and attach the log below?
rfc standardizes something.
wikipedia writes something else.
then come vsftpd developers and name the set file time command as MDTM time path.
therefore i added it as optional, enabled only by flag.
authors of ftp servers are strange folks, you can send them the link to wikipedia. dont ask me.
btw here is my config for hetzner sftp, it works fine for me
[sbsftp]
type = sftp
host = u123456.your-storagebox.de
user = u123456
pass = heil-hetzner
md5sum_command = none
sha1sum_command = none
Here is my log. Thanks.
rclone.log (42.4 KB)
Shouldn't rclone respect --no-update-modtime flag? I have the flag, but rclone tries to update the modtime anyway.
That's for if files are identical:
--no-update-modtime Don't update destination mod-time if files identical.
With a new copy, it will update the mod time.
It's not a new copy. The files are there, and have identical sizes. Does it assume the files are different, because they have different modtimes?
Which part shows the files are there? I only see new file transfers.
When I login with FileZilla, the files are there. I use the same remote directory for these tests, and the files are there all the time.
For some of them rclone detects modtime difference, as in lines 113-119 of my log, but not for others, but it transfers them anyway. I don't understand why it does that.
FTP is a really tough one to use as it's not supported very well via the GO libraries unfortunately.
So if the times are different, it's going to upload a file, which is what you are seeing.
1 Like
I'll take a look into the log.
I think the issue with --no-update-modtime and FTP is okay for now. Modtime is not implenented mainstream. This branch is experimental and alpha.
I will clear stuff like flags only after I have low-level modtime working basically and setup fuller testing with pureftpd,proftpd,vsftpd. I'll get to it in September.
ftp in rclone is incomplete. with your help we may move it a little forward. for now you have at least lftp on linux and filezilla client on windows.
I can do more tests if that helps with the development. Just tell me what you need.
For now with all these problems I am starting to think it would be less hassle to write my own software to to do the job 
this comment is about mfmt issue
i will analyze other issues (time comparison and --no-update-xxx) later
2021/07/19 18:55:06 DEBUG : FTP Rx: " MFMT"
2021/07/19 18:55:06 DEBUG : FTP Rx: " MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.groupname*;UNIX.mode*;UNIX.owner*;UNIX.ownername*;"
ftp server advertises MFMT
ftp backend in mainstream rclone would just ignore it
i added code to pick it so we can have modtime change support in rclone
2021/07/19 18:55:06 DEBUG : FTP Tx: "MFMT 20060918115522 /test_rclone/current/project1.compiled"
my code decides to use MFMT to set transferred file time (mainstream ftp backend would not even try, it just reports modtime unsupported).
probably it should obey --no-update-modtime and skip setting time but it's another story
2021/07/19 18:55:06 DEBUG : FTP Rx: "226 Transfer complete"
2021/07/19 18:55:06 DEBUG : FTP Tx: "MKD /test_rclone/old_2021-07-19_18-55"
the log also shows parallel actions from other threads interleaved
note: ftp backend has internal concurrency, so setting --transfers 1 with it is not enough, we'd better use
--transfers 1 --checkers 1 --ftp-concurrency 1 --log-format date,time,microseconds --dump bodies -vv --log-file log
but even without that i think i can guess what's really related to your issue as i worked with rclone ftp recently
2021/07/19 18:55:06 DEBUG : FTP Rx: "501 /test_rclone/current/project1.compiled: Operation not permitted"
2021/07/19 18:55:06 ERROR : project1.compiled: Failed to copy: SetModTime: 501 /test_rclone/current/project1.compiled: Operation not permitted
ftp server returns error to MFMT contrary to advertising it before
ok, yet another ftp server madness
we already had disable_epsv and disable_mlsd
to solve your case i'm gonna add disable_mfmt
btw, proftpd in 1.3.2 introduced tls support, but if it was compiled with openssl, it was buggy, because stubborn openssl guys introduced quirks in tls transactions timings.
boringssl from google did not have these quirks. after 2 years of quirrels with openssl devs (i added links to their hot threads on this forum) some bold proftpd guy found a way to mitigate openssl quirks, so in proftpd 1.3.7a and later it looks better (in my config for hetzner i don't need to disable tls13, it just workd).
so i added disable_tls13 to mitigate older proftpd.
to understand why rclone copied some files but not others i'm also gonna add output of parsed MLSD (listing) to see what times/sizes ftp backend sees in reality. some fkng ftp servers just don't include file year in the listing so my code tries its best and can default to current year after heuristics, and this can distort the time seen. mainstream rclone just declared modtime unsupported, at all.
1 Like
this is how i started this experimental rclone branch. it works fine for me and supports modtime. 3 cases: hetzner's proftpd (even with tls on), pureftpd embedded in keenetic routers (i don't need tls for them because their exchanges are hidden behind vpn) and vsftpd on ubuntu 20.04 vps configured by me.
btw vsftpd does not have MFMT (in 2021!) but can be configured to use MDTM with non-standard syntax for that (these guys are mad) so i added the writing_mdtm quirk.
but writing something for 2-3 use cases, for myself (i understand my pros and cons, i know rclone, i don't need no-update-xxx btw) and something for a wider audience with a miryad combinations of flags/versions/settings are two different stories. especially boring is writing documentation explaining the quirks, who should use them, why and when, without hate speech
1 Like
Thank you for doing this. I am hoping to be able to use it for my needs too. With the MFMT issue, perhaps I will be able to turn off reporting support for it, and it won't be a problem even without flag. But my FTP server is maintained by another person, so it's not quick and easy
Looks like I cannot disable MFMT on the sever without disabling other necessary commands along with it.
I will look forward to a flag to disable use of MFMT in rclone, if that's not a big problem.
this release has new flag --ftp-disable-mfmt
if you add it, ftp backend will ignore the MFMT command advertised by your upstream ProFtpd.
consequently, backend will report to higher levels that modtime is unsupported.
they will show modtime in listings but will not try to compare it for sync, looking only on size/name.
updating file time after upload will not be attempted at all. uploaded files will be stamped by the time of upload (just like now when backend attempts to apply MFMT to make uploaded file have modtime of source but hits errors).
disclaimer: just married compiled, not tested.
if you know golang, check me here Commits ¡ ivandeex/ftp ¡ GitHub and here Commits ¡ ivandeex/rclone ¡ GitHub, if not - forget it, just try the build.
as far as i remember you will also need --ftp-disable-tls13 supported by this branch (you probably have proftpd older than 1.3.7, built with quirky openssl).
try to use bare minimum of flags if possible. if it works without --ftp-tls-cache-size, pls omit it. thx
p.s. i will later think why --no-update-modtime didn't stop SetModTime, whether it's correct.
1 Like
Thank you, it works, and surprisingly, tls-cache-size is not necessary any more. Have you found a fix for TLS without the cache? With the mainstream version the problem is still there, so it's not like my server fixed itself.
Yes, I have ProFTPD 1.3.6, because it's the latest stable version. My friend who runs the server doesn't want to use unstable versions because of potential bugs (it's a server for backups, so it should be as reliable as possible).
I don't know golang, but it looks similar to C. I would probably get it if I needed to.
stable proftpd 1.3.6 has problems with tls13, so we add the disable-tls13 flag for rclone to refuse tls13 proposals and downgrade connection to tls12. if you run your client without the flag and it breaks, we will know for sure that it helped in your case.
unstable proftpd 1.3.7a made improvements with tls13 so we don't need that flag (i guess hetzner has it, at least i don't need the flag).
tls-cache-size was an attempt to mitigate authentication of tls ftp data streams by the tls preshared-key from the tls ftp command stream optionally requested by some ftp servers.
this attempt has not been confirmed yet, i guess this will need some tls changes on golang side to be fully resolved.
as we can see your ftp server does not request it (as of the time of last probe).
besides tls changes this branch has modtime patches to make rclone pick optional MFMT,MDTM,MLSD commands if advertised by ftp server (so rclone ftp can read and set modtime),
though in your case this made things worse by uncovering a quirk with your ftp server settings, so we added flag --ftp-disable-mfmt to revert this feature.
also this branch has patches changing the way rclone handles timeouts between ftp data and control streams.
this change can improve uploads of large (>1G) files preventing spurious Close timed out (reported as EOF for TLS streams) errors.
this part is unfinished and needs thorough testing and polishing. I don't know if it helped in your use-case. (though it helped here FTP Upload Timed Out - #9 by ivandeex)
1 Like
I just found the --size-only flag, which makes rclone ignore the modtime entirely, even on backends that support it. If I use it, rclone will not use the MFMT at all, which means that you don't have to make a special switch for it
Also, for those who care about modtimes, but have problems with MFMT, like me: with the --update flag rclone will update files on destination only if the source modtime is newer, and will not try to update destination file's modtime spearately (it will update automatically when the new file version is uploaded). This is ideal for me.
EDIT: I was mistaken. The --size-only flag only affects detecting file changes. If rclone detects a file has changed, it will try to update its modtime. Without the flag it thinks all the files have changed and updates their modtimes, that's why I thought it disabled MFMT, when no file have changed in size.
But --update seems to work fine in this regard.
EDIT 2: looks like --update alone does not prevent MFMT use too. Sorry for the confusion. The --ftp-disable-mfmt flag is necessary in the end.
1 Like