might test --no-update-modtime --no-update-dir-modtime
fwiw, ftp is a poor choice for backups and rclone support for ftp has always had too many issues.
i would use smb to enable file transfer verification using checksums.
i have a howto about that - https://forum.rclone.org/t/how-to-access-smb-samba-with-rclone/42754
fwiw, plugging in an external usb to an internet facing router might not be a secure option.
can purchase a raspberry pi zero and plug usb harddrive into that.