Hi, I am using rclone to backup local files to an FTP server, and it mostly works fine. There are about 62k files to check by size.
The problem is that sometimes there are some intermittent errors in the communication with the server, despite the fact that I have set low-level-retries to one million. The two most common are:
- Couldn't move: Move NewObject failed: object not found
- error reading destination directory: 425 PASV: Address already in use
The first one appears at the start only, and does not occur on the second retry.
The second error appears 2-5 times on each full retry, each time at different files/directories.
This causes each full retry to fail for these random intermittent failures. Perhaps if I set the retries to 50 it would do a full cycle once without any error, but it would take too long (each full cycle is ~40 minutes).
But my main issue is that the full retry is a waste of time and bandwidth in this case, because only a few files are failing. So rclone could just retry the failed files only, not everything over again, and that would take max 30 seconds (these are checks that fail, not transfers).
If it is an issue remembering which files failed, because of memory needed if there is a large amount of them, there could be a limit of remembered files to retry (configurable), and if it's reached, the full retry cycle could be forced.
Also if any of the failed file fails again with retry, it could be put to the list again (or just not removed from the retry list), and retried until all succeed or file retries limit is reached.
I don't know why low-level-retries do not fix the issue, perhaps there are some 1-2 second connection issues, and low-level-retry does not wait between the attempts, and makes the million trials within fraction of a second.
I am using a 1.55.1 version with modifications without which my FTP server sill not function (I will use the latest official when the modifications are implemented in it).
Windows 10, 64-bit
FTP with crypt
rclone.exe "copy" "D:\" "**private**" "--backup-dir=**private**" "--config=**private**" "--password-command=**private**" "--filter-from=**private**" "--size-only" "--bwlimit=5M:50M" "--retries-sleep=10s" "--ftp-concurrency=1" "--transfers=1" "--retries=1" "--low-level-retries=1000000" "--max-backlog=100000" "-vv"]
[**private**] type = ftp host = **private** user = **private** port = 6621 pass = **private** explicit_tls = true disable_mfmt = true disable_tls13 = true encoding = Slash,Asterisk,Ctl,Dot concurrency = 3 [**private**_crypt] type = crypt remote = **private** directory_name_encryption = false password = **private**
Below is a fragment of the debug log showing the problem. Interestingly, the manual says with the -vv option rclone should print all low-level retries, but there don't seem to be any here.
2021/11/18 15:54:17 DEBUG : Programowanie/Lazarus-Win/HAWK/src/components/graphics32-master_MOD/Documentation/Source/Units/GR32_Backends/Interfaces/IDeviceContextSupport/_Body.htm: Sizes identical 2021/11/18 15:54:17 DEBUG : Projekty/MartinoEscapeRoom/magic-rooms-repo/controller/sources/soft_demobrd_v1.0_F746G/Debug/Middlewares/Third_Party/LwIP/src/core/ipv6/icmp6.o: Excluded 2021/11/18 15:54:17 ERROR : Programowanie/LBA/LBArchitect/Components/graphics32/Examples/Vcl/Resampling/Resamplers_Ex: error reading destination directory: 425 PASV: Address already in use 2021/11/18 15:54:17 DEBUG : Programowanie/LBA/LBArchitect/Components/graphics32/Examples/Vcl/Resampling/PixelF_Ex/createbundle.sh: Unchanged skipping 2021/11/18 15:54:17 DEBUG : Programowanie/C-STM32/DISCO_F746G/zabawa/zabawa/SW4STM32/zabawa/Drivers/Components/st7735/Release_Notes.html: Sizes identical