What is the problem you are having with rclone?
Hi, I am using rclone to backup local files to an FTP server, and it mostly works fine. There are about 62k files to check by size.
The problem is that sometimes there are some intermittent errors in the communication with the server, despite the fact that I have set low-level-retries to one million. The two most common are:
- Couldn't move: Move NewObject failed: object not found
- error reading destination directory: 425 PASV: Address already in use
The first one appears at the start only, and does not occur on the second retry.
The second error appears 2-5 times on each full retry, each time at different files/directories.
This causes each full retry to fail for these random intermittent failures. Perhaps if I set the retries to 50 it would do a full cycle once without any error, but it would take too long (each full cycle is ~40 minutes).
But my main issue is that the full retry is a waste of time and bandwidth in this case, because only a few files are failing. So rclone could just retry the failed files only, not everything over again, and that would take max 30 seconds (these are checks that fail, not transfers).
If it is an issue remembering which files failed, because of memory needed if there is a large amount of them, there could be a limit of remembered files to retry (configurable), and if it's reached, the full retry cycle could be forced.
Also if any of the failed file fails again with retry, it could be put to the list again (or just not removed from the retry list), and retried until all succeed or file retries limit is reached.
I don't know why low-level-retries do not fix the issue, perhaps there are some 1-2 second connection issues, and low-level-retry does not wait between the attempts, and makes the million trials within fraction of a second.
What is your rclone version (output from rclone version
)
I am using a 1.55.1 version with modifications without which my FTP server sill not function (I will use the latest official when the modifications are implemented in it).
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10, 64-bit
Which cloud storage system are you using? (eg Google Drive)
FTP with crypt
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone.exe "copy" "D:\" "**private**" "--backup-dir=**private**" "--config=**private**" "--password-command=**private**" "--filter-from=**private**" "--size-only" "--bwlimit=5M:50M" "--retries-sleep=10s" "--ftp-concurrency=1" "--transfers=1" "--retries=1" "--low-level-retries=1000000" "--max-backlog=100000" "-vv"]
The rclone config contents with secrets removed.
[**private**]
type = ftp
host = **private**
user = **private**
port = 6621
pass = **private**
explicit_tls = true
disable_mfmt = true
disable_tls13 = true
encoding = Slash,Asterisk,Ctl,Dot
concurrency = 3
[**private**_crypt]
type = crypt
remote = **private**
directory_name_encryption = false
password = **private**
A log from the command with the -vv
flag
Below is a fragment of the debug log showing the problem. Interestingly, the manual says with the -vv option rclone should print all low-level retries, but there don't seem to be any here.
2021/11/18 15:54:17 DEBUG : Programowanie/Lazarus-Win/HAWK/src/components/graphics32-master_MOD/Documentation/Source/Units/GR32_Backends/Interfaces/IDeviceContextSupport/_Body.htm: Sizes identical
2021/11/18 15:54:17 DEBUG : Projekty/MartinoEscapeRoom/magic-rooms-repo/controller/sources/soft_demobrd_v1.0_F746G/Debug/Middlewares/Third_Party/LwIP/src/core/ipv6/icmp6.o: Excluded
2021/11/18 15:54:17 ERROR : Programowanie/LBA/LBArchitect/Components/graphics32/Examples/Vcl/Resampling/Resamplers_Ex: error reading destination directory: 425 PASV: Address already in use
2021/11/18 15:54:17 DEBUG : Programowanie/LBA/LBArchitect/Components/graphics32/Examples/Vcl/Resampling/PixelF_Ex/createbundle.sh: Unchanged skipping
2021/11/18 15:54:17 DEBUG : Programowanie/C-STM32/DISCO_F746G/zabawa/zabawa/SW4STM32/zabawa/Drivers/Components/st7735/Release_Notes.html: Sizes identical