Again...remote folders wiped out, help!

[Bazzu85]

@Ole

I wouldn't call it an assumption, I see it as an interpretation of the log where a sync with --retries=5 is viewed as 5 syncs with --retries=1.

Nope man. I was suggesting to use the --retries 1 and after this try, do nothing on this folder.
Moving from a --retries 5 to 5 times --retries 1 is nonsense..

can you point me the log you're mentioning with the 39 seconds downtime?

In all my test with --retries 5 and a sleep between retries of 60seconds.. if I turn off the drive and after 10 seconds I turn it back on, drivepool stabilize itself before a new sync try starts..

[Ole (Ole Frost)]

I disagree with your conclusions and questions, because I see glitches in the reasoning behind them.

Do you agree that rclone will get a wrong picture of your local filesystem if a part of the DrivePool is offline for a few milliseconds while the OS is performing this DIR:

@Ole

2021/08/21 00:49:06 rclone performs a DIR of the source top folder and finds (almost) nothing

[Bazzu85]

@Ole

I disagree with your conclusions and questions, because I see glitches in the reasoning behind them.

ok, lets talk about this glitches..

@Ole

Do you agree that rclone will get a wrong picture of your local filesystem if a part of the DrivePool is offline for a few milliseconds while the OS is performing this DIR

yep if you try to read the folder suddenly after the drive goes offline. If you wait some seconds the drivepool return automatically in a consistent state (if the drive come back online obviously)

[Ole (Ole Frost)]

@Bazzu85

ok, lets talk about this glitches..

We do.

Do you understand and agree with this:

@Ole

2021/08/21 00:49:06 rclone sync B starts with --retries=1
2021/08/21 00:49:06 rclone performs a DIR of the source top folder and finds (almost) nothing
2021/08/21 00:49:06 rclone performs a DIR on the target top folder and finds a lot of files and folders
2021/08/21 00:49:06 rclone starts deleting all the extra files and folders on the target
..some time later.. all extra files and folders the target have been deleted (there was no errors)
..some time later.. Sync B completes without errors

[Bazzu85]

@Ole

Do you understand and agree with this:

@Ole

> 
> ```

yes, if the drive cotaining the folder is still offline yes.

this is the reason why I'm using now for every sync --retries 5 and --retries-sleep 60s. in this way if during the first o other attempts the drive goes offline rclone will wait 60seconds to do another retry.
additionally I added PING localhost -n 61 >NUL between every sync command to avoid problems on the fifth attempt, waiting other 60 second before start sync B
example:

@ECHO OFF
For /f "tokens=1-3 delims=/:" %%a in ("%DATE%") do (set mydate=%%c%%b%%a)
For /f "tokens=1-3 delims=/:," %%a in ("%TIME%") do (set mytime=%%a%%b%%c)
rem ECHO %mydate%_%mytime%

REM Console height / width
rem MODE 200,13 | ECHO off
REM Console title
TITLE Media

REM SET /p RCLONE_BWLIMIT="> BWLIMIT: "
REM SET /p RCLONE_TRANSFERS="> TRANSFERS: "
SET RCLONE_BWLIMIT=6000
SET RCLONE_TRANSFERS=4
SET RCLONE_CHECKERS=8
SET RCLONE_STATS=10s

tasklist /FI "IMAGENAME eq rclone.exe" 2>NUL | find /I /N "rclone.exe">NUL
if "%ERRORLEVEL%"=="0" exit

REM JOB UNICO PER LE CARTELLE ANIME
SET RCLONE_TRANSFERS=4
N:\rclone\rclone.exe sync "H:\Anime" Media:"Anime"  --backup-dir=Media:"00 - Backup/Anime/%mydate%_%mytime%" --fast-list --checksum --verbose --transfers %RCLONE_TRANSFERS% --checkers %RCLONE_CHECKERS% --bwlimit %RCLONE_BWLIMIT% --contimeout 60s --timeout 300s --retries 5 --low-level-retries 10 --rc --stats %RCLONE_STATS% --stats-file-name-length 150 --exclude "desktop.ini" --links --ignore-size --log-file "r:\rclone scripts\media_%mydate%_%mytime%.log" --track-renames --track-renames-strategy modtime --retries-sleep 60s

PING localhost -n 61 >NUL

REM JOB UNICO PER LE CARTELLE CARTONI
SET RCLONE_TRANSFERS=4
N:\rclone\rclone.exe sync "H:\Cartoni" Media:"Cartoni"  --backup-dir=Media:"00 - Backup/Cartoni/%mydate%_%mytime%" --fast-list --checksum --verbose --transfers %RCLONE_TRANSFERS% --checkers %RCLONE_CHECKERS% --bwlimit %RCLONE_BWLIMIT% --contimeout 60s --timeout 300s --retries 5 --low-level-retries 10 --rc --stats %RCLONE_STATS% --stats-file-name-length 150 --exclude "desktop.ini" --links --ignore-size --log-file "r:\rclone scripts\media_%mydate%_%mytime%.log" --track-renames --track-renames-strategy modtime --retries-sleep 60s

PING localhost -n 61 >NUL

this cover all the cases if a drive goes offline for some seconds..

[Ole (Ole Frost)]

@Bazzu85

this cover all the cases if a drive goes offline for some seconds..

No, I see a glitch in the reasoning behind it.

@Bazzu85

yes, if the drive cotaining the folder is still offline yes.

I see ambiguity in your if… (and it thereby hides the glitch). Let’s try again:

@Ole

Do you understand and agree with this:

@Ole

> 2021/08/21 00:49:06 rclone sync B starts with --retries=1
2021/08/21 00:49:06 rclone performs a DIR of the source top folder and finds (almost) nothing
2021/08/21 00:49:06 rclone performs a DIR on the target top folder and finds a lot of files and folders
2021/08/21 00:49:06 rclone starts deleting all the extra files and folders on the target
..some time later.. all extra files and folders the target have been deleted (there was no errors)
..some time later.. Sync B completes without errors
> ```

@Ole

if a part of the DrivePool is offline for a few milliseconds while the OS is performing this DIR:

@Ole

2021/08/21 00:49:06 rclone performs a DIR of the source top folder and finds (almost) nothing

[Bazzu85]

@Ole

No, I see a glitch in the reasoning behind it.

I'll listen..

@Ole

I see ambiguity in your if… (and it thereby hides the glitch). Let’s try again:

there's no ambiguity. When the drive goes offline, the drivepool fix itself with the other drive. After the drive goes back online (few seconds), even drivepool will fix itself.
So making a time-line.

At 00.00.30 the drive goes offline
At 00.00.30 the drivepool is an image of all online drives exept the offline one
At 00.00.31 the drive come back online
At 00.00.31 the drivepool see that the drive is back online and start to fix the image of the drivepool including the content of the drive
At 00.00.32 (or 33 or 35 or 37 it dipends on the machine load, usually within a couple of seconds) the drivepool finish is fix and is back at the 00.00.30 state

what's is not clear? the downtime for the drive is less than a second. the drivepool need some seconds to stabilize. Waiting 60 second before starting again (next attempt o a new sync) is enought..

[Ole (Ole Frost)]

I cannot see an answer to my question (Do you understand and agree with ... if ...)

[Bazzu85]

@Ole

Do you understand and agree with this:

@Ole

> 
> ```

yes if you launch the same sync command (on the same folder) twice.
I'm not dumb. If you use the --retries 5 or launch the sync command with --retries 1 5 times is the same thing. so please move on.

[Ole (Ole Frost)]

@Bazzu85

if you launch the same sync command (on the same folder) twice.

I doubt we can reach a common view on this.

I therefore suggest you select the solution you find best and add --max-delete as a precaution.