Rclone sync C:/ Drive i/o errors and Access is denied

What is the problem you are having with rclone?


Main issue is Hello.txt is still in target directory, but not in source.

sync not working when syncing local files it works like copy unless i use --ignore-errors OR --delete-during flag. Delete-during deletes the 2 hello.txt files for testing in videos / music / documents.

What is your rclone version (output from rclone version)

C:\Users\PC>rclone version
rclone v1.55.1

  • os/type: windows
  • os/arch: 386
  • go/version: go1.16.3
  • go/linking: dynamic
  • go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

win 10 64bit

Which cloud storage system are you using? (eg Google Drive)

Local for testing
(same tree as gdrive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --create-empty-src-dirs C:\Users\PC\ "E:\Backup\PC Files" --include-from "C:\Users\PC\.rclone\Directories\C-Drive\UserFiles-Local.txt" --fast-list -L -P --copy-links --ignore-case-sync --log-level INFO --log-file "C:\Users\PC\.rclone\Log Files\C-Drive\UserFiles_Local.log" --drive-chunk-size 128m --buffer-size 256m

The rclone config contents with secrets removed.

LOCAL For testing. 

UserFiles-Local.txt

/Documents/**
/Downloads/**
/Music/**
/Pictures/**
/Videos/**

A log from the command with the -vv flag

C:\Users\PC>rclone sync C:\Users\PC\ "E:\Backup\PC Files" --include-from "C:\Users\PC\.rclone\Directories\C-Drive\UserFiles-Local.txt" -l -P --ignore-case-sync --log-level ERROR --log-file "C:\Users\PC\.rclone\Log Files\C-Drive\UserFiles_Local.log" --drive-chunk-size 128m --buffer-size 256m --skip-links --delete-during
Transferred:           165 / 165 Bytes, 100%, 42 Bytes/s, ETA 0s
Errors:                 4 (retrying may help)
Checks:            102851 / 102851, 100%
Deleted:                2 (files), 0 (dirs)
Elapsed time:         8.4s

C:\Users\PC>rclone sync C:\Users\PC\ "E:\Backup\PC Files" --include-from "C:\Users\PC\.rclone\Directories\C-Drive\UserFiles-Local.txt" -l -P --ignore-case-sync --log-level ERROR --log-file "C:\Users\PC\.rclone\Log Files\C-Drive\UserFiles_Local.log" --drive-chunk-size 128m --buffer-size 256m --skip-links --ignore-errors
Transferred:           165 / 165 Bytes, 100%, 501 Bytes/s, ETA 0s
Errors:                 4 (retrying may help)
Checks:            102849 / 102849, 100%
Deleted:                0 (files), 34699 (dirs)
Elapsed time:        16.3s

UserFiles_Local.log https://gist.github.com/Lawrence575/a57e1d4df7540e91de525cc08da8d46c

For future notes/viewers, this also happens on the cloud sync for me atlest. Not sure as to why its trying to pull documents/my music etc. as they don't exist on my machine in the working "Users/PC/" folder... that its assigned to.

Basicly rclone is somehow escaping the --include-from UserFiles-Local I would do a filter file but i'm not sure what all i'd have to filter for C:\Users\PC as there are around 25 full items including 23 folders, and 2 files, a link to my backup script to run from cmd by typing in back and having it auto file for me. And then NTUSER.DAT that for obvious reasons I wouldn't want synced (cant carry it over for new windows user/install. as this backup is for re-deploying windows software appdata backups for settings/configs. To replace data I could use VCC but I don't really see a need for it as I wouldn't be able to pluck out certain files for manual restoring, or runing rclone from E: to C: on windows reinstall to copy back settings / game files, qbitt settings, ini files, configs for certain profiles that I have merged over. etc.

NEW LOG FILE WITH ERRORS ONLY: + --delete-during --local-no-check-updated Flags enabled

2021/05/21 20:14:41 ERROR : Cookies: failed to open directory "Cookies": open \\?\C:\Users\PC\Cookies: Access is denied.
2021/05/21 20:14:41 ERROR : Application Data: failed to open directory "Application Data": open \\?\C:\Users\PC\Application Data: Access is denied.
2021/05/21 20:14:41 ERROR : Local Settings: failed to open directory "Local Settings": open \\?\C:\Users\PC\Local Settings: Access is denied.
2021/05/21 20:14:41 ERROR : My Documents: failed to open directory "My Documents": open \\?\C:\Users\PC\My Documents: Access is denied.
2021/05/21 20:14:41 ERROR : NetHood: failed to open directory "NetHood": open \\?\C:\Users\PC\NetHood: Access is denied.
2021/05/21 20:14:41 ERROR : PrintHood: failed to open directory "PrintHood": open \\?\C:\Users\PC\PrintHood: Access is denied.
2021/05/21 20:14:41 ERROR : AppData/Local/Application Data: failed to open directory "AppData/Local/Application Data": open \\?\C:\Users\PC\AppData\Local\Application Data: Access is denied.
2021/05/21 20:14:41 ERROR : Recent: failed to open directory "Recent": open \\?\C:\Users\PC\Recent: Access is denied.
2021/05/21 20:14:41 ERROR : SendTo: failed to open directory "SendTo": open \\?\C:\Users\PC\SendTo: Access is denied.
2021/05/21 20:14:41 ERROR : Start Menu: failed to open directory "Start Menu": open \\?\C:\Users\PC\Start Menu: Access is denied.
2021/05/21 20:14:41 ERROR : Templates: failed to open directory "Templates": open \\?\C:\Users\PC\Templates: Access is denied.
2021/05/21 20:14:41 ERROR : Documents/My Music: failed to open directory "Documents/My Music": open \\?\C:\Users\PC\Documents\My Music: Access is denied.
2021/05/21 20:14:41 ERROR : Documents/My Pictures: failed to open directory "Documents/My Pictures": open \\?\C:\Users\PC\Documents\My Pictures: Access is denied.
2021/05/21 20:14:41 ERROR : Documents/My Videos: failed to open directory "Documents/My Videos": open \\?\C:\Users\PC\Documents\My Videos: Access is denied.
2021/05/21 20:14:41 ERROR : AppData/Local/History: failed to open directory "AppData/Local/History": open \\?\C:\Users\PC\AppData\Local\History: Access is denied.
2021/05/21 20:14:42 ERROR : AppData/Local/Temporary Internet Files: failed to open directory "AppData/Local/Temporary Internet Files": open \\?\C:\Users\PC\AppData\Local\Temporary Internet Files: Access is denied.
2021/05/21 20:14:42 ERROR : AppData/Local/Microsoft/Windows/INetCache/Content.IE5: failed to open directory "AppData/Local/Microsoft/Windows/INetCache/Content.IE5": open \\?\C:\Users\PC\AppData\Local\Microsoft\Windows\INetCache\Content.IE5: Access is denied.
2021/05/21 20:14:42 ERROR : AppData/Local/Microsoft/Windows/INetCache/Low/Content.IE5: failed to open directory "AppData/Local/Microsoft/Windows/INetCache/Low/Content.IE5": open \\?\C:\Users\PC\AppData\Local\Microsoft\Windows\INetCache\Low\Content.IE5: Access is denied.
2021/05/21 20:14:42 ERROR : AppData/Local/Microsoft/Windows/Temporary Internet Files: failed to open directory "AppData/Local/Microsoft/Windows/Temporary Internet Files": open \\?\C:\Users\PC\AppData\Local\Microsoft\Windows\Temporary Internet Files: Access is denied.
2021/05/21 20:14:43 INFO  : There was nothing to transfer
2021/05/21 20:14:43 ERROR : Can't retry any of the errors - not attempting retries
2021/05/21 20:14:43 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                19 (no need to retry)
Checks:             34284 / 34284, 100%
Elapsed time:         1.8s

2021/05/21 20:14:43 Failed to sync with 19 errors: last error was: failed to open directory "AppData/Local/Microsoft/Windows/Temporary Internet Files": open \\?\C:\Users\PC\AppData\Local\Microsoft\Windows\Temporary Internet Files: Access is denied.

So apparently I did some more digging and its trying to look threw protected os files. anyway to black list them?

hi,

my music is a link folder pointing to the real music folder
image

i was able to get rclone to copy the files inside the links without errors.
%rcmd% copy %source% %dest% --copy-links -vv
perhaps run the commands with admin privelges?

what VCC is, did you mean VSS?
it is possible to create a VSS snapshot and have rclone use that as documented in my wiki

i have never used rclone for local to local.
perhaps you need to run rclone using admin privileges.
using fastcopy, it was able to copy the files from the links.

Yes i meant vss, sorry I was very tired been trying to get this working for a few hours before reaching out, I was looking through your wiki as someone brought this up already.

as for the admin part. I already run this with admin privileges. it auto elevates at the start of my cmd prompt with:

@echo off
if _%1_==_payload_  goto :payload

:getadmin
    echo %~nX0: elevating self
    set vbs=%temp%\getadmin.vbs
    echo Set UAC = CreateObject^("Shell.Application"^)                >> "%vbs%"
    echo UAC.ShellExecute "%~s0", "payload %~sdp0 %*", "", "runas", 1 >> "%vbs%"
    "%temp%\getadmin.vbs"
    del "%temp%\getadmin.vbs"
goto :eof

:payload
backupscript bellow

I'm sorry what did you mean for fast copy?

as for the local, I would run robocopy (thats what I used before switching to online and local backup) But.. seeing as this was a problem for local AND cloud, I rather keep it to just rclone for now.

I know you mentioned VSS yes that's what I was referring to. Wouldn't it be better to just do the way i'm doing it, for plucking out files in that 1 day window instead of reversing to an entire snapshot?
but still keeping the snapshot for re-deployment?

2:

Removing copy-links fixes it. (and it deletes successfully.)

But with the adding of copy-links it creates the IO errors again even with admin pillages.


2021/05/22 08:06:21 ERROR : Documents/My Music: failed to open directory "Documents/My Music": open \\?\C:\Users\PC\Documents\My Music: Access is denied.
2021/05/22 08:06:21 ERROR : Documents/My Pictures: failed to open directory "Documents/My Pictures": open \\?\C:\Users\PC\Documents\My Pictures: Access is denied.
2021/05/22 08:06:21 ERROR : Documents/My Videos: failed to open directory "Documents/My Videos": open \\?\C:\Users\PC\Documents\My Videos: Access is denied.
2021/05/22 08:06:22 ERROR : Local file system at //?/E:/Backup/PC Files: not deleting files as there were IO errors
2021/05/22 08:06:22 ERROR : Local file system at //?/E:/Backup/PC Files: not deleting directories as there were IO errors
2021/05/22 08:06:22 ERROR : Attempt 1/3 failed with 4 errors and: not deleting files as there were IO errors
2021/05/22 08:06:22 ERROR : Documents/My Videos: failed to open directory "Documents/My Videos": open \\?\C:\Users\PC\Documents\My Videos: Access is denied.
2021/05/22 08:06:22 ERROR : Documents/My Music: failed to open directory "Documents/My Music": open \\?\C:\Users\PC\Documents\My Music: Access is denied.
2021/05/22 08:06:22 ERROR : Documents/My Pictures: failed to open directory "Documents/My Pictures": open \\?\C:\Users\PC\Documents\My Pictures: Access is denied.
2021/05/22 08:06:22 ERROR : Local file system at //?/E:/Backup/PC Files: not deleting files as there were IO errors
2021/05/22 08:06:22 ERROR : Local file system at //?/E:/Backup/PC Files: not deleting directories as there were IO errors
2021/05/22 08:06:22 ERROR : Attempt 2/3 failed with 4 errors and: not deleting files as there were IO errors
2021/05/22 08:06:22 ERROR : Documents/My Music: failed to open directory "Documents/My Music": open \\?\C:\Users\PC\Documents\My Music: Access is denied.
2021/05/22 08:06:22 ERROR : Documents/My Pictures: failed to open directory "Documents/My Pictures": open \\?\C:\Users\PC\Documents\My Pictures: Access is denied.
2021/05/22 08:06:22 ERROR : Documents/My Videos: failed to open directory "Documents/My Videos": open \\?\C:\Users\PC\Documents\My Videos: Access is denied.
2021/05/22 08:06:22 ERROR : Local file system at //?/E:/Backup/PC Files: not deleting files as there were IO errors
2021/05/22 08:06:22 ERROR : Local file system at //?/E:/Backup/PC Files: not deleting directories as there were IO errors
2021/05/22 08:06:22 ERROR : Attempt 3/3 failed with 4 errors and: not deleting files as there were IO errors
2021/05/22 08:06:22 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 4 (retrying may help)
Checks:            102843 / 102843, 100%
Elapsed time:         0.7s

2021/05/22 08:06:22 Failed to sync with 4 errors: last error was: not deleting files as there were IO errors

I moved to a filter to block out desktop.ini's *.logs sharex/logs for working directory's as there not valid for what I'm doing to sync over from the user-files. (doc/pictures/music etc.)

Here is my filter below: it seems to get rid of the other folders it was pulling from the root source directory so at-lest that's a step forward in the right direction.

# Main Filter:
- desktop.ini
- /Documents/ShareX/Logs/**
+ /Documents/**
+ /Downloads/**
+ /Music/**
+ /Pictures/**
+ /Videos/**
# exclude everything else
- *

Should I try adding in these to the filter as its just links? I just need the manual shortcuts to sync over instead of the windows autogenerated. as from what I can see, all i'm syncing is just links to the folders i'm already syncing over right?

\Documents\My Pictures
\Documents\My Music
\Documents\My Videos

Edit for testing: So I tried doing it just now, I did this earlier but I must of had conflicting flags. because now its working properly:

rclone sync C:\Users\PC\ "E:\Backup\PC Files" --filter-from "C:\Users\PC\.rclone\Directories\C-Drive\Filter-UserFiles-Local.txt" --fast-list -P --ignore-case-sync --log-level INFO --log-file "C:\Users\PC\.rclone\Log Files\C-Drive\UserFiles_Local.log" --buffer-size 256m --copy-links

updated Filter

# Main Filter:
- desktop.ini
- /Documents/My Music/**
- /Documents/My Pictures/**
- /Documents/My Videos/**
- /Documents/ShareX/Logs/**
+ /Documents/**
+ /Downloads/**
+ /Music/**
+ /Pictures/**
+ /Videos/**
# exclude everything else
- *

UserFiles_Local.log

2021/05/22 08:19:23 INFO  : Documents/ShareX/Screenshots/2021-05/cmd_ddGuBOOQFK.png: Copied (new)
2021/05/22 08:19:23 INFO  : 
Transferred:   	    2.756M / 2.756 MBytes, 100%, 31.731 MBytes/s, ETA 0s
Checks:             34282 / 34282, 100%
Transferred:            2 / 2, 100%
Elapsed time:         0.3s

merged to top reply.

at this point, what is the exact problem?

better to test using rclone ls not rclone copy

fastcopy is an open-source copy program, been using it for approx 10 years.
i only use robocopy in case i need one of it unique flags in a server environment.
the problem is robocopy is still after all these years, it cannot checksum the source and dest, so that is no go for me.
i do not want to trust all my backups to just rclone, as one bug there can destroy all the backups, local and cloud. i have a script that uses a combination of vss, rclone, 7zip, fastcopy and veeam.

as for vss, if you read my wiki, there is no mention for reversing or entire snapshot.
i copy just the folders/files from the temporary snapshot to the dest.
when the command is done, windows auto-deletes the snapshot.

Sorry! I got it confused with the VSS and a vmsn similar to how I save snapshots of my vm. As I skimmed through it last night.

So it due to the conflicting flags and not using a filter file. Managed to fix it because it was trying to pull pieces of the name from different directories.

2021/05/22 09:29:03 INFO  : Users/PC/.rclone/Backup.bat.lnk: Copied (replaced existing)
2021/05/22 09:29:05 INFO  : Users/PC/.rclone/Directory/.-Drive -Master Example.txt: Copied (new)
2021/05/22 09:29:07 INFO  : Users/PC/.rclone/Directory/C-Drive/MediaLibrary.include: Copied (new)
2021/05/22 09:29:07 INFO  : Users/PC/.rclone/Directory/C-Drive/UserFiles-Local.filter: Copied (new)
2021/05/22 09:29:08 INFO  : Users/PC/.rclone/Directory/C-Drive/UserFiles-Cloud.filter: Copied (new)
2021/05/22 09:29:08 INFO  : Users/PC/.rclone/Directory/C-Drive/Main.filter: Copied (new)
2021/05/22 09:29:09 INFO  : Users/PC/.rclone/Directory/D-Drive/File: Copied (new)
2021/05/22 09:29:10 INFO  : Users/PC/.rclone/Directory/E-Drive/Media-LocalRestore.txt: Copied (new)
2021/05/22 09:29:11 INFO  : Users/PC/.rclone/Directory/E-Drive/Software.include: Copied (new)
2021/05/22 09:29:14 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s012.vmdk: Copied (new)
2021/05/22 09:29:16 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s013.vmdk: Copied (new)
2021/05/22 09:29:19 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s014.vmdk: Copied (new)
2021/05/22 09:29:22 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s015.vmdk: Copied (new)
2021/05/22 09:29:24 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s016.vmdk: Copied (new)
2021/05/22 09:29:26 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s017.vmdk: Copied (new)
2021/05/22 09:29:28 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s018.vmdk: Copied (new)
2021/05/22 09:29:31 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s019.vmdk: Copied (new)
2021/05/22 09:29:33 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004.vmdk: Copied (new)
2021/05/22 09:29:38 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-Snapshot15.vmsn: Copied (new)
2021/05/22 09:29:43 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-Snapshot18.vmsn: Copied (new)
2021/05/22 09:29:45 INFO  : Users/PC/.rclone/Directory/X-Drive/File: Copied (new)
2021/05/22 09:29:48 INFO  : ProgramData/Radarr/logs/radarr.txt: Copied (replaced existing)
2021/05/22 09:29:49 INFO  : Users/PC/.config/rclone/rclone.conf: Copied (replaced existing)
2021/05/22 09:29:51 INFO  : ProgramData/Sonarr/logs/sonarr.txt: Copied (replaced existing)
2021/05/22 09:29:53 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM.nvram: Copied (replaced existing)
2021/05/22 09:29:54 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM.vmx: Copied (replaced existing)
2021/05/22 09:29:56 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM.vmsd: Copied (replaced existing)
2021/05/22 09:29:57 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM.vmxf: Copied (replaced existing)
2021/05/22 09:29:58 INFO  : Users/PC/.rclone/Scripts/Backup.bat: Copied (replaced existing)
2021/05/22 09:30:00 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MLV2_IMAGE_CACHE/845536481777942558.jpg: Copied (new)
2021/05/22 09:30:01 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MLV2_IMAGE_CACHE/845536498668535869.jpg: Copied (new)
2021/05/22 09:30:03 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MLV2_IMAGE_CACHE/845536503819927562.jpg: Copied (new)
2021/05/22 09:30:04 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MLV2_IMAGE_CACHE/845536511147376660.jpg: Copied (new)
2021/05/22 09:30:06 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MLV2_IMAGE_CACHE/845536519158759434.jpg: Copied (new)
2021/05/22 09:30:07 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MLV2_IMAGE_CACHE/845536531057475594.jpg: Copied (new)
2021/05/22 09:30:09 INFO  : Users/PC/AppData/Roaming/qBittorrent/qBittorrent-data.ini: Copied (replaced existing)
2021/05/22 09:30:10 INFO  : Users/PC/AppData/Roaming/VMware/inventory.vmls: Copied (replaced existing)
2021/05/22 09:30:11 INFO  : Users/PC/AppData/Roaming/VMware/preferences.ini: Copied (replaced existing)
2021/05/22 09:30:13 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MessageLoggerV2Data.config.json: Copied (replaced existing)
2021/05/22 09:30:14 INFO  : Users/PC/AppData/Roaming/BetterDiscord/plugins/MessageLoggerV2DataBackup.config.json: Copied (replaced existing)
2021/05/22 09:34:08 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s008.vmdk: Copied (new)
2021/05/22 09:35:45 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s005.vmdk: Copied (new)
2021/05/22 09:36:05 INFO  : Users/PC/.VMware/Virtual Machines/Windows 10 x64 - Work VM/Windows 10 x64 - Work VM-000004-s001.vmdk: Copied (new)
2021/05/22 09:36:06 INFO  : Users/PC/.rclone/Directories/X-Drive/.txt: Deleted
2021/05/22 09:36:06 INFO  : Users/PC/.rclone/Directories/E-Drive/Software.txt: Deleted
2021/05/22 09:36:06 INFO  : Users/PC/.rclone/Directories/C-Drive/UserFiles-Local.txt: Deleted
2021/05/22 09:36:07 INFO  : Users/PC/.rclone/Directories/D-Drive/.txt: Deleted
2021/05/22 09:36:07 INFO  : Users/PC/.rclone/Directories/E-Drive/Media-LocalRestore.txt: Deleted
2021/05/22 09:36:07 INFO  : Users/PC/.rclone/Directories/C-Drive/Main.txt: Deleted
2021/05/22 09:36:07 INFO  : Users/PC/.rclone/Directories/C-Drive/MediaLibrary.txt: Deleted
2021/05/22 09:36:07 INFO  : Users/PC/.rclone/Directories/.-Drive -Master Example.txt: Deleted
2021/05/22 09:36:08 INFO  : Users/PC/.rclone/Directories/C-Drive/UserFiles-Cloud.txt: Deleted
2021/05/22 09:40:18 INFO  : 
Transferred:   	    1.013G / 1.013 GBytes, 100%, 2.436 MBytes/s, ETA 0s
Checks:             49815 / 49815, 100%
Deleted:                9 (files), 218 (dirs)
Transferred:           43 / 43, 100%
Elapsed time:     11m55.3s

That's the main reason why I switched to it for local and cloud (cloud is crypt).

Mentioning the bug. I was going to tinker later on by adding a team drive to directly copy over server side. (I believe the limit is 2tb for server side) to full sync my current backup before it starts the new one for redundancy. as I manually launch my backups for now, with confirmation on each section.

good that you got it working as you need.

my point is not to use any one tool for local and for cloud.

i do not use the desktop/documents/my music folders at all.
for my local computer, i use veeam for a bare-metal recovery image backups.
i keep my data in c:\data and use those tools i mentioned above for backups to local server and cloud.

i do not use any google products. i am a combintaiton of wasabi, a s3 clone known for hot storage and aws s3 deep glacier to very cold storage, $1.00/TB/month

if you do plan to make use of my wiki, this is a snippet of what my script will generate and execute.

RcloneSyncFiles=C:\data\rclone\scripts\rclone.exe sync "b:\mount\rcloner\kdbx_files_wasabi_en07_20210522.122911\data\keepass\database" "wasabi_en07datacrypt:en07data/kdbx/rclone/backup" --backup-dir=wasabi_en07datacrypt:en07data/kdbx/rclone/archive/20210522.122911 --stats=0 --fast-list  --log-level=DEBUG --log-file=C:\data\rclone\logs\kdbx_files_wasabi_en07\20210522.122911\rclone.log --config=C:\data\rclone\scripts\wasabi.conf
RcloneCryptCheckFiles=C:\data\rclone\scripts\rclone.exe cryptcheck "b:\mount\rcloner\kdbx_files_wasabi_en07_20210522.122911\data\keepass\database" "wasabi_en07datacrypt:en07data/kdbx/rclone/backup" --stats=0 --fast-list --s3-no-check-bucket=true --log-level=DEBUG --log-file=C:\data\rclone\logs\kdbx_files_wasabi_en07\20210522.122911\rclone.log --config=C:\data\rclone\scripts\wasabi.conf
DeleteVShadowAfter=cmd /c rmdir b:\mount\rcloner\kdbx_files_wasabi_en07_20210522.122911 /q

So you go in and move the normal tree: to c:\data ? If that's so, I used to just hard link it to my E: drive but it got too annoying to do so as I wasn't able to find a way to automate it. My full windows reinstall is almost all automated. with using winget to grab my needed installs that are supported. Then an rclone restore to copy over the files but keep the newer versions of files in place. and only overwrite the data portion to specific folders of the installs like appdata/roaming etc.

Only reason I'm using google is because I have an unlimited main google drive now for a business g-suite. So I'm in the transition of creating a new config file to move over those files to my gdrive (15gb limit removed). and just keep a team-drive as a backup till I can find something either cheaper, or more reliable for backups. as backblaze and aws charge for download/streaming data from the cloud. (media library) If I remember correctly when I looked into it around a year ago. Isn't there a wait time to access the cold/hot storage to push / pull info from it?

i have little to nothing inside there.
i work on a lot of computers local, remote and virtual , so everything i need is in c:\data.
and i choose portable apps such as 7zip, keepass, double commander and keep firefox profile inside c:\data

no need for that as i use veeam agent, i can revert to any point in time that i need or just pull folders/files as needed.
it does bare metal recovery backups.
i can use the instant recovery features to boot my backup image into a hyper-v virtual machine on the fly or use veeam boot disc to take a backup and get it to boot unto new dissimilar hardware, like acronis universal restore.
and it is free to use.

imho, nothing better than the combination of wasabi and aws s3 deep glacier.

wasabi:

  • no egress fees
  • no wait time to push/pull.
  • no api fees or quotas.
  • no throttling
  • i use it for streaming every day.

i keep recent backups in wasabi, in case i need to recover from a disaster.
and older backups in aws deep glacier for $1.00/TB/month.
yes, that does have a delay and costs to download but for once in a lifetime disaster, where local copies are corrupt, wasabi backups are corrupt, paying aws not a problem.

Okay, so your basically using deep glacier as a retention for backups up to around a month or so? with the dates auto labeled then?

deep glacier has a minimum retention period of 90 days.
if i upload 1TB today, and delete it tomorrow, i be charged $3.03 =3 X $1.01/TB/month
after 90 days, that data can be deleted without additional charge.
i do not upload lots of small files. i upload large veeam backup files and .7z of local folders using rclone copy --immutable

the retention period can be customized on the aws website.

  1. delete x days after upload
  2. delete on a specific date.

Ah, I see. So you can upload the 1tb, but not delete it is what your saying? So (well i forgot the flag) but there was a flag to auto label todays date. and upload to that folder. , would that still be the same if you dont delete the old upload? due to them being labeled up to 90 days? Say you backup once a week. may 21st then upload on may 28th under the flag it would label at may 28th. so it wouldn't delete the 21st. would you still be charged for the upload but not the deletion fee due to it being separated in that one folder? Similar to synology history.

having the veeam backup being uploaded to that specific folder. for the retention of so many days for that week. (not sure if that makes since fully of what I'm trying to explain) Can think of it file history on windows 10 but a lot more.. valuable. for backups for the cold storage. Same for the small files that 7z is being used on (wouldn't the 7z just be this? for example: )

for /d %%X in (*) do "%ProgramFiles%\7-Zip\7z.exe" a "%%X.7z" "%%X\"

for a very basic compression of a directory.

Wouldn't AWS cold storage + wasabi basically become a non physical synology drive of some sorts with some features stripped due to having to set it up for zips to retain not uploading lots of small files? (for the more older backups in cold storage)

no such flag, my scripts creates the command on the fly.

as for veeam, i backup the repository, which is a single folder and --backup-dir does not work with deep glacier

as for synbox, i do use them with customers and rclone runs well on it.
but i prefer the free windows 2019 server - hyperv edition and refs file system.

Right, so wouldn't that solve the issue then of rclone accidently charging you by doing an accidental server side delete for a new recursive backup in aws? because its billed via deletion/request (or am I wrong on that where its 1$ per tb upload/stored also or is it just 1$tb per deletion. https://aws.amazon.com/glacier/pricing/?nc=sn&loc=3 all i could find on the mater.) Because rclone would automatically be pointed to that directory. Or does that rename of the new day happen after wasabi gets the 'hot' backup. So wouldn't you use wasabi for media library then in another section have your actual backups that need to be offloaded etc?
Say your media library is around 5tbs and actual backup of local is around 1tb would it be 7tbs total with a 1tb buffer to have them be offloaded to glacier after a set time frame from veeam? Trying to wrap my head around a more practical use case for keeping a media directory in a hot storage for streaming to it. while still being in a "valut" like storage for sonarr/radarr.

not possible.

  1. rclone copy --immutable`
  2. no s3:DeleteObject for bucket policy.
  3. object lock on bucket
  4. mfa delete on bucket

there is on rename, no sure what you mean?

never used sonarr/radarr

Main reason I used robocopy in the past, was because everything I did was local, then I lost all my data because hard-drive died so I looked into cloud based solution's, and I'm using rclone now mostly for offloading c drive backups to local copy for quick restores, and then cloud incase harddrive dies, and things that are hard to get back, like personal written programs, data files, statements etc. + a backlog of shows/movies for sonarr/radarr. Due to streaming sites kind of being taken down by DMCA requests. Also it's because its command line based. Even though I'm a windows user, I still mount my rclone mounts on my android if I need to grab something while I'm away or stream a movie/show etc.

that is why i use veeam, to protect against that.

my media files are inside a rclone crypt.

i cannot mount as i would never root my android phone or my android tablet
i use a cheap raspberry pi and on that a rclone mount and jellyfin media server.

i have a wiki on how to use android devices as media streaming server here

right sorry, Rename as in you said your script auto creates time stamped folders to avoid conflicting right?
So I was talking about it making or renaming a new "temp" folder, So it would keep the newest backup separate. Simaler to how if you hold down a button on the camera and take multiple photos on android the backend auto add's the seconds to the file name, but for this it would be set to month/day/year. instead of hours/minutes/seconds. for the folder so rclone knows where to offload it to cold storage. after the wasabi retention is over and it sees its time to move it to the cold storage. Not sure if I'm explaining this exactly right with the correct terminology. Trying to clarify what I was asking about:

For example this:

Wouldn't your script auto handle this so nothing conflicts or causes any mergers? So if you needed to backup twice in one day it would be put in that days folder instead but auto copy old data and only replace the old data with a checksum of changed files to keep the upload from the main host machine minimal?
(This is just an example of my theory) for better understanding.

copy RealBench_v2.43 - 5.21.21 > RealBench_v2.43 - 5.22.21
Sync RealBench_v2.43 > RealBench_v2.43 - 5.22.21 (replaces 2 files as there newer then dest files) 

or would it just reupload the entire RealBench_v2.43 folder and adhere the 5.22.21 date for organization.

I have an extremely shitty time explaining, and I tend to overcomplicate things. My bad if i'm making this harder then it should be to try and explain my question.

take a few minutes to undertand --backup-dir and the commands i posted.

the full sync is in the main folder, and the incremental changes do into the --backup-dir folder using a time-stamp.

that is not a problem as i use a date+timestamp accurate to one second
20210522.122911

as for cold storage, i never use sync and never copy lots of small files, not cost effective
rclone copy --immutable of veeam backup files and 7zip files.

correct