Files that were already synced being re-synced in same run

Help and Support
1

What is the problem you are having with rclone?

I do periodic backups of all my drives daily. Unfortunately none of my other backups ever run since my C:/ backup is stuck backing up "forever". This seems to be due to the backing up of my WSL VHD being repeated when it is rechecked. It takes about 20 hours for it to back up and by the time it's done the drive will have changed causing it (as far as I understand) to be re-synced. I want to backup this file as it contains a lot of important stuff, but this resyncing behavior is not meshing well with this file. Is there any way to avoid this?

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: windows/amd64
  • go version: go1.15.5

Which cloud storage system are you using? (eg Google Drive)

Google Drive with crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync C:/ udCryptC: --verbose=1 --fast-list --ignore-checksum --ignore-errors --transfers=15 --checkers=64 --tpslimit=10 --drive-chunk-size=16M --progress --exclude Windows/** --exclude Plex/** --exclude *.log --exclude *.LOG --exclude Recovery/** --exclude ProgramData/** --exclude **/Boot --exclude *.sys --exclude *.SYS --exclude Temp/** --exclude Boot/** --exclude $Recycle.Bin/** --exclude Config.MSI/** --exclude Recovery/** --exclude "System Volume Information"/** --exclude *.LOG1 --exclude *.LOG2 --exclude "Plex Media Server"/** --exclude cache2/** > rclone_c.log

The rclone config contents with secrets removed.

[udCryptC]
type = crypt
remote = ud:backup/desktop/C
filename_encryption = standard

A log from the command with the -vv flag

I've run it again with higher verbosity, but it's gonna take a few days to reach relevant point :slight_smile:
p.s. uploading a log this long is actually really really really hard lol

edit: it's blocking me from linking the logs directly, so here's a pastebin of the links...

2

hello and welcome to the forum,

not much rclone can about that.
each time a file changes rclone will copy it again.

one workaround is to run rclone, or some other linux backup program, inside wsl and backup the important files.
and on the winbox use --exclude="Users/Mike/AppData/Loc…c/LocalState/ext4.vhdx"

note: you can copy the rclone.conf file from winbox and re-use it inside wsl

 rclone version
rclone v1.56.2
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.10.60.1-microsoft-standard-WSL2 (x86_64)
3

Could I log this under a feature request/bug report then? I have some pretty finicky connectivity issues with WSL so running it directly on there would be a last resort honestly. Appreciate the suggestions regardless!

4

bug, what bug?

what exactly is the feature request?

what connection is finicky?

  • to the winbox host - is that reliable?
  • to the local lan
  • to the internet
5

Don't try to resync the same file that was already synced within the same run? Having unending runs due to this isn't exactly ideal and I can't be the only person with files that change during rclone's execution haha

WSL to the outside internet and local lan. WSLv2 doesn't like my network setup but I've at least got it working for what I need it for.

6

by default, if rclone has an issue with a file, rclone will retry three times, that is what might be happening.

without a debug log hard to tell.
best not to redirect the console output to > rclone_c.log
instead, use a real rclone log --log-level=DEBUG --log-file=c:\path\to\folder\rclone.log

fwiw,

  • for now, exclude ext4.vhdx from that sync so it will finish.
  • create a new rclone command to copy that single file
  • use a very large chunk size.

ext4.vhdx, is always open, always in use, always changing.
rclone is not going to be able reliably copy an in-use file, not sure there is a point in trying.
not sure how rclone is able to copy that file, as rclone should complain.
for example,

2021-10-16 20:53:10 ERROR : Users/Mike/NTUSER.DAT: Failed to copy: failed to open source object: The process cannot access the file because it is being used by another process.
2021-10-16 21:00:16 ERROR : Program Files (x86)/Steam/logs/appinfo_log.txt: corrupted on transfer: sizes differ 6355 vs 56514

possible workarounds:

  • before running sync, might try wsl -t DISTRO-NAME
  • not sure it will work, given the special nature of .vhdx used for wsl, but might try to create a VSS snapshot and backup that .vhdx from that VSS snapshot.
    whenever i run rclone, i always create a VSS snapshot and backup files from that snaphot.
    i have a wiki about that
    https://github.com/rclone/rclone/wiki/How-to-enable-VSS-for-rclone
  • other suggestions, but might be getting too off-topic for you?
7

@albertony, @Ole

any suggestions/comments

8

Hi 12340b1907,

I had a quick look in your logs and it looks like you are trying to use rclone for a complete system backup, including system settings and live/open (operating system) files, e.g. NTUSER.DAT and a live WSL VHD.

This isn’t something rclone is intended/designed to do - and certainly not something that is easily fixed/added.

So you need to make a decision. Either you exclude the live/open files from your backup, or you select a backup tool designed to handled them.

I use rclone for backup of my user data (documents, photos, etc.), but accept that I may have to restore programs and systems settings if the computer is lost/crashed. I therefore exclude NTUSER.DAT, AppData and similar.

In my WSL2 I work on files stored in /mnt/C/User/… They are then backed up as part of my backup of user files in Windows. I would use occasional or scripted wsl --export if I cared about my WSL settings.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.