When doing an rclone copy of C:\ drive from local Windows machine to B2, I see the following errors:
ERROR : Users/xyz/NTUSER.DAT{715fc15a-c81a-11ea-afe2-b42e9910d326}.TxR.1.regtrans-ms: Failed to copy: Checksum did not match data received (400 bad_request)
ERROR : Users/xyz/NTUSER.DAT{715fc15a-c81a-11ea-afe2-b42e9910d326}.TxR.0.regtrans-ms: Failed to copy: Checksum did not match data received (400 bad_request)
ERROR : Users/xyz/NTUSER.DAT{715fc15a-c81a-11ea-afe2-b42e9910d326}.TxR.blf: Failed to copy: Checksum did not match data received (400 bad_request)
ERROR : Users/xyz/NTUSER.DAT{715fc15a-c81a-11ea-afe2-b42e9910d326}.TxR.2.regtrans-ms: Failed to copy: Checksum did not match data received (400 bad_request)
These same errors appear on each rclone retry. After rclone starts the 3rd retry, it shows 3 of the above 4 errors again and then it ceases to print any further output and just hangs there. It was configured to print stats every 2 seconds. After waiting for 20 minutes, there was still no further rclone output.
What is your rclone version (output from rclone version)
rclone v1.52.2-232-gff843516-beta
os/arch: windows/amd64
go version: go1.14.6
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10 64 bit
Which cloud storage system are you using? (eg Google Drive)
B2 (native B2 API)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
those files are special low-level system files, used for transactional operations.
windows is not going to let you access them and you do not need to backup them up.
i see that you trying to copy your entire c: to cloud.
if you are looking for a solution for a bare-metal system backup for a windows computer, i suggest veeam agent, free edition. that is what i use.
it is easy to take the veeam backup files and use rclone to copy to cloud.
Hi @asdffdsa, much appreciated. Thank you for the hint about veeam agent, will check that out.
The only thing is that I may or may not be backing up the entire C:\ drive. I could also just be doing a few folders, it depends on the machine in question.
It's fine if rclone can't copy those files, I would just expect it to be able to carry on and do the rest which it is able to copy across. The most important thing would be to not hang, and then secondly it would be nice to have some kind of flag which can tell rclone to ignore these kind of errors. Other errors I've seen are:
Failed to copy: failed to open source object: Access is denied.
failed to open directory "Windows/CSC": open \\?\C:\Users\xyz\Documents\vssSnapshotTmp\Windows\CSC: Access is denied.
can't copy - source file is being updated (size changed from 1367872 to 1433792)
For this last one I believe the --local-no-check-updated flag allows ignoring the error, however for the other errors, I am not aware of a flag which would allow ignoring them.
By the way, while we're on it. If I'm using VSS, why would I still get errors about files being updated? As far as I understood, the VSS snapshot would have taken care of that?
trying to backup a windows system using vss and rclone is going to be a problem.
sorry, but i cannot help you with that, it is not going to work.
as i mentioned, i use veaam agent, but there are many free software that does the same thing.
there are many types of snaphosts, and the snapshot in the wiki is not for backing system folders and files.
i used it to backup certain data folders.
would need to know the path and name of the file.
perhaps you can not using my wiki correctly or trying to use it for something it was not designed for.
as for rclone hanging, would need a full debug log.
Hi @asdffdsa, I hear you. My intention is also not to do a full system backup, just a best effort, that's why it would be great if rclone could ignore some of these errors.
By the way, thank you very much for the cool wiki on how to use vss with rclone, it has helped me a lot! I will look at other ways of invoking vss, but yeah, the bottom line of this post is that I'd like to figure out if rclone either has a flag to ignore these errors or could maybe be changed to have such a flag?
Regarding a full log for the issue of rclone hanging, I don't have one right now and can't easily get to the machine where this has been happening, but will try to see if I can access the machine and get a log. What I have seen so far though looks like there is nothing helpful in the log, it simply stops...
Hi @ncw, that sounds promising
I assume that would cover the two access related errors mentioned above (file and directory access denied) but would it also cover the Failed to copy: Checksum did not match data received errors?
I didn't see one of those above? Presumably that is for a file that didn't change length but did get modified during transfer - like some kind of database? I don't think that anything will save you there other than using some kind of snapshot.
Perhaps your best strategy for a best efforts backup is to set --local-no-check-updated and then just use --retries 1 so rclone does one pass of that and stops regardless of whether there were errors or not.
The problem with files that get modified when being read is this...
B2 requires the sha1sum in advance, so we read the file and calculate it. We then upload the file and check the sha1sum matches after the upload. If the file was modified without changing its length then we'll notice at this point with the error you saw.
You can use --ignore-checksums to work around those errors, but you are losing data integrity checks...
Hi @ncw, I see, understood. That does make sense. I will try to experiment with the VSS settings to see if I can prevent the checksum errors that way rather.
Regarding the machine where rclone was hanging, it has been formatted and reinstalled since the error and now it is not failing anymore
