rclone check -v doesn't show once per minute update counts.
This kind of bothers me.
Actually rclone check -vv doesn't show once per minute update counts. (It shows each and every individual file, but when I'm checking 500,000 files I'd rather just see a number showing me how many I've done, rather than a giant spammy list of each filename)
I guess I'm used to cryptcheck and copy which both show once per minute update counts.
Is there anyway to make rclone check -v "A:" "B:" show once per minute update counts, let's say, updating me on how many total checks have been made? the same way cryptcheck does? (or the same way copy does during copy's checking portion).
My remote is a crypt, so I've always used cryptcheck, and liked it fine, but I got a new computer and I'm doing some drive cloning and so I now have a use case for a normal check. Is there some sort of --flag that will make rclone check -v spit out once a minute updates like almost all other rclone commands I use do?
I figure there's gotta be a flag for this, since -vv leaves this out, it has to be intentional? so there must just be a flag I'm unaware of the existence of?
Summary just told me 500,000 okay 14 files missing. I didn't bother to output to a logfile because I assumed after it was done it would re-list the errors specifically for me to see. I knew the number of errors would be small.
Again cryptcheck and copy's internal check portion both do it like this.
I can just run check -v --stats 60m "A:" "B:" though to get it to show me only errors and occasionally updates, that should allow me to see/find the errors.
Although due to my own ineptitude I'm now gonna end up running this one check for 3--4days
--combined string Make a combined report of changes to this file
--differ string Report all non-matching files to this file
--dst-only string Report all files missing from the source to this file
--match string Report all matching files to this file
--src-only string Report all files missing from the destination to this file
You can specify any of the file names as - which means output to stdout.
The --differ, --dst-only, --match and --src-only flags write file paths, one per line of type indicated.
So --differ will contain all files which are different on the source and dest, etc
The --combined file contains all file paths with a symbol at the front of the line to tell you what happened to it.
= means the files matched, - means the file was missing on the destination, + means the file was extra on the destination and * means that the file was present in source and destination but different.
okay so I definitely want to use either --src-only or --dst-only but .... why does dst-only report files missing from the source? when dst stands for destination?
although... to avoid confusion, and make sure all I learn about is missing files on the destination I could in theory just run check -v --one-way --differ "A:" "B:"
that's probably what I'll try.
edit: BTW, it'll be at least 25hours before I can attempt this command.
after rclone copy failed my dates, windows copy ALSO failed my dates (although managed to randomly get like 50% of them right which is almost worse than getting 100% of them wrong)
Now i'm using fastcopy but it's not going that fast, but in a small test it did fix the dates.
Yes, I'm aware fastcopy has a verification option, but my current plan is to leave that off, then run rclone check, then throw the old hdd in a closet :-p
How about this suggestion from @asdffdsa - does that work for you? I think the emphasis on the missing is good as that is generally what people are concerned about.
I like rclone check the best - but I'm biased! I use it to verify everything!
I would like at some point to get rclone copy local -> local to preserve everything so you don't need fastcopy any more.
PS I have a closet with a lot of old hard disks.... About 50 at last count And that is after I destroyed about 50 with a sledge hammer recently.
The --differ, --missing-on-dst, --missing-on-src and --src-only flags write paths, one per line, to the file name (or
stdout if it is -) supplied. What they write is described in the
help below. For example --differ will write all paths which are
present on both the source and destination but different.
The --combined flag will write a file (or stdout) which contains all
file paths with a symbol and then a space and then the path to tell
you what happened to it. These are reminiscent of diff files.
= path means path was found in source and destination and was identical
- path means path was missing on the source, so only in the destination
+ path means path was missing on the destination, so only in the source
* path means path was present in source and destination but different.
Flags:
--combined string Make a combined report of changes to this file
--differ string Report all non-matching files to this file
--match string Report all matching files to this file
--missing-on-dst string Report all files missing from the destination to this file
--missing-on-src string Report all files missing from the source to this file
Btw, out of 700,000 files and folders, fastcopy made mistakes in transferring accurately the timestamp data on roughly 300 folders, all of them subfolders of root (out of a ridiculous roughly 3000 subfolders of root). I ranted about it in another thread, robocopy was able to do it perfectly using the option /dcopy:dat when the default is only /dcopy:da
I don't necessarily want to bash on fastcopy, that's an incredibly small number of mistakes, but since I was hyping fastcopy up earlier in this thread, I wanted to come back and give the negatives of the fastcopy life.
tl;dr
If you ever make rclone local -> local perfect in a future year, make it robocopy perfect!
and P.S. I still intend to run rclone check, it's just that 3tb and 700,000 files on an HDD means it's a very long/slow command so I haven't fully completed running the new and improved check yet.
The --one-way was needed because I didn't want to be spammed about files missing on the source. IMO it's sort of implied by --missing-on-dst that you don't want real-time instant feedback on the terminal of files missing on source... at least imo, but hey, that's what the --one-way flag is for I guess. I also expect very few files to be missing, so it would be nice if the list of missing files could be output to the terminal or whatever, but hey, I don't mind checking a log file. It's just weird. rclone has never made me check a log file for standard feedback before.
I only waited about 10minutes before writing this, so, I guess in a day or so I'll report back on what ends up appearing in that log file for me.
edit: Might actually be more like a week until I see that output log file.
check complete, no information was written to the missingfiles.log file, although maybe no data was meant to be written.
I suggest that you write that there were 0 differences if there were 0 differences because by writting nothing... it's not clear the writing to log step even worked. Or write something like the date/time the command was completed. Anything to prove the writing to log file process occurred.
Also there were 13 errors during the process, which I scrolled up and found were all "access denied" files. Which I expected. This information should also probably be repeated either to the terminal or the log file at the end of the process. Just for anyone who wasn't using a stats command. (These are files I don't care about as they're recycling bin and system volume information files).
Another minor quirk is that the terminal refers to the accessed denied files as "differences" they're not actually differences as neither the source nor destination file could be accessed, but if the terminal is going to lump them in and call them differences, it's a bit confusing the --missing-on-dst logfile doesn't lump them in too.
You should have seen that in the normal output of rclone check - just as it exists it writes logs about that.
Hmm, were these files that couldn't be read on the source or the destination? Are they present on the source and destination or just the source or just the dest?
They're actually present on the source and the destination and unreadable on both. It's not really too important, it's a bit of a fringe case.
This was the final terminal output.
2020/07/02 05:20:08 NOTICE: Local file system at //?/A:/: 13 differences found
2020/07/02 05:20:08 NOTICE: Local file system at //?/A:/: 13 errors while checking
2020/07/02 05:20:08 NOTICE: Local file system at //?/A:/: 752291 matching files
2020/07/02 05:20:08 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 14 (retrying may help)
Checks: 752295 / 752295, 100%
Elapsed time: 50h19m13.1s
The 14th error is the fact that there were 13 errors, a phrasing change rclone added a few versions ago, and I scrolled up to confirm the other 13 errors were all accessed denied related. The accessed denied errors weren't repeated at the end though, and didn't appear in the log file. The log file wasn't written to at all.
These were written to the "normal" log which you can capture with --log-file.
I only want to write valid file names to the check logs so they can be used in scripts.
Maybe I should write another one called --errors which has names of files with some sort of error in? You'd still have to look in the main log to find out what sort of error. I could add another symbol to the combined log for this.
I think putting errors into their log might make sense. Potentially with some sort of --errors flag. You might worry that it would be useless without the error message, but, I would not have needed to check the main log to tell what sort of errors these were. Why? Because "System Volume Information" and "$RECYCLE.BIN" are well known system files.
Wanting to keep the check log file as clean as possible so that it can be fed back into other (potentially rclone based) scripts actually makes a ton of sense, I hadn't thought of that. If that's the case, the usage case for the check --errors log file might be based around re-running an rclone command as an administrator on only the files which had errors. Of course that would only help in cases of access denied errors. It does seem like a super duper fringe case though.
I don't mean to nitpick. Everything absolutely worked 100%. I just wanted to report back, and wrap up the train of thought of my thread. The end result ended up with some slightly unexpected feedback, but not in a way that actually impacted me negatively.
And that is after I destroyed about 50 with a sledge hammer recently.