Sorry if I asked this before, but I guess I still don't understand. I want my cryptcheck command to run faster, and I'm already using --one-way. Could I also add --no-traverse to it?

Help and Support
1

Okay so here's the usage case.

I have a folder that has say 40,000 files on the cloud, and I put say 20 new files into it on my local machine. I want to then upload those files to the cloud, check --no-traverse makes this go 1000% faster.

Then sometime after that I want to run cryptcheck, and I only care, are those 20 new files a match? I don't care about the rest of the folder.

This is the command I run:

rclone cryptcheck --one-way -v --fast-list --drive-chunk-size 128M "%localpath%" "cleancrypt:%localpath%"

This command ends up taking about 100 times as long as the rclone copy I used to upload the files in the first place. Now I will say, I use --no-traverse when uploading in this case, because I know the files are newly named and totally unique (because they didn't used to exist).

So, I guess the question really is.... can --no-traverse work with --one-way on a cryptcheck? I guess not? because that wouldn't make any sense?

The funny thing though is I could just run something like I dunno...

rclone cryptcheck --one-way -v --fast-list --drive-chunk-size 128M "%localpath%\EXACTFILENAME" "cleancrypt:%localpath%\EXACTFILENAME"

twenty different times.... but that's a lot of extra work for my fingers, rather than for my computer... there should be some sort of way to make my folder wide cryptcheck into a file by file cryptcheck?

2

This is because it is checking all the files, not just the ones you uploaded.

If you had a list of the files you just uploaded you could run add --files-from with that list in to your cryptcheck and rclone would check only those.

However I note that when you do the initial upload of the 20 files, rclone does the exact same check cryptcheck does after it has uploaded the files.

3

Yes, I'm aware rclone performs the check during upload.
However this scenario arises when I'm trying to free up some local disk drive space.
So sometimes there has been a period of years between the initial upload and the need to perform the check again. There's never once been a problem, but I figure I might as well check.

More precisely what the command would look like would be:

rclone cryptcheck --one-way -v --fast-list --drive-chunk-size 128M "%localpath%\filesIdecidedtodelete" "cleancrypt:%localpath%"

I guess I left that part out because I didn't want to make my command look too confusing.

4

That will check all the files in "%localpath%\filesIdecidedtodelete" - if there aren't too many in there then it shouldn't take too long.

How many files are in there?

5

%localpath%\filesIdecidedtodelete the last time I did this had roughly 30 files in it.
but cleancrypt:%localpath% contained 40,000 files.

it took like 30-60minutes or so maybe, basically, it took exactly as much wasted time as a similar copy command would add (beyond the actual copying, before copying the first file) when --no-traverse wasn't used in the command.

given that the amount of time that appears to be "lost" is the exact amount of time --no-traverse saves, well, I dunno! I don't really understand code that well, It's just that antecdotally this task loses the exact amount of time I used to lose on another similar task (which was then fixed by you adding and then me starting to use the --no-travese flag for this task).

Sorry if this is all very confusing. The reason I mention the no-traverse is because ideally I'd try running --no-traverse with cryptcheck, although for some reason... in my brain, I imagine it won't work.

6

where do these 20 files come from?

do the 20 files have something in common, like a creation date, modification date or anything in common that is different from the other 40,000 files?

is there a way to put the file names into a text file?
if so, then you can use the flag --files-from

you are a windows user, correct?
perhaps this command can help forfiles

7

Is there a reason you're actually running that check?

8

well I figure if I'm going to delete forever a file off my local drives, I should verify my cloud backup exists before doing it. I mean ideally I'd keep both copies, and I usually do, but well, some things are just not that important, and so I do this.

9

That's fine but the only reason I mentioned it is the check is doing exactly what the copy is doing in the first place. So when you're doing a copy/sync its checking all the files are on the source and dest and copying them up or syncing them. When that completes without error, its like doing the check.

10

twenty different times.... but that's a lot of extra work for my fingers, rather than for my computer... there should be some sort of way to make my folder wide cryptcheck into a file by file cryptcheck

i tend to run paranoid and i would want to verify before delete.

here is how i do it.
let's say you have a folder with 2000 files and you want to check 10 of them

  1. you need to get a list of files. to do that i use double commander.
    using dc, i can copy a list of files to the clipboard and then paste the clipboard into a file named filelist.txt

image
image698×527 49.8 KB

this would be the filelist.txt
image

  1. now that i have a list of files i want to cryptcheck, i create a .cmd batch file and run it.
    the local folder is C:\data\2000files
    the remote folder is secret:\2000files
    the list of files is filelist.txt

this is the one line .cmd batch file
for /f %%u in (filelist.txt) do rclone.exe cryptcheck C:\data\2000files\%%u secret:/2000files/ --one-way

11

you can also do it this way.
you will still need to create the filelist.txt file.
rclone.exe cryptcheck C:\data\2000files secret:2000files --files-from=filelist.txt --one-way

but using
for /f %%u in (filelist.txt) do rclone.exe cryptcheck C:\data\2000files\%%u secret:/2000files/ --one-way
is a generalized way to take a list of files and run any command, not just rclone.

12

I think --no-traverse will work with cryptcheck :slight_smile:

13

okay, wow, yeah, I tried it, --no-traverse does appear to be a valid cryptcheck command, I don't know why I thought it wasn't. I guess the rclone documentation doesn't specifically say it is, but it also didn't specifically say it wasn't!

If this is indeed intended, maybe add it here? https://rclone.org/docs/#no-traverse
It says copy and move but not sync, but doesn't address check/cryptcheck. But I guess... commands have inheritance? so they inherit functions from other commands, I guess I should've just tried it rather than assuming it didn't work based on documentation.

edit: I tried to reply to this thread but I keep getting internal server error from the forum?

anyways here is the reply I wanted to write today:

So, yeah, the flag reads as valid, but, the benefits don't actually work. So, maybe it isn't worth editing the documentation, as this flag option on cryptcheck doesn't actually matter at all.

In a recent example 79 local files, 3000 files on the remote, cryptcheck took 6 minutes before the very first file actually checked. Which in a followup test without the --no-traverse flag on cryptcheck's like this also took 6 minutes before the first file was checked. That time is about the time I typically save on copy commands in situations like this where I choose to use the no-traverse flag. I think perhaps it's paradoxically unusable in this case for reasons above my comprehension.

edit2: To firm up this test case, I created 35 smallish test files which I then copy'd to the same target folder using --no-traverse, time 27seconds then I used the same command without --no-traverse and despite the fact that this time there wasn't even any copying to do, it takes more than 37minutes (I just cancelled it after waiting that long because I got bored)... so in conclusion, --no-traverse does in fact register as valid for cryptcheck, but has no benefits or function that I can see, but additionally maybe it doesn't need to function because this hellish slowdown is specific to copy not cryptcheck.

14

It affects any command which takes two remotes so sync/copy/move/check/cryptcheck.

It is probably a good idea with --one-way if there are a small number of files in the source, but a bad idea otherwise.

Fancy sending a pull request for the docs? You could use the online editor here: https://help.github.com/en/github/managing-files-in-a-repository/editing-files-in-another-users-repository on this file https://github.com/rclone/rclone/blob/master/docs/content/docs.md#--no-traverse

15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.