Help understanding checking with --check-first

Help and Support
1

What is the problem you are having with rclone?

Understanding the relationship between checkers and transfers.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0
- os/version: Microsoft Windows 11 Pro 23H2 (64 bit)
- os/kernel: 10.0.22631.2715 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.4
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

"C:\\Users\\Chad\\Documents\\PowerShell\\Modules\\TMCTools\\bin\\rclone.exe" "move" "--checkers" "8" "--config" "C:\\Users\\Chad\\Domains\\cdsconsulting.co\\logs\\Move-GOSharedDrives_20231204_124234\\TMCTools (Source).conf" "--log-file" "C:\\Users\\Chad\\Domains\\cdsconsulting.co\\logs\\Move-GOSharedDrives_20231204_124234\\TMCTools (Source).log" "--low-level-retries" "3" "--retries" "1" "--stats" "5s" "--stats-file-name-length" "0" "--temp-dir" "C:\\Users\\Chad\\Domains\\cdsconsulting.co\\logs\\Move-GOSharedDrives_20231204_124234" "--user-agent" "ISV|Transend Corporation|TMCTools/1.0" "--log-level" "DEBUG" "--drive-skip-dangling-shortcuts" "--drive-skip-shortcuts" "--drive-show-all-gdocs" "--check-first" "--create-empty-src-dirs" "--order-by" "size,mixed" "--transfers" "4" "--update" "--server-side-across-configs" "Source:BigData" "Target:BigData"

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Source]
type = drive
team_drive = ...PVA
scope = drive
service_account_file = C:\Temp\key.json
impersonate = user@example.org

[Target]
type = drive
scope = drive
service_account_file = C:\Temp\key.json
impersonate = user2@example.org
team_drive = ...PVA

A log from the command that you were trying to run with the -vv flag

2023/12/04 12:43:07 DEBUG : Google drive root 'BigData': Waiting for checks to finish
2023/12/04 12:43:07 INFO  : Google drive root 'BigData': Checks finished, now starting transfers
2023/12/04 12:43:07 DEBUG : Google drive root 'BigData': Waiting for transfers to finish
2023/12/04 12:43:11 INFO  : 
Transferred:   	          0 B / 147.758 MiB, 0%, 0 B/s, ETA -
Checks:                 0 / 4, 0%
Transferred:            0 / 50453, 0%
Elapsed time:        36.8s
Checking:
 * microsoft-graph-docs/.git/HEAD:  0% /21, 0/s, -, moving
 * microsoft-graph-docs/.git/objects/pack/pack-6a0150eebdca9b44b3cc100e828006b940e05353.idx:  0% /25.658Mi, 0/s, -, moving
 * microsoft-graph-docs/.git/objects/pack/pack-6a0150eebdca9b44b3cc100e828006b940e05353.pack:  0% /771.079Mi, 0/s, -, moving
 * microsoft-graph-docs/.git/refs/remotes/origin/HEAD:  0% /30, 0/s, -, moving

2023/12/04 12:43:14 INFO  : microsoft-graph-docs/.git/refs/remotes/origin/HEAD: Moved (server-side)
2023/12/04 12:43:15 INFO  : microsoft-graph-docs/.git/objects/pack/pack-6a0150eebdca9b44b3cc100e828006b940e05353.idx: Moved (server-side)
2023/12/04 12:43:16 INFO  : microsoft-graph-docs/.git/HEAD: Moved (server-side)
2023/12/04 12:43:16 INFO  : microsoft-graph-docs/.git/objects/pack/pack-6a0150eebdca9b44b3cc100e828006b940e05353.pack: Moved (server-side)

Before transfers start, I see that checking has completed:

2023/12/04 12:43:07 INFO  : Google drive root 'BigData': Checks finished, now starting transfers

But then, during the transfer phase, the logs show the files being checked again:

Checking:
 * microsoft-graph-docs/.git/HEAD:  0% /21, 0/s, -, moving
 * microsoft-graph-docs/.git/objects/pack/pack-6a0150eebdca9b44b3cc100e828006b940e05353.idx:  0% /25.658Mi, 0/s, -, moving
 * microsoft-graph-docs/.git/objects/pack/pack-6a0150eebdca9b44b3cc100e828006b940e05353.pack:  0% /771.079Mi, 0/s, -, moving
 * microsoft-graph-docs/.git/refs/remotes/origin/HEAD:  0% /30, 0/s, -, moving

And the running total of checks is initialized to zero:

Checks:                 0 / 4, 0%

When using --check-first, do all files get checked up front and then checked again during the transfer stage? If that's the case, I'm not sure what I'm gaining using --check-first.

2

I think the second one is not really "Checking", despite the name. "Checking" can mean a lot of different things:

(from What is rclone checking during a purge?)

I think this is the one you're seeing, which is really more "moving" than "checking":

3

You are correct.

It does say moving on the end of the line, but it isn't particularly obvious!

I should probably rename the Checkers to something else - any suggestions?

4

So the checkers perform their named role of checking files but also have additional tasks that come into play during the transfer phase? If so, I can't think of another name off the top of my head that seems better. Workers, maybe, but transfer threads also perform work, so that's confusing, too.

In my log, the checkers say they are performing moves, which seems odd to me. Since this is a server-side operation with Google Drive, shouldn't that be a rename? If it really is a move, why wouldn't it be happening via a transfer thread?

5

Back in the mists of time, rclone only did two things transfers and checks, but as rclone grew the checkers started doing more and more things. They are much more like a set of workers yet, but I think that might be even more confusing.

These moves are really server side moves - renames on Google Drive. The underlying primitive in rclone is called Move which is where the name comes from, but I agree it could be clearer!

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.