Is it possible to disable checkers?

#6

You need to make and use your own client API/key:

https://rclone.org/drive/#making-your-own-client-id

[felix@gemini ~]$ rclone copy GD: --files-from filesfrom.txt /home/felix/out/ -vv
2019/04/05 14:56:48 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "GD:" "--files-from" "filesfrom.txt" "/home/felix/out/" "-vv"]
2019/04/05 14:56:48 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/05 14:56:49 INFO  : Local file system at /home/felix/out: Waiting for checks to finish
2019/04/05 14:56:49 INFO  : Local file system at /home/felix/out: Waiting for transfers to finish
2019/04/05 14:56:50 INFO  : test/file049.bin: Copied (new)
2019/04/05 14:56:50 INFO  :
Transferred:   	   13.396k / 13.396 kBytes, 100%, 6.652 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:          2s

2019/04/05 14:56:50 DEBUG : 4 go routines active
2019/04/05 14:56:50 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "copy" "GD:" "--files-from" "filesfrom.txt" "/home/felix/out/" "-vv"]

What’s actually in your files from file? I tested with a few files in there and works super fast. I kicked it off at the root of my GD: and I have almost 60TB of data.

#7

Are you writing to another Google Drive mount?

2019/04/05 14:39:08 DEBUG : a8/a816830000000000000000000000000000000000000000000000000000000000: Failed to pre-allocate: operation not supported
2019/04/05 14:39:08 INFO  : a8/a816830000000000000000000000000000000000000000000000000000000000: Copied (new)
2019/04/05 14:39:08 DEBUG : a2/a216270000000000000000000000000000000000000000000000000000000000: Failed to pre-allocate: operation not supported

Is this a mount?

2019/04/05 14:36:15 INFO  : Local file system at /home/bmmcginty/gdrivewtf/copy2: Waiting for transfers to finish
#8

I created a test directory for this test. It’s got 256 subdirectories, with 10 files per eachd irectory. (I realize this is odd, but it works well for storing hashed content, for example.) I do have my own client_id configured, as well.

#9

Based on your errors, you aren’t using your own API key as the log is littered with 403 rate limits which means you are not using your own API key.

If you are writing back to a Google Mount drive, it goes back to creating 3 files per second so it takes a long time with a lot of little files.

When you create a key, you need to run through rclone config and add it in.

#10

It’s a local filesystem path.

#11

What’s the file system?

#12

The FS is ext3.
I checked my project via the google console, and I am using my own API key.
It doesn’t make sence, though, that RClone is running all these checks, and _then downloading the files, rather than passing those checked files off to start being downloaded immediately.
It’s like they’re being queued for some indeterminit amount of time or reason.

#13

Can you share you files file? I’m happy to create a bunch of test files and try it out as I can’t reproduce with 30 files in random directories in a large GD.

#14

You can pull the files with --drive-root-folder-id 1LMEvB-pzcrkIg2h2IT9N6qKxi07qmhoe
or get them here (they’ll unzip into your current directory, so you’ll want to be in an empty dir):
0x0.st/zZ8e.zip
Thanks.

#15

also, file used with --files-from is here:
0x0.st/zZKt.txt

#16

I got all the files down, but the txt file seems to not be found for me.

#17

Okay, try:
http://sprunge.us/iZum6M
(I assume your issue was retrieving the --files-from file, which is relinked above.)
Thank you much.

#18

That helps as I can definitely recreate the issue now.

@ncw - I believe relates back to the number of files in the files list as it has over 2000 files and it seems to burst and look for all the files before transferring?

I can see for the files in the from, it seems to do an API hit for each file. If I ramp it up, the list / gets error out like crazy.

@bmmcginty - what’s the use case in having the list? Perhaps there is a different way to achieve it?

#19

Does --max-backlog have any effect if that is set low?

#20

Doesn’t seem to as I tried setting it to 5 and 10.

#21

The original set of files needed from the gdrive directory is much smaller than the directory itself, but large enough to not want to run Rclone a couple thousand times. This is still true half way through the transfer, as well.

#22

I was more asking what is generating the file list?

You’d finish much much quicker looping through it many times than trying to let it run this way.

I did something easy like

for i in `cat from`; do rclone copy GD:tt/$i /home/felix/zz -P; done

I was trying to understand why you can’t filter/regex it and do it that way rather than dumping a file list. filtering handles it much better.

#23

Honestly, it never occurred to me to use filters instead of files-from. I still think the files-from is a bug, and I’m wondering if I should open an issue on Github.
All that being said, include-from works perfectly, and starts pulling files almost immediately. Thanks!
To answer an earlier question, The original include-from file was generated dynamically from a secondary database.

1 Like
#24

Sweet!

I’d probably open an issue on it as it doesn’t seem to be working well with large number of files.

#25

Ah, I think this is related to

There is a tentative fix in that thread, but I was trying to think of a better one!