Rclone check head or list object from source

Help and Support
1

What is the problem you are having with rclone?

I'm trying to check files' existence in dest with command rclone check source:/bucket dest:/bucket --one-way --files-from local_file_list_path --no-traverse .
the file local_file_list_path contains 100,000,000+ lines of objectKeys, it is a large number and I'm trying to avoid to list source bucket or head every object. Is there any way to make it possible that rclone check process doesn't request the source bucket but just check files from the "files-from" exist in dest bucket?

Run the command 'rclone version' and share the full output of the command.

rclone v1.47.0

  • os/arch: linux/amd64
  • go version: go1.11.5

Which cloud storage system are you using? (eg Google Drive)

Xsky

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone check source:/bucket dest:/bucket --one-way --files-from local_file_list_path --no-traverse

The rclone config contents with secrets removed.

[xxxx]
type = s3
provider = Other
env_auth = false
access_key_id = xxx
secret_access_key = xxx
endpoint = xxx
list_chunk = 10000
read_only = true

A log from the command with the -vv flag

ignoring this
2

I just had a quick look at the code and rclone check seems to be ignoring --no-traverse which it shouldn't be.

Try this

v1.59.0-beta.6085.c615e72d0.fix-check-flags on branch fix-check-flags (uploaded in 15-30 mins)

3

Thanks so much. I may check this later because of other urgent works.

4

Ok, let me know if it it works and I'll merge it into the main code.

5

Sorry for late! I execute that same command above which is rclone check source:/bucket dest:/bucket --one-way --files-from local_file_list_path --no-traverse. And HEAD requests to source bucket still called as before. What I am trying to achieve is checking existence of filepaths in dest bucket. So, HEADing the source bucket is unnecessary for me. In addition, millions of HEAD requests cost a lot.

6

Maybe you should list the destination bucket to a file

rclone lsf -R --files-only dest:/bucket | sort > dest-files

then use the comm tool to find the differences (this assumes local_file_list_path is sorted and contains only files - if not you'll need to sort/edit it or regenerated it with rclone).

comm -23 local_file_list_path dest_files

This will give you a list of all files which are in the source but not the dest - use -13 for files which are in the dest but not the source or use -3 to see both at once.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.