What is the problem you are having with rclone?
I'm trying to check files' existence in dest with command
rclone check source:/bucket dest:/bucket --one-way --files-from local_file_list_path --no-traverse .
local_file_list_path contains 100,000,000+ lines of objectKeys, it is a large number and I'm trying to avoid to list source bucket or head every object. Is there any way to make it possible that rclone check process doesn't request the source bucket but just check files from the "files-from" exist in dest bucket?
Run the command 'rclone version' and share the full output of the command.
- os/arch: linux/amd64
- go version: go1.11.5
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
rclone check source:/bucket dest:/bucket --one-way --files-from local_file_list_path --no-traverse
The rclone config contents with secrets removed.
type = s3
provider = Other
env_auth = false
access_key_id = xxx
secret_access_key = xxx
endpoint = xxx
list_chunk = 10000
read_only = true
A log from the command with the
I just had a quick look at the code and
rclone check seems to be ignoring
--no-traverse which it shouldn't be.
v1.59.0-beta.6085.c615e72d0.fix-check-flags on branch fix-check-flags (uploaded in 15-30 mins)
Thanks so much. I may check this later because of other urgent works.
Ok, let me know if it it works and I'll merge it into the main code.
Sorry for late! I execute that same command above which is
rclone check source:/bucket dest:/bucket --one-way --files-from local_file_list_path --no-traverse. And HEAD requests to source bucket still called as before. What I am trying to achieve is checking existence of filepaths in dest bucket. So, HEADing the source bucket is unnecessary for me. In addition, millions of HEAD requests cost a lot.
Maybe you should list the destination bucket to a file
rclone lsf -R --files-only dest:/bucket | sort > dest-files
then use the
comm tool to find the differences (this assumes local_file_list_path is sorted and contains only files - if not you'll need to sort/edit it or regenerated it with rclone).
comm -23 local_file_list_path dest_files
This will give you a list of all files which are in the source but not the dest - use
-13 for files which are in the dest but not the source or use
-3 to see both at once.