With using copy --no-traverse --files-from-raw, it takes 1-5 minutes to start copying

What is the problem you are having with rclone?

with using the following command, it takes 1-5 minutes to start copying.

rclone --stats 0 --log-level DEBUG --no-traverse copy --files-from-raw files_from_raw.txt remote_s3_compatible_site:bucket local_dir

Therefore, it seems that "Local file system at local_dir: Waiting for checks to finish" is slow.
It would be appreciated if you could have any suggestion to avoid the slow.

What is your rclone version (output from rclone version)

1.52.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu

Which cloud storage system are you using? (eg Google Drive)

S3 compatible

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --no-traverse --stats 0 --log-level DEBUG copy --files-from-raw a.txt hogehoge_s3_compatible:bucket local_dir
2020/07/01 09:12:44 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone" "--no-traverse" "--stats" "0" "--log-level" "DEBUG" "copy" "--files-from-raw" "a.txt" "hogehoge_s3_compatible:bucket" "local_dir"]
2020/07/01 09:12:44 DEBUG : Using config file from "/home/wsl/.config/rclone/rclone.conf"
2020/07/01 09:12:44 DEBUG : Using v2 auth
2020/07/01 09:14:54 DEBUG : Local file system at local_dir: Waiting for checks to finish
2020/07/01 09:14:54 DEBUG : Local file system at local_dir: Waiting for transfers to finish
2020/07/01 09:14:54 INFO  : SABM/alphanumeric/surface/ship/2020063023/A_SNAG01SABM302300RRA_C_RJTD_20200630230000_001.txt: Copied (new)

What is in

files_from_raw.txt ?

Try adding --size-only to the transfer so rclone doesn't have to HEAD each object to work out its modtime (or --checksum or --use-server-modtime --update)

How many lines in the files_from_raw.txt file?

If there are a huge number, you'll probably be better off not using --no-traverse - --no-traverse works best for copying a small number of files into a large file tree.

What is the directory structure like - is it lots and lots of files in a single directory?

Thank you for your kind reply.

I could avoid the slow with the following options.

rclone --ignore-checksum --ignore-existing --no-traverse --no-update-modtime --size-only --stats 0 --log-level DEBUG copy --files-from-raw files_from_raw.txt

The number of lines of files_from_raw.txt is 150. The lines are as follows.

/RJTD/bufr/satellite/mhs/2020063023/A_INMC46RJTD302308_C_RJTD_20200630230800_001.bin
/RJTD/bufr/satellite/mhs/2020063023/A_INMC47RJTD302308_C_RJTD_20200630230800_001.bin
/RJTD/bufr/satellite/mhs/2020063023/A_INMC48RJTD302308_C_RJTD_20200630230800_001.bin
/SABM/alphanumeric/surface/ship/2020063023/A_SNAG01SABM302300_C_RJTD_20200630230000_001.txt
/SABM/alphanumeric/surface/ship/2020063023/A_SNAG01SABM302300RRA_C_RJTD_20200630230000_001.txt
/SBBR/bufr/surface/aws/2020063023/A_ISAI01SBBR302300RRD_C_RJTD_20200630230000_002.bin
/WBSB/alphanumeric/upper_air/pilot_mobil/2020060300/A_UQBD20WBSB010000_C_RJTD_20200603000000_001.txt
...

Anyway, the problem is solved. Thanks a lot ♪.

Great :slight_smile: Glad that is working for you.

This is quite interesting. Had not noticed --files-from-raw previously.

I run a differential check + move on a set of ~200k files where 40 - 50k can change each week. The diff_files.txt file contains a single list with those 50k files. Understandably it takes quite a long time before the move begins using --files-from. Would you recommend --files-from-raw in this use case? (the txt file format is compatible). [[ I'm testing now. But thought might be helpful to know your view in any case, for others. ]]

--files-from-raw just disables the comment parsing and the space removal (which was a mistake in --files-from but I didn't want to break backwards compat.)

So I don't think you'll see much difference.

:weary:

Still looking forward to the new flags in rclone check

These are in the latest beta now coming to 1.53 soon!

Indeed! Just waiting (no hurry) for it to hit master before editing the script :slight_smile:

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.