"Excluded from sync (and deletion)" log meaning after rclone copy/move

Help and Support
1

What is the problem you are having with rclone?

I have a directory of about 50000 text files that I am trying to copy to s3 but I get this log after a copy/move operation is complete. This happens for 6 minutes or so until it starts copying again even though I scheduled a copy for every 90s. I was wondering what this meant and why this long delay occurs.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

AWS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --bwlimit 12M

A log from the command with the -vv flag

2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-114.txt: Excluded from sync (and deletion)
2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-115.txt: Excluded from sync (and deletion)
2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-116.txt: Excluded from sync (and deletion)
2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-117.txt: Excluded from sync (and deletion)
2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-118.txt: Excluded from sync (and deletion)
2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-119.txt: Excluded from sync (and deletion)
2022/03/01 11:19:50 DEBUG : 2022/02/23/00/rclone-move-testfile-12.txt: Excluded from sync (and deletion)
2

hello and welcome to the forum,

the command snippet you posted does not match the log snippet?

hard to tell, as most the help and support questions were not answered?

3

Hi,

Sorry I didn't include the entire command, but this is the whole copy command (I ran it as a subprocess within a python script).

 [
        "rclone",
        "-vv",
        f"--bwlimit={bandwidth_limit_bytes}",
        "copy",
        Path(local_dir),
        dest,
        "--order-by=modtime,descending",
    ]
4

yeah, i also script rclone using python.

can you add -vv to the command and post the top 30 lines?

5

Here's the top few lines

2022/03/01 11:09:40 INFO  : Starting bandwidth limiter at 12Mi Byte/s
2022/03/01 11:09:40 DEBUG : --max-age 1w to 2022-02-22 11:09:40.912502 -0800 PST m=-604799.963138835
2022/03/01 11:09:40 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "--bwlimit=12M" "--log-file=./out2.log" "copy" "<local_src_path>" "localhost:<dest_path>" "--order-by=modtime,descending" "--max-age=7d"]
2022/03/01 11:09:40 DEBUG : Creating backend with remote "<local_src_path>"
2022/03/01 11:09:40 DEBUG : Using config file from "<local_rclone.conf_path>"
2022/03/01 11:09:40 DEBUG : Creating backend with remote "localhost:<dest_path>"
2022/03/01 11:09:41 DEBUG : test1.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:09:41 DEBUG : test1.txt: Unchanged skipping
2022/03/01 11:09:41 DEBUG : test.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:09:41 DEBUG : test.txt: Unchanged skipping
2022/03/01 11:09:41 DEBUG : tst4.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:09:41 DEBUG : tst4.txt: Unchanged skipping

2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-106.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-106.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-107.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-107.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-256.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-256.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-253.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-253.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-259.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-259.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-258.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-258.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-257.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-257.txt: Unchanged skipping
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-250.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:11:11 DEBUG : 2022/02/25/01/rclone-move-testfile-250.txt: Unchanged skipping

after about a 1000 lines of the above logs it gets to this:

2022/03/01 11:19:49 DEBUG : 2022/02/23/21/rclone-move-testfile-14.txt: Unchanged skipping
2022/03/01 11:19:49 DEBUG : 2022/02/23/21/rclone-move-testfile-5.txt: Unchanged skipping
2022/03/01 11:19:49 DEBUG : 2022/02/23/21/rclone-move-testfile-3.txt: Unchanged skipping
2022/03/01 11:19:49 DEBUG : 2022/02/23/21/rclone-move-testfile-0.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/03/01 11:19:49 DEBUG : 2022/02/23/21/rclone-move-testfile-0.txt: Unchanged skipping
2022/03/01 11:19:49 DEBUG : S3 bucket <path>: Waiting for transfers to finish
2022/03/01 11:19:49 INFO  : There was nothing to transfer
2022/03/01 11:19:49 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:             50048 / 50048, 100%
Elapsed time:      10m8.6s

2022/03/01 11:19:49 DEBUG : 5 go routines active
2022/03/01 11:19:49 INFO  : Starting bandwidth limiter at 12Mi Byte/s
2022/03/01 11:19:49 DEBUG : --min-age 1w to 2022-02-22 11:19:49.620827 -0800 PST m=-604799.966162329
2022/03/01 11:19:49 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "--bwlimit=12M" "move" "--log-file=./out2.log" "<local_src_path>" "localhost:<dest_path>" "--min-age=7d" "--order-by=modtime,descending" "--delete-empty-src-dirs" "--fast-list"]
2022/03/01 11:19:49 DEBUG : Creating backend with remote "<path>"
2022/03/01 11:19:49 DEBUG : Using config file from "<rclone_path>"
2022/03/01 11:19:49 DEBUG : Creating backend with remote "localhost:<dest_path>"
2022/03/01 11:19:49 DEBUG : tst4.txt: Excluded
2022/03/01 11:19:49 DEBUG : test1.txt: Excluded
2022/03/01 11:19:49 DEBUG : test.txt: Excluded
2022/03/01 11:19:49 DEBUG : 2022/02/23/00/rclone-move-testfile-0.txt: Excluded from sync (and deletion)
2022/03/01 11:19:49 DEBUG : 2022/02/23/00/rclone-move-testfile-1.txt: Excluded from sync (and deletion)
2022/03/01 11:19:49 DEBUG : 2022/02/23/00/rclone-move-testfile-10.txt: Excluded from sync (and deletion)

Sorry if I didn't fully understand what you needed, hope this helps!

6

yes, that helps, the complete command which has --max-age=7d

rclone has to check every source file against the filter --max-age=7d

if the source file does not match the filter, and thus rclone will not copy that source to dest.
then rclone will output Excluded from sync (and deletion)

7

Ahh got it, thank you! Is there a way to speed up the process of checking for this max age?

8

is there a specifc reason to use --order-by=modtime,descending?
that can slow rclone.

rclone has to check every source file and decide to what to do.
--- copy the file to dest
or
--- not not copy the file to dest.

9

Well the reason I had in mind was specifically because we wanted the newest files to be sent first.

So what I gather from what you said is in the case that I have, there is not alternative to reduce the execution time because rclone must check each file for the parameters I specified. So in the case of --max-age, this is unavoidable?

10

not sure your use-case and why the newest files must upload first.
but i would test without using --order-by

for a given source file, rclone has to decide what to do with it.
--max-age is not slowing down rclone, as the files are local.
what will slow down rclone is comparing the source file in fast local to the dest file in slow cloud.

in fact, using --max-age will speed up the entire command, as rclone has less files to check against slow dest.

1 Like
11

Just as an FYI, removing --order-by didn't have any effect. Not too sure, but as I start transferring larger files (.ats) and bigger directories, I have a strong feeling that this might scale to be quite a long amount of time (for reference right now its about 15 minutes).

12

ok, small number of local files.

for each local file that is uploaded to s3, rclone will first, calculate the md5 hash and then start the upload.
the larger file size, the longer amount of time needed to calculate that md5 hash.

13

Sorry, one last question - in theory, would not calculating the md5 checksum result in faster running? If so, is the way to do that through the --s3-disable-checksum flag?

14

correct.
i see that you take the time to read the documentation, so you know the downside to that flag.

1 Like
15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.