Does --min-age/--max-age slow down rclone move significantly?

Help and Support
1

What is the problem you are having with rclone?

Whenever I run rclone move with --min-age, it takes 20-25 minutes before it actually starts moving directories. In that time, it just gives me the "Excluded from sync (and deletion)" logs during that time. I assume this is because it checks the age of each file in the directory and excludes the ones that don't fill the criteria but I was hoping to speed up this process as it starts to take up several hours when I move from small text files like i used in the logs to larger files and directories.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

Which cloud storage system are you using? (eg Google Drive)

AWS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

subprocess.run(
        [
            "rclone",
            "-vv",
            f"--bwlimit=12M",
            "move",
            Path(local_dir),
            dest,
            f"--min-age=7d",
            "--order-by=modtime,descending",
            "--delete-empty-src-dirs",
            "--fast-list",
        ]
    )

A log from the command with the -vv flag

Here's what it looks like when I use --min-age:

2022/03/01 11:19:49 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "--bwlimit=12M" "move" "<local_src_path>" "localhost:<dest_path>" "--min-age=7d" "--order-by=modtime,descending" "--delete-empty-src-dirs" "--fast-list"]
2022/03/01 11:19:49 DEBUG : Creating backend with remote "<path>"
2022/03/01 11:19:49 DEBUG : Using config file from "<rclone_path>"
2022/03/01 11:19:49 DEBUG : Creating backend with remote "localhost:<dest_path>"
2022/03/01 11:19:49 DEBUG : tst4.txt: Excluded
2022/03/01 11:19:49 DEBUG : test1.txt: Excluded
2022/03/01 11:19:49 DEBUG : test.txt: Excluded
2022/03/01 11:19:49 DEBUG : 2022/02/23/00/rclone-move-testfile-0.txt: Excluded from sync (and deletion)
2022/03/01 11:19:49 DEBUG : 2022/02/23/00/rclone-move-testfile-1.txt: Excluded from sync (and deletion)
2022/03/01 11:19:49 DEBUG : 2022/02/23/00/rclone-move-testfile-10.txt: Excluded from sync (and deletion)

That goes on for about 25 minutes. Here's what it looks like when I remove --min-age

2022/03/01 12:01:11 INFO  : Starting bandwidth limiter at 12Mi Byte/s
2022/03/01 12:02:04 INFO  : 2022/03/02/02/rclone-move-testfile-151.txt: Deleted
2022/03/01 12:02:04 INFO  : 2022/03/02/02/rclone-move-testfile-147.txt: Deleted
2022/03/01 12:02:04 INFO  : 2022/03/02/02/rclone-move-testfile-150.txt: Deleted
2022/03/01 12:02:04 INFO  : 2022/03/02/02/rclone-move-testfile-144.txt: Deleted
2022/03/01 12:02:04 INFO  : 2022/03/02/02/rclone-move-testfile-148.txt: Deleted
2

hi,
never found a use for --order-by
but seems to me, rclone would have to scan the entire source first or at least, --max-backlog and then order the files.

and as a test, i would try this and time the result.
--- remove --order-by
--- add --no-traverse
--- add --dry-run

3

Should I do this while leaving in the --min-age flag?

4

well, since you plan to use --min-age, i would testing using that.

tho if i had time, i would test with that flag, and without that flag and compare the result.

and test with --dry-run

5

Got it thank you I'll try both and get back to you once I get results

6

and not knowing your use-case,
might want to know about --no-check-dest

7

I get the exact same logs as I got before, it keeps just giving me logs of the manner

Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:      12m0.0s

along with what I already sent in the beginning of the post.

I left the --min-age in because I need to have some sort of boundary for files being moved. I don't see a reason as to why to test it without --min-age since I already know that it doesn't behave like this and immediately begins deleting files.

8

hard to understand, need to post the exact command you tested?

sure, that logs would be the same/similar, but i thought that goal was to reduce the amount of time down from 25 minutes before starting to transfer files?

9

Sorry, should have provided more clarity, the command was:

subprocess.run(
        [
            "rclone",
            "-v",
            f"--bwlimit=12M",
            "move",
            "--no-traverse",
            "--dry-run",
            Path(local_dir),
            dest,
            f"--min-age=7d",
            "--delete-empty-src-dirs",
            "--fast-list",
        ]
    )

What I meant to say is I get no change in either the logs being produced or time the time being taken before the transfer of files.

10

can you post the real command, not the function that runs it
or the top 20+ lines of the debug log.

and --no-traverse should make a difference as per https://rclone.org/commands/rclone_copy/

and might try without --fast-list
" It may be slower because it can't be parallelized"

11

Ah you were right, I accidentally copied information from the wrong log file. Here's the first few lines of the log:

2022/03/10 12:52:24 INFO  : Starting bandwidth limiter at 12Mi Byte/s
2022/03/10 12:52:24 DEBUG : --min-age 1w to 2022-03-03 12:52:24.764258 -0800 PST m=-604799.967216303
2022/03/10 12:52:24 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "--bwlimit=12M" "move" "--no-traverse" "--dry-run" "--log-file=./out-3-15.log" "<>" "localhost:<>" "--min-age=7d" "--delete-empty-src-dirs" "--fast-list"]
2022/03/10 12:52:24 DEBUG : Creating backend with remote "<>"
2022/03/10 12:52:24 DEBUG : Using config file from "<>"
2022/03/10 12:52:24 DEBUG : Creating backend with remote "localhost:<>"
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-64.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-306.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-460.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-474.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-312.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-70.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-58.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-448.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-662.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-676.txt: Excluded
2022/03/10 12:52:24 DEBUG : 2022/02/25/01/rclone-move-testfile-886.txt: Excluded

It also ran much faster than what it used to, only took a few seconds. Sorry only saw your comment about --fast-list after I sent this - I can try that too.

12

sweet, 25 minutes to a few seconds!

13

But wouldn't --no-traverse also be slower if there was a much more larger number of files in the source? For example if there was more files in the source than files in the destination wouldn't --no-traverse slow it down? I might not be understanding it correctly please let me know :slight_smile:

14

well, it depends on your use case, and the mix of files.
as i understand it, it is not the total number of source files, but the number of sources files that need to be transferred.

but at this point, you get the idea, to test/mix flags and test with --dry-run
so you can answer your own question, run the test and share your results.

1 Like
15

I have ran into an issue during the --no-traverse in the move command. While I get logs like

{"level":"info","msg":"Starting bandwidth limiter at 12Mi Byte/s","source":"accounting/token_bucket.go:107","time":"2022-03-10T16:56:44.591921-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/08/16","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.747135-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/08","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.748818-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/05/03","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.749101-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/05/02","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.749346-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/05/01","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.749636-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/05/00","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.749925-08:00"}
{"level":"info","msg":"Removing directory","object":"2022/03/05","objectType":"string","source":"operations/operations.go:1105","time":"2022-03-10T16:56:44.750164-08:00"}

when I check these directories, nothing is deleted and it stays there regardless. The command I ran is the same as what was up in the previous comment. I don't exactly know how to show you that the directory has information in it but it does when I check it.

16

if the command you ran is the same, then why does that log look different.

need to post the full debug log or at least the top 20+ lines of the debug log which will have the exact command.

rclone ls

17

Sorry, I didn't use -vv, I'll post the full debug in a couple minutes.

18

Nevermind, I figured it out, it was my --min-age that was set to a wrong value :upside_down_face:

19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.