Can rclone save a list of checked flies when copying to use for next time?

Help and Support
1

What is the problem you are having with rclone?

I'm copying a very large number of files (50000+) from one google teamdrive to another with the --drive-server-side-across-configs flag. I'm not able to leave my device running 24/7, so rclone spends a lot of time checking files (this is with checkers set to 50). Is there a way for rclone to save a list of identical files somewhere so that it doesn't need to retrace its steps each day?

Or is this because of my low tpslimit?

What is your rclone version (output from rclone version)

1.53.1

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config ./rclone.conf copy --drive-server-side-across-configs --rc --rc-addr="localhost:5572" -vv --ignore-existing --checkers 50 --tpslimit 3 --transfers 3 --drive-chunk-size 32M --drive-acknowledge-abuse --log-file=log_rclone.txt
2

hello and welcome to the forum,

  • without knowing your use-case, hard to give good advice.
    since you are using --ignore-existing, is it safe to assume, that once a file has been copied from source to dest, the source file will not change?

  • rclone does not have a database list of files.

  • best to use defaults for that.
    during transfer, based on the replies from gdrive, rclone will adjust that.

each time rclone runs, rclone has to scan for new/changed files on the source and compare that to the dest.
if you run the rclone copy each day, might use --max-age=24h.
this would reduce the amount of files that need to be compared between the source and dest.

--drive-chunk-size 32M
as the copy is server-side, do not think that flag does anything.

3

Thanks for the advice, I've removed the tpslimit and drive chunk size flags.

What about --fast-list? Do you think that would help? Also, is 50 a good number of checkers?

My use case is that I need to copy the contents of a very big teamdrive to another. I've already move ~300GB but I think it's several terabytes in size. I close my computer at night so I can only run rclone for ~16 hours per day and it looks like it's taking several of them just to do the checking.

4

  • update rclone to latest stable, v1.57.0

  • gdrive has many quirks and then combine that with --drive-server-side-across-configs,
    not sure what to tell you exactly.

  • imho, about --fast-list, if you are going to run the rclone copy every day.
    then for each run, test different commands and compare the results.

  • perhaps @Animosity022 or @VBB might know.

run rclone on a cheap/free virtual machine from google or some another provider.

5

There's no full command to look at.
There's no rclone.conf to look at.
There's no log file to look at.

Let's get the basics first and then I will look at it. That's why we have the template so we don't go back and forth and we get all the required information in one shot.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.