What is the fastest way to rclone backend restore-status with 2+ millions objects

Help and Support
1

Problem I am having with rclone

In rclone v1.64 rclone updated rclone backend restore-status which works with few number of files.

but I have 2+ millions objects in Scaleway Glacier about 3TB in Size. Now I need to restore to 'Standard' and check restore-status.

What is the fastest way to rclone backend restore-status command with 2+ millions objects?

I tried with --fast-list

I am using a 4 Core CPU, 8 GB RAM, Ubuntu 22.04 LTS

'rclone version' output.

rclone v1.64.0

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-83-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.1
  • go/linking: static
  • go/tags: none

Cloud storage system I am using.

Scaleway

Command I am trying to run

rclone backend restore-status -o all --verbose --progress --fast-list "scaleway:bucket"

'rclone config redacted' full output.

[SW_Scaleway_GLC]
type = s3
provider = Scaleway
env_auth = false
access_key_id = XXX
secret_access_key = XXX
region = nl-ams
endpoint = s3.nl-ams.scw.cloud
acl = private
storage_class = GLACIER
max_upload_parts = 1000
2

And what is speed of your command? Has it finished for 2m objects?

3

With the command its finishing after about 24 minutes. though I have added --max-backlog 1000000 --transfers 30000 --checkers 60000

but its seems --max-backlog 1000000 --transfers 30000 --checkers 60000 has no effect on it. without --max-backlog 1000000 --transfers 30000 --checkers 60000 its behaving identical

4

what can be the fastest solution for rclone backend restore and rclone backend restore-status?

5

restore-status is quite a low level command and I don't think any of the flags will speed it up.

It will effectively be doing --fast-list due to the way it works.

The only thing I can think of would be to run this from a scaleway VM in the same datacenter to reduce roundtrip latency on the API.

6

which flag will speed up the process of rclone backend restore and rclone backend restore-status

7

If you don't mind using up more APIs then increasing --checkers will speed up rclone backend restore. Or you could try --fast-list but I think more --checkers will help more.

There aren't any flags which will speed up rclone backend restore-status - that is quite a low level command.