Syncing OR Copying Large Number of Files from Rackspace to S3 causes Hang

Help and Support
1

What is the problem you are having with rclone?

I have been trying for a few days now to copy millions of files from Rackspace CloudFiles storage into AWS S3. One of the containers that I am currently trying to copy from has around 10Million files within it.

I started running a sync 2 hours ago now and I'm still at 0 bytes transferred. Is there a way to speed up this process, or is something wrong. I have checked both the source and destination have connection, by copying over a small directory from the source to the destination, which worked fine. Is this a limitation with the number of files?

This is being run on an AWS EC2 Linux Instance, with RCLone Version 1.55.1

Trying to Migrate from Rackspace CloudFiles into Amazon S3

The command run is:
rclone sync -i -P -vv --fast-list rackspace:src-dir remote:dest-dir

The output is:

2021/06/29 09:18:52 DEBUG : Creating backend with remote “rackspace:src-dir”
2021/06/29 09:18:52 DEBUG : Creating backend with remote “remote:dest-dir”
Transferred:  	    0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:   2h7m6.6s```
2

hello and welcome to the forum,

if you use -i, rclone is waiting for you to tell it what to do.

you did not post the config file, so i cannot tell what backend rackspace uses.
if it is s3, then you might want to use --checksum, as documented at
https://rclone.org/s3/#avoiding-head-requests-to-read-the-modification-time

3

Hello,

Sorry for not providing the config, please see the following:

====                 ====
rackspace            swift
remote               s3

I have tried removing -i and I have tried adding --checksum but still nothing happens

4

something has to happen, only way to know if to use a debug log and post it to the forum.

if this is the first time trying to transfer the files, i would use rclone copy

post the config file itself, not just the name of the remotes, redact id/secrets

5

Hi @asdffdsa,

I have been running with -vv see below the response we get

2021/06/29 12:33:03 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "-P" "-vv" "--fast-list" "--transfers" "64" "--checksum" "rackspace:src-dir" "remote:dest-dir"]
2021/06/29 12:33:03 DEBUG : Creating backend with remote "rackspace:src-dir
2021/06/29 12:33:03 DEBUG : Creating backend with remote "remote:dest-dir"
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:      3m14.7s

Here is the information from the config file:

type = s3
provider = AWS
env_auth = true
region = eu-west-2
location_constraint = eu-west-2
acl = private

[rackspace]
type = swift
env_auth = false
user = ******
key = *************
auth = https://lon.auth.api.rackspacecloud.com/v1.0
user_id = 3
endpoint_type = public
6

try adding --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file rclone.log
run the command for a couple of minutes
kill the command
post the rclone.log file

7

Quick question before I post the log (it's quite large) Does RClone first need to read through all of the files to compile a list / get information on progress?

I attempted this on a smaller dataset and it seemed to work perfectly fine, could it be that rclone is still calculating the 10,000,000 files?

8

rclone copy should start to copy without the need to read through all of the files.

can you post the top part of the log to pastebin.com or some such service.
does not need to be the entire log, perhaps the first two hundred lines.