How to move 2 millions of files?

What is the problem you are having with rclone?

I need to move 2.3 Millions of files from Oracle Cloud Gen1 (Swift) to Oracle Cloud Bucket Gen2 (S3 Compatible).

The commands Sync, Move and Copy don't start the transfer, how can I do that?

What is your rclone version (output from rclone version)

Rclone Download v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Oracle Linux 7.7 64bit

Which cloud storage system are you using? (eg Google Drive)

Oracle Cloud Gen 1 (Swift) to Oracle Cloud Gen2 (s3 Compliant)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy Gen1:CONTAINER GEN2:Bucket --progress --verbose

paste command here

rclone copy Gen1:CONTAINER GEN2:Bucket --progress --verbose

#### A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)
<!-- Please **DO NOT REDACT** any information except passwords/keys/personal info. -->

What do you mean they don't start the transfer? Logs?

Perhaps it's just taking extremely long to start?
--verbose != -vv

$user@hetz 12:02:08[~] rclone lsd crypt2: -vv
    2020/04/28 12:02:17 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "lsd" "crypt2:" "-vv"]
    2020/04/28 12:02:17 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
              -1 2017-09-24 23:09:22        -1 home
              -1 2019-06-10 21:14:02        -1 kopia_lapka
    2020/04/28 12:02:18 DEBUG : 7 go routines active
    2020/04/28 12:02:18 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "lsd" "crypt2:" "-vv"]
    $user@hetz 12:02:18[~] rclone lsd crypt2: --verbose
              -1 2017-09-24 23:09:22        -1 home
              -1 2019-06-10 21:14:02        -1 kopia_lapka

This is happening:

[root@rclone opc]# rclone sync GEN1:CONTAINER GEN2:Bucket --progress --verbose
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 0.0s

The transfer doesn't start.

Do you have 2 million files in a single directory? Can you use --fast-list as that'll take more memory but would return. You should use -vv rather than verbose so you can see more information.

When you say the transfer does not start, are you waiting a second? 30 seconds? 5 minutes?

Yes, 2 million files in a single Directory.

I have waited more than 20 hours. I put the process on screen and after when I see it didn't start.

I tried to use --fast-list, but the samething is happening.

2020/04/28 12:31:38 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 0.0s

You really need to add -vv so you know what's happening. And share the log.

I suspect it is listing the directory which is taking a very long time. You can get some idea by watching the rclone memory use go up.

Note that you'll need lots of memory for 2.3 million files in one directory.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.