What is the problem you are having with rclone?
I need to move 2.3 Millions of files from Oracle Cloud Gen1 (Swift) to Oracle Cloud Bucket Gen2 (S3 Compatible).
The commands Sync, Move and Copy don't start the transfer, how can I do that?
What is your rclone version (output from rclone version
)
Rclone Download v1.51.0
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Oracle Linux 7.7 64bit
Which cloud storage system are you using? (eg Google Drive)
Oracle Cloud Gen 1 (Swift) to Oracle Cloud Gen2 (s3 Compliant)
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy Gen1:CONTAINER GEN2:Bucket --progress --verbose
paste command here
rclone copy Gen1:CONTAINER GEN2:Bucket --progress --verbose
#### A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)
<!-- Please **DO NOT REDACT** any information except passwords/keys/personal info. -->
calisro
(Rob)
April 28, 2020, 3:09am
2
What do you mean they don't start the transfer? Logs?
D4oS
April 28, 2020, 10:07am
3
Perhaps it's just taking extremely long to start?
--verbose != -vv
$user@hetz 12:02:08[~] rclone lsd crypt2: -vv
2020/04/28 12:02:17 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "lsd" "crypt2:" "-vv"]
2020/04/28 12:02:17 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
-1 2017-09-24 23:09:22 -1 home
-1 2019-06-10 21:14:02 -1 kopia_lapka
2020/04/28 12:02:18 DEBUG : 7 go routines active
2020/04/28 12:02:18 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "lsd" "crypt2:" "-vv"]
$user@hetz 12:02:18[~] rclone lsd crypt2: --verbose
-1 2017-09-24 23:09:22 -1 home
-1 2019-06-10 21:14:02 -1 kopia_lapka
This is happening:
[root@rclone opc]# rclone sync GEN1:CONTAINER GEN2:Bucket --progress --verbose
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 0.0s
The transfer doesn't start.
Do you have 2 million files in a single directory? Can you use --fast-list as that'll take more memory but would return. You should use -vv rather than verbose so you can see more information.
When you say the transfer does not start, are you waiting a second? 30 seconds? 5 minutes?
Yes, 2 million files in a single Directory.
I have waited more than 20 hours. I put the process on screen and after when I see it didn't start.
I tried to use --fast-list, but the samething is happening.
2020/04/28 12:31:38 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 0.0s
calisro
(Rob)
April 28, 2020, 12:35pm
7
You really need to add -vv so you know what's happening. And share the log.
ncw
(Nick Craig-Wood)
April 28, 2020, 5:12pm
8
I suspect it is listing the directory which is taking a very long time. You can get some idea by watching the rclone memory use go up.
Note that you'll need lots of memory for 2.3 million files in one directory.
system
(system)
Closed
June 28, 2020, 1:14pm
9
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.