Rclone sync S3 to S3 runs for hours and copy nothing

Help and Support
1

What is the problem you are having with rclone?

I'm trying to make a copy between a ceph bucket on an Object Storage S3 (OVH), but no objects are transferred. The source is readable with rclone ls. On the other hand, copying another bucket, which has fewer objects, from the same source to the destination works.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 5.10.0-19-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.5
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Source : Ceph Object Storage
Destination : OVH S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mkdir prod-ovh:/docapi-fr01
rclone lsd prod-ovh:
          -1 2023-07-06 15:00:41        -1 docapi-fr01
rclone sync prod-ceph:/docapi-fr01 prod-ovh:/docapi-fr01 -vv

The rclone config contents with secrets removed.

[prod-ovh]
type = s3
provider = Other
env_auth = false
access_key_id = XXX
secret_access_key = XXX
acl = private
region = gra
location_constraint = gra
endpoint = https://s3.gra.io.cloud.ovh.net/

[prod-ceph]
type = s3
provider = Ceph
access_key_id = XXX
secret_access_key = XXX
endpoint = XXX
acl = private

A log from the command with the -vv flag

2023/07/06 15:29:35 DEBUG : Creating backend with remote "prod-ceph:/docapi-fr01"
2023/07/06 15:29:35 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/07/06 15:29:35 DEBUG : name = "prod-ceph", root = "/docapi-fr01", opt = &s3.Options{Provider:"Ceph", EnvAuth:false, AccessKeyID:"XXX", SecretAccessKey:"XXX", Region:"", Endpoint:"https://XXX", STSEndpoint:"", LocationConstraint:"", ACL:"private", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:5242880, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:false, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/06 15:29:35 DEBUG : Resolving service "s3" region "us-east-1"
2023/07/06 15:29:35 DEBUG : fs cache: renaming cache item "prod-ceph:/docapi-fr01" to be canonical "prod-ceph:docapi-fr01"
2023/07/06 15:29:35 DEBUG : Creating backend with remote "prod-ovh:/docapi-fr01"
2023/07/06 15:29:35 DEBUG : name = "prod-ovh", root = "/docapi-fr01", opt = &s3.Options{Provider:"Other", EnvAuth:false, AccessKeyID:"XXX", SecretAccessKey:"XXX", Region:"gra", Endpoint:"https://s3.gra.io.cloud.ovh.net/", STSEndpoint:"", LocationConstraint:"gra", ACL:"private", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:5242880, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:false, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false},UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/06 15:29:35 DEBUG : Resolving service "s3" region "gra"
2023/07/06 15:29:35 DEBUG : fs cache: renaming cache item "prod-ovh:/docapi-fr01" to be canonical "prod-ovh:docapi-fr01"

2023/07/06 15:30:35 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       1m0.0s

2023/07/06 15:31:35 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       2m0.0s

2023/07/06 15:32:35 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       3m0.0s

2023/07/06 15:33:35 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:       4m0.0s

[...]
2

Is it very large bucket? millions of objects?

3

Have a look at this thread:

If it applies to you try the latest beta:

sudo -v ; curl https://rclone.org/install.sh | sudo bash -s beta

4

Yes, about 80 million objects.

5

And similar problem but with different underlying cause:

At the minimum you should try:

rclone sync prod-ceph:/docapi-fr01 prod-ovh:/docapi-fr01 --checksum --fast-list --s3-list-version 2 -vv

To see how talking between servers goes add --dump headers - maybe like here one of your servers is not very fast... listing 80 million objects will take time. One HTTP requests gets you 1000 objects - if you see that server does 10 per second it will take 2-3h.

Make sure you have enough RAM.... all listing goes into memory - about 1KB per object. So you need 20GB RAM (source and destination) to absorb it without swapping.

6

Installation fails.

# curl https://rclone.org/install.sh | sudo bash -s beta
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4707  100  4707    0     0  17433      0 --:--:-- --:--:-- --:--:-- 17433
Archive:  rclone-beta-latest-linux-amd64.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of rclone-beta-latest-linux-amd64.zip or
        rclone-beta-latest-linux-amd64.zip.zip, and cannot find rclone-beta-latest-linux-amd64.zip.ZIP, period.

I'll read the thread thanks !

7

I'll give it a try, keep you in touch, thank you.

8

I think no point to investigate it - just grab it from here:

https://beta.rclone.org/v1.64.0-beta.7132.f1a842081/

1 Like
9

With --dump-headers, we can see that there were 80095 HTTP REQUEST, and as many HTTP RESPONSE (200). Processing took about 2 hours.

The server has 30G of RAM. I restarted the copy with the options indicated, with the same result.

1 Like
10

For actual transfer I would recommend to set --checkers 32 --transfers 16 - default is (8/4). You can even try higher values. It helps if you have a lot of small objects.

11

No more success.

12

so it still does nothing?

13

Indeed. I tried different options (including checkers and transfers).

14

You said that for 2h your see listing going on - does it finish? And then rclone does nothing?

15

Yes, we see HTTP REQUEST and as many HTTP RESPONSE (200) with dump-headers, but rclone does not copy.

16

If you see requests it means it is still listing... it can take few hours with your dataset size. It lists both source and destination at the same time. Only when both listings are finished it will start transferring.

Problem would be if listing is finished (no more network traffic) and it still does nothing.

17

We saw the requests and processing took two hours. No data was copied, however, and a subsequent cron copy didn't copy anything either.

18

So rclone finishes? I was under impression that it sort of freezes....

19

Yes, it ends. I don't understand the problem. We're now looking at a solution for synchronizing directly from the source (rados gateway sync).

20

could you generate DEBUG log?

basic command will suffice:

rclone sync prod-ceph:docapi-fr01 prod-ovh:docapi-fr01 -vv --log-file /path/to/rclone.log