Rclone sync copy a parte of the bucket not all files

Help and Support
1

What is the problem you are having with rclone?

i run this cmd : rclone sync -vv --ignore-existing --exclude "directory/*" src:/bucket-src dest:bucket-destination. The bucket source has 600GO when the sync finished i had the message:
Transferred: 2.540 MiB / 2.540 MiB, 100%, 23.028 KiB/s, ETA 0s
Checks: 1496368 / 1496368, 100%
Deleted: 5 (files), 0 (dirs)
Transferred: 9 / 9, 100%
Elapsed time: 2h7m58.8s

200 GO is not copied. It seems like rcolone doesn't notice other files in bucket source ??

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

I used OVH object storage.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vv --ignore-existing --exclude "directory/*" src:/bucket-src dest:bucket-destination

2

Post content of your debug log file. The answer is most likely there.

BTW. What do you need --ignore-existing flag for?

3

Because i already copy some of the files. In debug file there i have this
emise-de-decorations.jpg: Destination exists, skipping
2024/07/02 11:47:10 DEBUG : toto.jpg: Destination exists, skipping
2024/07/02 11:47:10 DEBUG : tata.pdf: Destination exists, skipping
....
2024/07/02 11:47:26 INFO :
Transferred: 8.013 GiB / 8.013 GiB, 100%, 4.231 MiB/s, ETA 0s
Checks: 1493435 / 1493435, 100%
Deleted: 994 (files), 0 (dirs)
Transferred: 3929 / 3929, 100%
Elapsed time: 2h9m33.9s

4

Yes. So all works as you requested in your command.

BTW - you do not need --ignore-existing to skip already copied files. This flag usage is a bit different - check docs.

I think you have to remove this flag and then it will work - run first with -vv --dry-run to check.

5

in fact it works but it doesn't copy all the files. i still have 200GB in my source bucket that is not copied to the destination. however, the sync cmd is terminated without error !

6

when using --dry-run, rclone will not copy the files. need to check the debug output.
if that looks ok, then remove remove --dry-run and rclone should copy the files.

7

I still have the same issue :
buket-src GRA S3 3207052 665.5 GB
bucket-dest GRA S3 1531709 402.2 GB

It doesn't copy all the content ?

8

pick a single file that is missing on the dest.
run rclone copy on that single file
and/or
run rclone check

either way, now is the time to post details, such as debug logs

9

How do you get bucket size? Have you tried ‘rclone size’?

10

The bucket size i got it from the cloud ovh interface. for the debug, i already post it there is no error mentioned.
2024/07/02 11:47:10 DEBUG : toto.jpg: Destination exists, skipping
2024/07/02 11:47:10 DEBUG : tata.pdf: Destination exists, skipping
2024/07/10 15:33:50 DEBUG : 6d095.jpeg: Unchanged skipping
2024/07/10 15:33:50 DEBUG : 00579c.jpeg: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/07/10 15:33:50 DEBUG : 00579c.jpeg: Unchanged skipping
2024/07/10 15:33:50 DEBUG : vf.pdf: Size and modification time the same (differ by 0s, within tolerance 1ns)
2024/07/10 15:33:50 DEBUG : 31f.pdf: Unchanged skipping
2024/07/10 15:33:50 DEBUG : d9f72ab72a76.jpeg: Size and modification time the same (differ by 0s, within tolerance 1ns)
..........
2024/07/02 11:47:26 INFO :
Transferred: 8.013 GiB / 8.013 GiB, 100%, 4.231 MiB/s, ETA 0s
Checks: 1493435 / 1493435, 100%
Deleted: 994 (files), 0 (dirs)
Transferred: 3929 / 3929, 100%
Elapsed time: 2h9m33.9s

11

that might include other information, such as versions.
run rclone size on the source and dest.

that is just a short log snippet, that shows rclone comparing files and that there is nothing to copy.
so rclone seems to be working fine.

you keep posting that a few times. now, we need a specific example.
pick a file that exists in the source but rclone will not copy to the dest.

as mentioned, should run rclone check

12

In addition you use some exclusions:

Maybe this is what you think about as "missing" when actually you excluded it?

13

It s not easy to find a file not copied. the bucket has a thousand of file....I assume that the size is not the same so it mean the there is some files missing in the destination bucket. i can't post all the debug file. it contains the same lines that post like skipping files and it end up by mention the time...etc.

14

no it s not that. there is 200 GB is missing

15

It is very easy. Please read posts.

This will tell you all differences - see docs for details.

These are just your words. You want help please provide some facts. Good start could be run rclone size on both remotes as whatever your provider tools are showing might be simething different.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.