Hi,
I'm trying to synchronize two buckets, one from Wasabi (source) to AWS S3 (destination) to keep a safe full copy somewhere.
What is the problem you are having with rclone?
When syncing two buckets, rclone does not seem to see the files in the destination (AWS).
I tried different buckets with different permissions (Enable public access, disable ACL, disable versioning, changing object Owner to the writer, bucket policy ) ... but every time rclone is copying the entire source bucket to AWS and replacing the files that were already here and identical. Our goal is to sync directly to Glacier Deep Archive. I tried using Standard to see if there was a difference but same problem.
At first, I was synchronizing from Wasabi to local storage without any problem. I switched to AWS and the check are all replaced by transfers and I'm not sure where the problem is from.
Run the command 'rclone version' and share the full output of the command.
rclone v1.65.1
- os/version: debian 12.4 (64 bit)
- os/kernel: 6.1.0-17-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.5
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Source : Wasabi S3. The user has full access to the bucket.
Destination : AWS S3.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Script.sh
wasabi_bucket=$1
local_datalocation=$2
aws_bucket="XXXX-nl-test"
rclone sync WasabiCN:"$wasabi_bucket" AWS-nicolas:"$aws_bucket/$local_datalocation" --update --checksum --fast-list --create-empty-src-dirs --ignore-errors --progress --metadata --multi-thread-streams=16 --transfers=16 --checkers=32 --log-file="$local_datalocation"log/"$wasabi_bucket".log --log-level=DEBUG --exclude=/.tt_rt/**
CLI
sh /data/script.sh "XXXXXX-archives-2020-2024" "/data/Wasabi1/"```
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[AWS-nicolas]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = ca-central-1
location_constraint = ca-central-1
server_side_encryption = AES256
storage_class = DEEP_ARCHIVE
[AWS-nicolasSTD]
type = s3
provider = AWS
access_key_id = XXX
secret_access_key = XXX
region = ca-central-1
location_constraint = ca-central-1
server_side_encryption = AES256
storage_class = STANDARD
[WasabiCN]
type = s3
provider = Wasabi
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.ca-central-1.wasabisys.com
### Double check the config for sensitive info before posting publicly
A log from the command that you were trying to run with the -vv flag
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/cats/kublai97.jpg: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/cats/kublai98.jpg: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/cats/kublai99.jpg: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/barrent-mission-trois-mois.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/barrez-vous-foutez-camp.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/battent-pour-autorise-encourage.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/beau-geste-pardon.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/beau-quand-meme.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/beaucoup-sensationnel-comme-ca.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/ben-moi-non-plus.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/besoin-quelquun-mission-confiance.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bestiole-gueule-nom-age.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bibi-commandes-projet-petarade-mieux.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bien-bouffe-non.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bien-manger-important.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bien-marrer-branche.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bienvenue-casser-graine-maison.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bigleux-bite-pisser.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/blase.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bohort-frustre.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/boire-un-coup-assure.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bon-allez-jeteins.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bon-bouquin.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bon-comme-ca-hein.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bon-gre-mal-gre.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bon-jabandonne.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonheur-solitude-errance.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonjour-bienvenue-une-connerie.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonjour-la-pedagogie.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonjour-restez-assis.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonne-mine-bien-reposee.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonne-nuit-alors.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonne-nuit.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bonnes-choses-ont-une-fin.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bosse-jour-armee-pas-stand-crepes.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bouffe-en-plus.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bouffer-les-rognons.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bouffer-nez-conneries.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bouh-hi.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bouh.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/boulot-evitera-penser-reste.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bravo-parlez-heros.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bravo-pouvez-flamber.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bretagne-fond-du-trou.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/bricole-en-plus-fait-plaisir.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/brique-tarte.gif: Need to transfer - File not found at Destination
2024/01/30 11:20:31 DEBUG : NL_AWS_2024/gif/buffet-a-vaisselle.gif: Need to transfer - File not found at Destination
...
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/blase.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bon-comme-ca-hein.gif: md5 = 1d308516cfc940ca967d4eb412466f63 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bon-comme-ca-hein.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonheur-solitude-errance.gif: md5 = 8e4c538232079f4011847f6196a6f338 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonheur-solitude-errance.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/battent-pour-autorise-encourage.gif: md5 = f86d8efc14f568316b726660de154e2d OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/battent-pour-autorise-encourage.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 56 with 5242880 bytes and etag "8a96124f8a62da11a11e51586d3382b1"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 56/66 (288358400-293601280) size 5Mi finished
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bon-jabandonne.gif: md5 = b196a557ac17ec9e558e9748949d5cff OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bon-jabandonne.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonjour-la-pedagogie.gif: md5 = 3a105ea0ad051acc71525f28c84737ae OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonjour-la-pedagogie.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 48 with 5242880 bytes and etag "f798ef0402b4aba36490cbd934db0b7c"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 48/66 (246415360-251658240) size 5Mi finished
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonnes-choses-ont-une-fin.gif: md5 = 4c23d7868fe8c252b32cd6f7bfb3dac6 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonnes-choses-ont-une-fin.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bouffe-en-plus.gif: md5 = 54ec9e5e53894b19aad9da8716828861 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bouffe-en-plus.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonne-nuit-alors.gif: md5 = a4e0f6b537599e260b85c5b169528da5 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonne-nuit-alors.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bouffer-nez-conneries.gif: md5 = 0c6ba62361ad6c39ebc21f8c904e3498 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bouffer-nez-conneries.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bon-gre-mal-gre.gif: md5 = 74ca70c84648ec877786266a6e797903 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bon-gre-mal-gre.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bosse-jour-armee-pas-stand-crepes.gif: md5 = 39f59e932673622619adc5990604fdbb OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bosse-jour-armee-pas-stand-crepes.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonjour-bienvenue-une-connerie.gif: md5 = 6f26f2e234a3aa8c8f6a2bd9c576c229 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonjour-bienvenue-une-connerie.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bouh.gif: md5 = 32d69ce3a05428e72e1eb350b94be891 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bouh.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonne-mine-bien-reposee.gif: md5 = 5b0725f0ec4e93b9db54204c5cff521e OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonne-mine-bien-reposee.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bravo-parlez-heros.gif: md5 = 6a6eb54df96ab1d4c6c229076117d6dd OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bravo-parlez-heros.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/boulot-evitera-penser-reste.gif: md5 = 8cf1c8edc0d10b53171ebc05d2e1230f OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/boulot-evitera-penser-reste.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 55 with 5242880 bytes and etag "2d988f5c7c3fa4edac369206d9345b25"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 55/66 (283115520-288358400) size 5Mi finished
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bouffer-les-rognons.gif: md5 = 53a69946626cbadb2636a69217af8a93 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bouffer-les-rognons.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bonne-nuit.gif: md5 = 481bd9ff681200ec45b2e9cc5b2bd894 OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bonne-nuit.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 53 with 5242880 bytes and etag "51f3a75a3236cfeb847d49b466df0b83"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 53/66 (272629760-277872640) size 5Mi finished
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 61 with 5242880 bytes and etag "de4d28a2771ddbe0174c02ce2f5026c5"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 61/66 (314572800-319815680) size 5Mi finished
2024/01/30 11:20:38 DEBUG : NL_AWS_2024/gif/bouh-hi.gif: md5 = 846dd69db1e17e9c5001243c7e5016ab OK
2024/01/30 11:20:38 INFO : NL_AWS_2024/gif/bouh-hi.gif: Copied (new)
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 58 with 5242880 bytes and etag "b6d61fd31ba166be7e63a8734a34aeb6"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 58/66 (298844160-304087040) size 5Mi finished
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 50 with 5242880 bytes and etag "e7946b5e2954350312cb0a8a0e4b5dfd"
2024/01/30 11:20:38 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 50/66 (256901120-262144000) size 5Mi finished
2024/01/30 11:20:39 DEBUG : NL_AWS_2024/gif/bonjour-restez-assis.gif: md5 = 7e24e51fc27572107c1f64fa3fc113a1 OK
2024/01/30 11:20:39 INFO : NL_AWS_2024/gif/bonjour-restez-assis.gif: Copied (new)
2024/01/30 11:20:39 DEBUG : NL_AWS_2024/gif/bravo-pouvez-flamber.gif: md5 = 1c1c418068bdb775697598bc12d98206 OK
2024/01/30 11:20:39 INFO : NL_AWS_2024/gif/bravo-pouvez-flamber.gif: Copied (new)
2024/01/30 11:20:39 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multipart upload wrote chunk 57 with 5242880 bytes and etag "07cfa920b52010b1646b4c579d86f2bf"
2024/01/30 11:20:39 DEBUG : 1b0f70fcdf1080965b571403/x64-Windows10.0-KB4486153-x64.cab: multi-thread copy: chunk 57/66 (293601280-298844160) size 5Mi finished
2024/01/30 11:20:39 DEBUG : NL_AWS_2024/gif/brique-tarte.gif: md5 = f7ce1c54a03d8e544ad875af69e06f05 OK
2024/01/30 11:20:39 INFO : NL_AWS_2024/gif/brique-tarte.gif: Copied (new)
2024/01/30 11:20:39 DEBUG : NL_AWS_2024/gif/buffet-a-vaisselle.gif: md5 = aed35013b74d4c7cd413b81c160aacd0 OK
2024/01/30 11:20:39 INFO : NL_AWS_2024/gif/buffet-a-vaisselle.gif: Copied (new)
2024/01/30 11:20:39 DEBUG : NL_AWS_2024/gif/bricole-en-plus-fait-plaisir.gif: md5 = a2b79b8a4ad69dfacdd6813466f05911 OK
2024/01/30 11:20:39 INFO : NL_AWS_2024/gif/bricole-en-plus-fait-plaisir.gif: Copied (new)
2024/01/30 11:20:39 DEBUG : NL_AWS_2024/gif/bretagne-fond-du-trou.gif: md5 = 3a5d5056a726ea9553969bd031be405c OK
2024/01/30 11:20:39 INFO : NL_AWS_2024/gif/bretagne-fond-du-trou.gif: Copied (new)
Thank you so much for your help!