Unsynchronized directories (S3 storage)

Help and Support
1

Hello,

I'm using Rclone to back up a private Nextcloud. However, I'm encountering a problem for which I can't find a solution, hence my message today.

Rclone only backs up certain folders... Below, you'll find all the commands so you can contextualize.

Thank you,
Biniou

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1-DEV
- os/version: debian 12.5 (64 bit)
- os/kernel: 6.1.0-18-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --delete-before "/media/disk3_4T_SSD/nextcloud/data" scw:nextcloud-rclone/backup_hebdo

Current remotes:

Name                 Type
====                 ====
scw                  s3

/root/.config/rclone/rclone.conf

[scw]
type = s3
provider = Scaleway
env_auth = false
access_key_id = PRIVATE
secret_access_key = PRIVATE
endpoint = s3.fr-par.scw.cloud
acl = private
region = fr-par

A log from the command that you were trying to run with the -vv flag

root@nextcloud:/# rclone sync -vv --delete-before "/media/disk3_4T_SSD/nextcloud/data" scw:nextcloud-rclone/backup_hebdo
2024/04/28 00:20:44 DEBUG : rclone: Version "v1.60.1-DEV" starting with parameters ["rclone" "sync" "-vv" "--delete-before" "/media/disk3_4T_SSD/nextcloud/data" "scw:nextcloud-rclone/backup_hebdo"]
2024/04/28 00:20:44 DEBUG : Creating backend with remote "/media/disk3_4T_SSD/nextcloud/data"
2024/04/28 00:20:44 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/04/28 00:20:44 DEBUG : Creating backend with remote "scw:nextcloud-rclone/backup_hebdo"
2024/04/28 00:20:44 DEBUG : Waiting for deletions to finish
2024/04/28 00:21:20 DEBUG : S3 bucket nextcloud-rclone path backup_hebdo: Waiting for checks to finish
2024/04/28 00:21:20 DEBUG : S3 bucket nextcloud-rclone path backup_hebdo: Waiting for transfers to finish
...
...
...
2024/04/28 00:23:36 DEBUG : S3 bucket nextcloud-rclone path backup_hebdo: Waiting for transfers to finish
2024/04/28 00:23:36 INFO  : There was nothing to transfer
2024/04/28 00:23:36 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:             19978 / 19978, 100%
Elapsed time:      2m51.8s

2024/04/28 00:23:36 DEBUG : 4 go routines active

Here are the folders on the Scaleway S3 storage and locally on my server:

root@nextcloud:/# rclone lsd scw:nextcloud-rclone/backup_hebdo/Antoine/files
           0 2024-04-28 00:17:31        -1 Documents
           0 2024-04-28 00:17:31        -1 Documents partagés
           0 2024-04-28 00:17:31        -1 Médias
           0 2024-04-28 00:17:31        -1 Sauvegardes

root@nextcloud:/media/disk3_4T_SSD/nextcloud/data/Antoine/files# ls -lia
total 40
2123999 drwxr-xr-x 10 www-data www-data 4096 Mar 24 18:46  .
2123990 drwxr-xr-x  7 www-data www-data 4096 Mar 23 18:23  ..
2124245 drwxr-xr-x 23 www-data www-data 4096 Apr  2 21:46  Documents
2124247 drwxr-xr-x  3 www-data www-data 4096 Mar 23 18:23 'Documents partagés'
2124248 drwxr-xr-x  2 www-data www-data 4096 Mar 23 17:32  Ecoles
2124249 drwxr-xr-x  2 www-data www-data 4096 Mar 23 17:32  Informatique
2124250 drwxr-xr-x  2 www-data www-data 4096 Mar 23 17:32 'Jeux vidéos'
2124251 drwxr-xr-x  5 www-data www-data 4096 Mar 24 18:43  Médias
2124252 drwxr-xr-x  2 www-data www-data 4096 Mar 23 17:32  Notes
2124253 drwxr-xr-x  3 www-data www-data 4096 Apr  5 00:15  Sauvegardes
2

welcome to the forum,

the version you are using is from 2022.
perhaps run rclone selfupdate and test again.

are those folders empty?

3

Hello and thank you for your initial response.

I didn't realize I was using an old version of rclone... I managed to update it and I'm now on version 1.66.

My folders were indeed empty, I didn't think that could affect their presence on S3 storage... I just reran my command, adding a file to an empty directory. Now, this directory along with the file are appearing correctly on Scaleway.

Do you know if all S3 storages react in the same way?

It's working now.

Thank you very much!!

4

well, you were using an old version of rclone, which, by default, did not copy empty folders.
for that, needed to use --create-empty-src-dirs

whereas, the latest rclone, introduced a major backwards incompatible change/bug.
now, rclone does sync empty dirs.

this is yet another issue with rclone.
unlike most other s3 tools, by default, rclone will refuse to create an empty folder in s3.
NOTICE: S3 bucket zork path folder: Warning: running mkdir on a remote which can't have empty directories does nothing
to workaround around, need to use --s3-directory-markers

5

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.