Sync copies everything

Help and Support
1

What is the problem you are having with rclone?

I followed this guide https://www.scaleway.com/en/docs/encrypt-object-storage-using-rclone-crypt/¹ to setup a bucket with Scaleway that I will use as a destination for my encrypted backup.
For the initial copy I used this command:

$ rclone copy --progress --s3-storage-class=GLACIER some/local/folder/ scaleway-encrypted-bucket:/

This finished after about 24h. Just to check if everything was copied I ran

rclone sync --progress --s3-storage-class=GLACIER some/local/folder/ scaleway-encrypted-bucket:/

and to my surprise everything was copied again! I retried the command using --size-only, --ignore-checksums etc but -vv revealed that it still checks for the MD5 hash and just copies over everything, even though the remote should contain all the files already.

What am I doing wrong here?

What is your rclone version (output from rclone version)

$ rclone --version
rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.4
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

$ uname -a
Linux kn-portable-arch 5.12.13-arch1-2 #1 SMP PREEMPT Fri, 25 Jun 2021 22:56:51 +0000 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

Scaleway C14 cold storage https://www.scaleway.com/en/c14-cold-storage/

The command you were trying to run (eg rclone copy /tmp remote:tmp)

See problem description.

The rclone config contents with secrets removed.

{
    "scaleway-encrypted-bucket: {
        "directory_name_encryption": "true",
        "filename_encryption": "standard",
        "password": "<redacted>",
        "remote": "scaleway:some-bucket",
        "type": "crypt"
    },
    "scaleway": {
        "access_key_id": "<redacted>",
        "acl": "private",
        "bucket_acl": "private",
        "endpoint": "https://some-bucket.s3.nl-ams.scw.cloud",
        "env_auth": "false",
        "location_constraint": "nl-ams",
        "provider": "Other",
        "region": "nl-ams",
        "secret_access_key": "<redacted>",
        "type": "s3"
    }
}

A log from the command with the -vv flag

See problem description.

¹ I am not allowed to use links, yet.

2

If you run the command and produce a log file, we can see why it's being recopied. Other wise, we can consult our resident physic and see if we can consult the spirit world for guidance.

3

Other wise, we can consult our resident physic and see if we can consult the spirit world for guidance.

No reason for getting sarcastic, sigh.

$ rclone sync --progress --ignore-checksum --size-only -vv --log-file=/tmp/sync.log --s3-storage-class=GLACIER some/local/folder/ scaleway-encrypted-bucket:/

Here are the logs for the command running a couple of seconds:

2021/06/30 14:15:16 DEBUG : Using config file from "/home/some-user/.config/rclone/rclone.conf"
2021/06/30 14:15:16 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "sync" "--progress" "--ignore-checksum" "--size-only" "-vv" "--log-file=/tmp/sync.log" "--s3-storage-class=GLACIER" "some/local/folder" "scaleway-encrypted-bucket:/"]
2021/06/30 14:15:16 DEBUG : Creating backend with remote "some/local/folder/"
2021/06/30 14:15:16 DEBUG : fs cache: renaming cache item "some/local/folder/" to be canonical "/some/local/folder"
2021/06/30 14:15:16 DEBUG : Creating backend with remote "scaleway-encrypted-bucket:/"
2021/06/30 14:15:16 DEBUG : Creating backend with remote "scaleway:some-user-photos"
2021/06/30 14:15:16 DEBUG : scaleway: detected overridden config - adding "{<redacted>}" suffix to name
2021/06/30 14:15:16 DEBUG : fs cache: renaming cache item "scaleway:some-bucket" to be canonical "scaleway{<redacted>}:some-bucket"
2021/06/30 14:15:16 DEBUG : fs cache: switching user supplied name "scaleway:some-bucket" for canonical name "scaleway{<redacted>}:some-bucket"
2021/06/30 14:15:17 DEBUG : 2015-more_md.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:17 DEBUG : Bild0115.gif: Computing MD5 hash of encrypted source
2021/06/30 14:15:17 DEBUG : BFS_128_samples_full_load_jack.png: Computing MD5 hash of encrypted source
2021/06/30 14:15:17 DEBUG : DSC08393.JPG: Computing MD5 hash of encrypted source
2021/06/30 14:15:17 DEBUG : BFS_128_samples_full_load_jack.png: MD5 = 9b6207b3edaf40c7efa14dc3f2787da4 OK
2021/06/30 14:15:17 INFO  : BFS_128_samples_full_load_jack.png: Copied (new)
2021/06/30 14:15:17 DEBUG : DSC08393.JPG: MD5 = c423f769f8e59925006e5579bac3929d OK
2021/06/30 14:15:17 INFO  : DSC08393.JPG: Copied (new)
2021/06/30 14:15:17 DEBUG : Ich2.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:17 DEBUG : Bild0115.gif: MD5 = 6faf09acdd956203c20fe79206e8b2b6 OK
2021/06/30 14:15:17 INFO  : Bild0115.gif: Copied (new)
2021/06/30 14:15:17 DEBUG : OTF_Rust_15.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:18 DEBUG : Ich2.jpg: MD5 = b9c63bead5ed1124ac5196c81c1fa37e OK
2021/06/30 14:15:18 INFO  : Ich2.jpg: Copied (new)
2021/06/30 14:15:18 DEBUG : Rust-6.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:20 DEBUG : Rust-6.jpg: MD5 = e3d21dd0ad465e86d3e03f5a865ffa07 OK
2021/06/30 14:15:20 INFO  : Rust-6.jpg: Copied (new)
2021/06/30 14:15:20 DEBUG : Rust_on_iron.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:20 DEBUG : OTF_Rust_15.jpg: MD5 = e7f767fa3a2bc0ba3f7a335564afce4a OK
2021/06/30 14:15:20 INFO  : OTF_Rust_15.jpg: Copied (new)
2021/06/30 14:15:20 DEBUG : Screenshot from 2016-02-24 02-43-36.png: Computing MD5 hash of encrypted source
2021/06/30 14:15:21 DEBUG : Screenshot from 2016-02-24 02-43-36.png: MD5 = 07b993663626903e0177cb6863e1996a OK
2021/06/30 14:15:21 INFO  : Screenshot from 2016-02-24 02-43-36.png: Copied (new)
2021/06/30 14:15:23 DEBUG : cltextures.com-original-metal-84-rusty-color-metal-texture-5-texturepalace.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:24 DEBUG : 2015-more_md.jpg: MD5 = 8384923cee054eaf424b592d84b7646c OK
2021/06/30 14:15:24 INFO  : 2015-more_md.jpg: Copied (new)
2021/06/30 14:15:25 DEBUG : Ich.tif: MD5 = 461d824b902fb17d35236ed9e0e6caec OK
2021/06/30 14:15:25 INFO  : Ich.tif: Copied (new)
2021/06/30 14:15:25 DEBUG : headphone_repair.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:25 DEBUG : cltextures.com-original-metal-84-rusty-color-metal-texture-5-texturepalace.jpg: MD5 = 120c7f944ee09f3239ddbde875e8b4fc OK
2021/06/30 14:15:25 INFO  : cltextures.com-original-metal-84-rusty-color-metal-texture-5-texturepalace.jpg: Copied (new)
2021/06/30 14:15:25 DEBUG : headphone_repair.jpg: MD5 = 3ead51062f06d2c0dc3b061cbb5cd156 OK
2021/06/30 14:15:25 INFO  : headphone_repair.jpg: Copied (new)
2021/06/30 14:15:27 DEBUG : metal-texture-rust08.jpg: Computing MD5 hash of encrypted source
2021/06/30 14:15:27 DEBUG : netflix_blocks_vpn.png: Computing MD5 hash of encrypted source
...

Note that I cannot upload files as a new user.

4

That's just a joke my friend as humor cures all.

That log only shows new files being copied.

You can use any service like pastebin/etc to use logs.

5

That log only shows new files being copied.

That's the problem, they are not new and already present in the remote.

6

Can you pick one file.

Run a rclone ls on the source, rclone ls one on the remote, rclone copy and share all that.


felix@gemini:~$ rclone ls hosts
      130 hosts
felix@gemini:~$ rclone copy hosts GD:
felix@gemini:~$ rclone copy hosts GD: -vv
2021/06/30 08:34:59 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/06/30 08:34:59 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "hosts" "GD:" "-vv"]
2021/06/30 08:34:59 DEBUG : Creating backend with remote "hosts"
2021/06/30 08:34:59 DEBUG : fs cache: adding new entry for parent of "hosts", "/home/felix"
2021/06/30 08:34:59 DEBUG : Creating backend with remote "GD:"
2021/06/30 08:34:59 DEBUG : GD: detected overridden config - adding "{TKSWb}" suffix to name
2021/06/30 08:34:59 DEBUG : fs cache: renaming cache item "GD:" to be canonical "GD{TKSWb}:"
2021/06/30 08:34:59 DEBUG : hosts: Size and modification time the same (differ by -367.653µs, within tolerance 1ms)
2021/06/30 08:34:59 DEBUG : hosts: Unchanged skipping
2021/06/30 08:34:59 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.3s

2021/06/30 08:34:59 DEBUG : 4 go routines active
felix@gemini:~$ rclone lsl GD:hosts
      130 2021-06-30 08:34:39.455000000 hosts

As one example, that means the file is not on the remote and it copied it as new file so it's either you have a wrong path and it's copying somewhere new or something along those lines. The logs do not indicate the file exists on the same remote/path that it is copying to.

7

Two troubleshotting tips I often use:

rclone lsl some/local/folder --include “headphone_repair.jpg”
rclone lsl scaleway-encrypted-bucket:/ --include “headphone_repair.jpg”

rclone md5sum some/local/folder --include “headphone_repair.jpg”
rclone md5sum scaleway-encrypted-bucket:/ --include “headphone_repair.jpg”
8

Thank you for the quick reply. It is quite tedious to remove all the sensitive information from the logs, hence I will show only a relevant excerpt from the logs.

Syncing a single file works as expected:

$ rclone sync -vv --s3-storage-class=GLACIER some/local/folder/headphone_repair.jpg scaleway-encrypted-bucket:/
<7>DEBUG : Using config file from "/home/some-user/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "sync" "-vv" "--s3-storage-class=GLACIER" "some/local/folder/headphone_repair.jpg", "scaleway-encrypted-bucket:/]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "some/local/folder/headphone_repair.jpg"
<7>DEBUG : fs cache: adding new entry for parent of "some/local/folder/headphone_repair.jpg", "/some/local/folder"
<7>DEBUG : Creating backend with remote "scaleway-encrypted-bucket:/"
<7>DEBUG : Creating backend with remote "scaleway:some-bucket"
<7>DEBUG : scaleway: detected overridden config - adding "{<redacted>}" suffix to name
<7>DEBUG : fs cache: renaming cache item "scaleway:some-bucket" to be canonical "scaleway{<redacted>}:some-bucket"
<7>DEBUG : fs cache: switching user supplied name "scaleway:some-bucket" for canonical name "scaleway{<redacted>}:some-bucket"
<7>DEBUG : headphone_repair.jpg: Size and modification time the same (differ by 0s, within tolerance 1ns)
<7>DEBUG : headphone_repair.jpg: Unchanged skipping
<6>INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.4s

Note that when syncing the whole folder the file that was skipped above is now copied again:

$ rclone sync -vv --s3-storage-class=GLACIER some/local/folder scaleway-encrypted-bucket:/ 2>&1 | tee /tmp/sync.log | grep headphone
<7>DEBUG : headphone_repair.jpg: Computing MD5 hash of encrypted source
<7>DEBUG : headphone_repair.jpg: MD5 = 7eaef9a70d7f1f9eb3b84b7ecae600fe OK
<6>INFO  : headphone_repair.jpg: Copied (new)
^C
9

Thanks, but those commands, lsl and md5sum do not output anything when using --include "headphone_repair.jpg". Interestingly, this will show the md5sum:

$ rclone md5sum ./some/local/folder  | grep headphone
86a2c4cbdc7ee8c67fc046ef0f24c57a  headphone_repair.jpg
^C

Note that rclone md5sum scaleway-encrypted-bucket:/ did nothing, neither with --include "headphone_repair.jpg" nor without.

10

Then I suggest you/we start by finding out why you cannot list the files - before moving onto filters, md5sums and sync.

What have you tried?

11

Here's what I tried to list the contents:

$ rclone lsl -vv scaleway-encrypted-bucket:
<7>DEBUG : Using config file from "/home/some-user/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "lsl" "-vv" "scaleway-encrypted-bucket:"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "scaleway-encrypted-bucket:"
<7>DEBUG : Creating backend with remote "scaleway:some-bucket"
<7>DEBUG : 4 go routines active
$ rclone lsl -vv scaleway-encrypted-bucket:/
<7>DEBUG : Using config file from "/home/some-user/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "lsl" "-vv" "scaleway-encrypted-bucket:/"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "scaleway-encrypted-bucket:/"
<7>DEBUG : Creating backend with remote "scaleway:some-bucket"
<7>DEBUG : 4 go routines active
$ rclone lsl -vv scaleway-encrypted-bucket:/ --include “headphone_repair.jpg”
<7>DEBUG : Using config file from "/home/some-user/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "lsl" "-vv" "scaleway-encrypted-bucket:/" "--include" "“headphone_repair.jpg”"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "scaleway-encrypted-bucket:/"
<7>DEBUG : Creating backend with remote "scaleway:some-bucket"
<7>DEBUG : 4 go routines active
12

This could be a limitation of the c14 Glacier storage type, or a bug in rclone.

13

Your seem familiar with linux, so I will make comments brief (ask if in doubt).

I suggest you make new folders/buckets with simple test content that can be shared without redacting the folder/file name and then try this:
(assuming scaleway-encrypted-bucket is based on scaleway:some-bucket)

ll ./some/local/folder | grep headphone
rclone lsl ./some/local/folder | grep headphone
rclone lsl scaleway:some-bucket
rclone touch scaleway:some-bucket/testfileplain
rclone lsl scaleway:some-bucket
rclone lsl scaleway-encrypted-bucket:
rclone touch scaleway-encrypted-bucket:testfilecrypt
rclone lsl scaleway-encrypted-bucket:
rclone lsl scaleway:some-bucket

References: Commands, rclone ls, rclone touch

PS: No need for -vv or logs at this point, just the normal console output :slight_smile:

1 Like
14

Thank you for the suggestion, I will do this later today.

15

Sorry for the delay.

Here's the config I use for the test:

{
  "scaleway": {
    "access_key_id": "<redacted>",
    "acl": "private",
    "bucket_acl": "private",
    "endpoint": "https://s3.nl-ams.scw.cloud",
    "env_auth": "false",
    "location_constraint": "nl-ams",
    "provider": "Other",
    "region": "nl-ams",
    "secret_access_key": "<redacted>",
    "type": "s3"
  },
  "scaleway-encrypted-bucket": {
    "directory_name_encryption": "true",
    "filename_encryption": "standard",
    "password": "<redacted>",
    "remote": "scaleway:klingtnet-rclone-test-bucket",
    "type": "crypt"
  }
}

As test data I used rclone's git repository, i.e. ./some/local/folder will be ./rclone and headphone will be replaced by README.md. Here's the script, based on the commands you suggested:

#!/usr/bin/env bash

ls -al ./rclone | grep README.md
rclone lsl ./rclone | grep README.md
rclone lsl scaleway:klingtnet-rclone-test-bucket
rclone touch scaleway:klingtnet-rclone-test-bucket/testfileplain
rclone lsl scaleway:klingtnet-rclone-test-bucket
rclone lsl scaleway-encrypted-bucket:
rclone touch scaleway-encrypted-bucket:testfilecrypt
rclone lsl scaleway-encrypted-bucket:
rclone lsl scaleway:klingtnet-rclone-test-bucket

This is the output of test.sh:

$ bash -x test.sh 2>&1 | tee test.log
+ ls -al ./rclone
+ grep README.md
-rw-r--r--  1 alinz alinz    6361 Jul  4 13:32 README.md
+ rclone lsl ./rclone
+ grep README.md
     6361 2021-07-04 13:32:23.370319455 README.md
     3765 2021-07-04 13:32:23.403647128 docs/README.md
     3119 2021-07-04 13:32:23.473635240 librclone/README.md
      167 2021-07-04 13:32:23.466969707 graphics/logo/README.md
     1063 2021-07-04 13:32:23.430309266 fstest/testserver/init.d/README.md
      929 2021-07-04 13:32:23.430309266 fstest/testserver/images/test-hdfs/README.md
      346 2021-07-04 13:32:23.430309266 fstest/testserver/images/test-sftp-openssh/README.md
      853 2021-07-04 13:32:23.426976498 fs/rc/js/README.md
+ rclone lsl scaleway:klingtnet-rclone-test-bucket
        0 2021-07-04 13:55:34.215055494 testfileplain
       32 2021-07-04 13:53:55.641436814 tq5tg8hr3ffiurfq210hf3up8o
+ rclone touch scaleway:klingtnet-rclone-test-bucket/testfileplain
+ rclone lsl scaleway:klingtnet-rclone-test-bucket
        0 2021-07-04 13:55:38.003782142 testfileplain
       32 2021-07-04 13:53:55.641436814 tq5tg8hr3ffiurfq210hf3up8o
+ rclone lsl scaleway-encrypted-bucket:
        0 2021-07-04 13:53:55.641436814 testfilecrypt
+ rclone touch scaleway-encrypted-bucket:testfilecrypt
+ rclone lsl scaleway-encrypted-bucket:
        0 2021-07-04 13:55:40.287158304 testfilecrypt
+ rclone lsl scaleway:klingtnet-rclone-test-bucket
        0 2021-07-04 13:55:38.003782142 testfileplain
       32 2021-07-04 13:55:40.287158304 tq5tg8hr3ffiurfq210hf3up8o
16

No problem at all, we all have other things in life :slight_smile:

Compliments for your choice of test data and picking up the intension of the commands.

Everything looks great so far - that is the config of the remotes is fine.

Now, let’s try a simple copy of README.md (without GLACIER). I would like to see the output from the first execution of rclone copy, you may use --dry-run to test the copy commands.

rclone lsl ./rclone --include=”/README.md”
rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
rclone lsl scaleway-encrypted-bucket: --include=”/README.md”

I have a hunch the issue has something to do with the GLACIER storage class. Let’s also try that out too (without encryption, to make things simpler):

rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
rclone touch scaleway:klingtnet-rclone-test-bucket/testfileglacier --s3-storage-class=GLACIER
rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”/RELEASE.md” --log-level=INFO
rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”/RELEASE.md” --log-level=INFO
rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
17

Here's the output of the first set of commands:

$ bash -x 1.sh
+ rclone lsl ./rclone --include=”rclone/README.md”
+ rclone lsl scaleway-encrypted-bucket: --include=”rclone/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --include=”rclone/README.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         1.0s

+ rclone lsl scaleway-encrypted-bucket: --include=”rclone/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --include=”rclone/README.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.8s

+ rclone lsl scaleway-encrypted-bucket: --include=”rclone/README.md”

Before running the second set of commands I cleared the bucket.

+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
+ rclone touch scaleway:klingtnet-rclone-test-bucket/testfileglacier --s3-storage-class=GLACIER
+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-07 07:19:18.047113937 testfileglacier
+ rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”rclone/RELEASE.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.5s

+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-07 07:19:18.047113937 testfileglacier
+ rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”rclone/RELEASE.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.4s

+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-07 07:19:18.047113937 testfileglacier
18

I also ran a third set which copied into README.md into the encrypted bucket using storage mode GLACIER:

+ rclone lsl ./rclone --include=”rclone/README.md”
+ rclone lsl scaleway-encrypted-bucket: --include=”rclone/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --include=”rclone/README.md” --log-level=INFO --s3-storage-class=GLACIER
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.3s

+ rclone lsl scaleway-encrypted-bucket: --include=”rclone/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --include=”rclone/README.md” --log-level=INFO --s3-storage-class=GLACIER
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.4s

+ rclone lsl scaleway-encrypted-bucket: --include=”rclone/README.md”
+ rclone sync ./rclone scaleway-encrypted-bucket: --include=”rclone/README.md” --log-level=INFO --s3-storage-class=GLACIER
<6>INFO  : There was nothing to transfer
<6>INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.4s

Weirdly, this seems to work just fine.

19

Well actually all three failed :cry:

I better explain the thoughts leading to the list of commands:

I assume you execute the commands from a folder just above the rclone folder like you did in post 15.

I therefore expect ls- al ./rclone/README.md to return a single line like this:

$ ls -al ./rclone/README.md
-rw-r--r--  1 alinz alinz    6361 Jul  4 13:32 README.md

I then start each series of commands by verifying this assumption. I do this by listing the README.md file with rclone like we did in post 15. The only difference is that I use an rclone include filter instead of the grep command and limit the search to the file in the rclone top folder. It should look like this:

$ rclone lsl ./rclone --include=”/README.md”
     6361 2021-07-04 13:32:23.370319455 README.md

Then I copy the file twice. I expect the file to be copied the first time and no copy the second time (because it already exists in the target). These are the copy commands with expected results:

$ rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
# no output, the files doesn’t exist at the target yet

$ rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
2021/07/07 09:49:49 INFO  : README.md: Copied (new)
2021/07/07 09:49:49 INFO  : 
Transferred:        6.324Ki / 6.324 KiByte, 100%, 355.896 KiByte/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.0s

$ rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
         6361 2021-07-04 13:32:23.370319455 README.md

$ rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
2021/07/07 09:54:23 INFO  : There was nothing to transfer
2021/07/07 09:54:23 INFO  : 
Transferred:              0 / 0 Byte, -, 0 Byte/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         1.0s

$ rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
         6361 2021-07-04 13:32:23.370319455 README.md

It seems like you changed the include part of the commands and therefore rclone didn’t find or copy anything, so the tests didn't give us any information. Let's try again :smiley:

20

Sorry for adjusting the --include path, I thought to be clever there but in the end I it already assumed to be run from the parent directory of rclone. So, I ran the tests again, without modifying your commands and this is the result:

$ bash -x test.sh
+ ls -al rclone ./rclone
+ grep README.md
-rw-r--r--  1 alinz alinz    6361 Jul 10 11:25 README.md
-rw-r--r--  1 alinz alinz    6361 Jul 10 11:25 README.md
+ rclone lsl ./rclone --include=”/README.md”
+ rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.5s

+ rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.4s

+ rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-10 11:29:40.040607324 testfileglacier
+ rclone touch scaleway:klingtnet-rclone-test-bucket/testfileglacier --s3-storage-class=GLACIER
<3>ERROR : Attempt 1/3 failed with 1 errors and: touch: couldn't set mod time: can't set modified time
<3>ERROR : Attempt 2/3 failed with 1 errors and: touch: couldn't set mod time: can't set modified time
<3>ERROR : Attempt 3/3 failed with 1 errors and: touch: couldn't set mod time: can't set modified time
Failed to touch: touch: couldn't set mod time: can't set modified time
+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-10 11:29:40.040607324 testfileglacier
+ rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”/RELEASE.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.2s

+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-10 11:29:40.040607324 testfileglacier
+ rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”/RELEASE.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.2s

+ rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
        0 2021-07-10 11:29:40.040607324 testfileglacier
+ rclone lsl ./rclone --s3-storage-class=GLACIER --include=”/README.md”
+ rclone lsl scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.2s

+ rclone lsl scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md”
+ rclone copy ./rclone scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md” --log-level=INFO
<6>INFO  : There was nothing to transfer
<6>INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.3s

+ rclone lsl scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md”

For completeness, here's the test script:

ls -al rclone ./rclone | grep README.md

rclone lsl ./rclone --include=”/README.md”
rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
rclone lsl scaleway-encrypted-bucket: --include=”/README.md”
rclone copy ./rclone scaleway-encrypted-bucket: --include=”/README.md” --log-level=INFO
rclone lsl scaleway-encrypted-bucket: --include=”/README.md”

rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
rclone touch scaleway:klingtnet-rclone-test-bucket/testfileglacier --s3-storage-class=GLACIER
rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”/RELEASE.md” --log-level=INFO
rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER
rclone copy ./rclone scaleway:klingtnet-rclone-test-bucket --include=”/RELEASE.md” --log-level=INFO
rclone lsl scaleway:klingtnet-rclone-test-bucket --s3-storage-class=GLACIER

rclone lsl ./rclone --s3-storage-class=GLACIER --include=”/README.md”
rclone lsl scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md”
rclone copy ./rclone scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md” --log-level=INFO
rclone lsl scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md”
rclone copy ./rclone scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md” --log-level=INFO
rclone lsl scaleway-encrypted-bucket: --s3-storage-class=GLACIER --include=”/README.md”