Rclone sync s3 update use-server-modtime skips recently modified files

Help and Support
1

What is the problem you are having with rclone?

rclone is skipping files that are recently modified, "Destination is newer than source, skipping"

I am trying to sync files from my time machine storage to encrypted S3 Glacier. Using find, I can see files modified within the past 1 day. But looking at rclone output, it skips many files, giving the message noted above.

-- Find all time machine bundle files modified in the past 1 day
# find /opt/timemachine/macbook.backupbundle/ -mtime -1
...
/opt/timemachine/macbook.backupbundle/bands/2672
...

-- Attempt to sync recently modified time machine files to S3 encrypted destination
# rclone sync --update --use-server-modtime /opt/timemachine/ cryptice:/opt/timemachine/ --transfers=6 --verbose --verbose --verbose
2023/08/15 16:47:26 DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "sync" "--update" "--use-server-modtime" "/opt/timemachine/" "cryptice:/opt/timemachine/" "--transfers=6" "--verbose" "--verbose" "--verbose"]
...
...
2023/08/15 16:47:42 DEBUG : macbook.backupbundle/bands/2672: Destination is newer than source, skipping

How can the destination be newer than the source, when the source was just modified moments ago when time machine performed a backup? In fact, many files were modified:

# find /opt/timemachine/macbook.backupbundle/ -mtime -1 | wc -l
1578

What are you expecting?

I am expecting rclone to sync these 1578 files because the local modified time is more recent than what was previously uploaded to s3.

Run the command 'rclone version' and share the full output of the command.

# rclone version
rclone v1.63.1
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 4.15.0-188-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3
storage_class=DEEP_ARCHIVE

2

welcome to the forum,

--verbose - using it multiple times will not provide more info.
if you want a deeper look, try --dump=headers

can you run those tests using latest stable rclone and post the full output?
if possible, rclone copy --dry-run, just one file and post full output.

try to just use rclone, let's figure out what the timestamps are, see what rclone is seeing
for that one file, what is the output of
rclone lsl -vv on the source file
rclone lsl -vv --use-server-modtime on the dest file

or could run, for that one file, rclone check --use-server-modtime -vv

3

I believe I am using the latest stable rclone? I just ran rclone selftupdate which updated itself to 1.63.1

Perhaps this is the issue that is confusing:

# rclone lsl cryptice:opt/timemachine/macbook.backupbundle/bands/2672
 16777216 2023-08-15 04:58:43.259303500 2672

# ls --full-time -la /opt/timemachine/macbook.backupbundle/bands/2672
-rwxrwxrwx 1 foo foo 16777216 2023-08-15 04:58:43.259303500 +0000 /opt/timemachine/macbook.backupbundle/bands/2672

It appears they are indeed the same file, down to the millisecond. I checked a few others too.

The "destination is newer" is a confusing statement in this regard. A more appropriate statement would be "destination has same mode-time".

Why is this one different with --use-server-modtime?

# rclone lsl --use-server-modtime cryptice:opt/timemachine/macbook.backupbundle/bands/2672
 16777216 2023-08-15 07:09:12.000000000 2672
4

DEBUG : rclone: Version "v1.56.2"

5

Apologies. I updated it at some point then after posting the initial message because I'm using this now

# rclone version
rclone v1.63.1
6

i do not use that flag with s3, could be a subtlety i am missing.

you are comparing the local modtime against the server upload completion time.
so, from rclone perspective, the dest is newer than the source.
the timestamps show that.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.