Files being synced again

Help and Support
1

What is the problem you are having with rclone?

I have an encrypted remote on Dropbox and I decided to move to Google Drive recently so I used rclone to help me move my files. I initially used sync to copy the ALL of my Dropbox files into Google Drive, including unencrypted stuff and then I realized that using sync on the encrypted volumes would be better.

Now I was double checking that I haven't lost any files while moving all my files so I started mounting both remotes as well as a local copy to manually check for random files and file sizes/counts and everything seemed good.

I then added new files to the local encrypted remote and I synced it with the Dropbox encrypted remote and then I tried doing the same with Google Drive but I noticed that it was reuploading old files AGAIN.
I then investigated further and file numbers and sizes were still equivalent.

I then started playing around with the --dry-run and -vv and I found some mismatches in the modification time that were super small. It was only a portion of the remote that was affected; not all files. And we're only talking about encrypted volumes, I haven't checked unencrypted files yet.

Ironically, if I do a check or sync with --dry-run between local and Google Drive or Dropbox and Google Drive; this issue happens but if I do check or sync with --dry-run between local and Dropbox, everything works fine. It seems that the problem only arises once I start syncing into Google Drive.

Again, all those remotes are encrypted and I haven't checked unencrypted remotes yet.

Also for convenience purposes, is there a way to redact any file names and directories next time I need to post more logs? I am dealing with thousands of files here so I can't really do it manually if I were to post the whole logs.

Run the command 'rclone version' and share the full output of the command.

- os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
- os/kernel: 10.0.22621.819 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive, Dropbox and locally (all encrypted)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync ppics-dropbox: ppics-gdrive:

The rclone config contents with secrets removed.

[dropbox]
type = dropbox
token = 

[gdrive]
type = drive
scope = drive
token = 
team_drive = 

[ppics-dropbox]
type = crypt
remote = dropbox:Backup/ppics
password = 
password2 = 

[ppics-gdrive]
type = crypt
remote = gdrive:Backup/ppics
password = 
password2 =

[ppics]
type = crypt
remote = D:Backup/ppics
password = 
password2 =

A log from the command with the -vv flag

Those are some extracts for the same file when running both commands one time with -vv and one time without.

rclone sync ppics: ppics-gdrive: --progress --dry-run --log-file C:\Users\Tarek\Desktop\log.txt
2022/12/04 23:42:22 NOTICE: REDACTED.mp4: Skipped copy as --dry-run is set (size 414.693Mi)

rclone sync ppics: ppics-gdrive: --progress --dry-run -vv --log-file C:\Users\Tarek\Desktop\log.txt
2022/12/04 23:43:31 DEBUG : REDACTED.mp4: Modification times differ by 349.0361ms: 2021-10-10 06:17:49.6509639 +0200 EET, 2021-10-10 04:17:50 +0000 UTC

EDIT

This is running sync with --dry-run. Multiple combinations that don't make sense. I can add more combinations if it's going to help.

PS C:\Users\Tarek> rclone sync ppics-dropbox: ppics-gdrive: --progress --dry-run
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              8217 / 8217, 100%
Elapsed time:        23.9s
2022/12/05 05:35:33 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              8217 / 8217, 100%
Elapsed time:        23.9s

PS C:\Users\Tarek> rclone sync ppics: ppics-gdrive: --progress --dry-run
# REDACTED FILE NAMES
# This is not the full size of the remote.
Transferred:      172.642 GiB / 172.642 GiB, 100%, 9.479 GiB/s, ETA 0s
Checks:              8217 / 8217, 100%
Transferred:         1025 / 1025, 100%
Elapsed time:        21.2s
2022/12/05 05:36:26 NOTICE:
Transferred:      172.642 GiB / 172.642 GiB, 100%, 9.479 GiB/s, ETA 0s
Checks:              8217 / 8217, 100%
Transferred:         1025 / 1025, 100%
Elapsed time:        21.2s

PS C:\Users\Tarek> rclone sync ppics: ppics-dropbox: --progress --dry-run
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              8217 / 8217, 100%
Elapsed time:        18.1s
2022/12/05 05:39:17 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              8217 / 8217, 100%
Elapsed time:        18.1s
2

What it looks like is that the times are slightly out. Try your --dry-run sync with --modify-window 1s and see if that is successfull.

I suspect that the timestamps got truncated at some point in your copying.

You can also do a sync to set the times if you want using the --refresh-times flag, though note dropbox doesn't support setting modtimes without uploading the file again

  --refresh-times                        Refresh the modtime of remote files
3

Yea --refresh-times seems to have fixed it without having to reupload. I am solely using Google Drive now so I don't care about the files in Dropbox. That saved me a lot of bandwidth.

Any idea why the modtimes were changed in the first place? Any by modtime you mean modification time? Doesn't that only change when actually modifying a file?

Thanks again!

4

Different storage systems store the modification time (modtime) at different precisions.

Rclone adjusts to that automatically normally.

I'm not 100% clear on how your data has travelled but Dropbox has a precision of 1 second, the local file system probably 1ns or 100ns depending on os, and Drive 1ms.

I think you did your copy from Drive -> Dropbox but you are doing your check from local -> dropbox. If these are the original files then they will have more precision in their modtimes than they did on dropbox and that is what rclone is complaining about here Modification times differ by 349.0361ms. Drive modtimes are 1ms accuracy but rclone sees that the timestamps are out by > 1ms but < 1s.

5

I copied my data from Dropbox to Drive and then I did a check from local to Drive. I don't know if that makes a difference.
But yea the --refresh-times command fixed it.

6

Yes that will be it..

:slight_smile:

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.