Rclone uploads to OneDrive when no changes have been made

Help and Support
1

What is the problem you are having with rclone?

I am using rclone to sync a large number of photos to OneDrive. I want to run a nightly job to keep my photos synced to OneDrive.

The issue that I am having is that on subsequent syncs, files are being synced when nothing has changed.

Any idea on what flags I should be setting so that I do not keep reuploading files that have not changed?

What is your rclone version (output from rclone version)

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.6

I also tried the tests using the most recent version with the same results under Manjaro
rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.13.6

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 Desktop

Which cloud storage system are you using? (eg Google Drive)

Microsoft OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Here are the options I tried. All of them would sync files that hav not been changed.

2020/12/28 14:55:25 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "sync" "--checksum" "--progress" "-vvv" "--log-file=/opt/rclone/OneDrive_upload_20201
228144905.log" "--exclude" "/.dtrash/**" "/home/kevin/Pictures" "OneDrive:ryzen/Pictures"]

2020/12/28 15:11:30 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "sync" "--checksum" "--progress" "-vvv" "--log-file=/opt/rclone/OneDrive_upload_20201
228145952.log" "--exclude" "/.dtrash/**" "/home/kevin/Pictures" "OneDrive:ryzen/Pictures"]

2020/12/28 15:49:28 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "sync" "--ignore-size" "--checksum" "--progress" "-vvv" "--log-file=/opt/rclone/OneDr
ive_upload_20201228154112.log" "--exclude" "/.dtrash/**" "/home/kevin/Pictures" "OneDrive:ryzen/Pictures"]

2020/12/28 15:55:03 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "sync" "--ignore-size" "--ignore-checksum" "--progress" "-vvv" "--log-file=/opt/rclon
e/OneDrive_upload_20201228155231.log" "--exclude" "/.dtrash/**" "/home/kevin/Pictures" "OneDrive:ryzen/Pictures"]

2020/12/28 15:58:43 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "sync" "--ignore-size" "--ignore-checksum" "--progress" "-vvv" "--log-file=/opt/rclon
e/OneDrive_upload_20201228155742.log" "--exclude" "/.dtrash/**" "/home/kevin/Pictures" "OneDrive:ryzen/Pictures"]

The rclone config contents with secrets removed.

[OneDrive]
type = onedrive
token = {"access_token":"redacted","expiry":"2020-12-24T15:43:16.679410573-07:00"}
drive_id = redacted
drive_type = personal

A log from the command with the -vv flag

I have only included an extract from a single log. The logs I generated are HUGE, and too big for pastebin.

Here is the summary of each command:

Run 0
Full sync to one drive, with same options as run 2.  No Logfile

Run 1
Logfile Location: /opt/rclone/OneDrive_upload_20201228144905.log

Transferred:      172.295M / 172.295 MBytes, 100%, 479.344 kBytes/s, ETA 0s
Errors:                 0
Checks:             43249 / 43249, 100%
Transferred:          160 / 160, 100%
Elapsed time:        6m8s

Run 2
Logfile Location: /opt/rclone/OneDrive_upload_20201228145952.log

Transferred:        1.628G / 1.628 GBytes, 100%, 2.432 MBytes/s, ETA 0s
Errors:                 0
Checks:             86498 / 86498, 100%
Transferred:          735 / 735, 100%
Elapsed time:    11m25.5s

Run 3
Logfile Location: /opt/rclone/OneDrive_upload_20201228154112.log

Transferred:        1.935G / 1.935 GBytes, 100%, 4.093 MBytes/s, ETA 0s
Errors:                 0
Checks:             43249 / 43249, 100%
Transferred:         1206 / 1206, 100%
Elapsed time:        8m4s

Run 4
Logfile Location: /opt/rclone/OneDrive_upload_20201228155231.log

Transferred:     1005.194M / 1005.194 MBytes, 100%, 7.235 MBytes/s, ETA 0s
Errors:                 0
Checks:             43249 / 43249, 100%
Transferred:          322 / 322, 100%
Elapsed time:     2m18.9s

Run 5
Logfile Location: /opt/rclone/OneDrive_upload_20201228155742.log

Transferred:      326.577M / 326.577 MBytes, 100%, 6.898 MBytes/s, ETA 0s
Errors:                 0
Checks:             43249 / 43249, 100%
Transferred:           84 / 84, 100%
Elapsed time:       47.3s

Part of Log:
Pastebin Log - shortened_20201228144905.log

2

You are on an old version rclone, please update it.

Also check out:

3

Thanks, I will check out the link.

As noted in the original post, I did use the most recent version of rclone with the same results.

4

The logs need to be from the current version though as looking through older versions isn't really that helpful to find issues. Please post debugs with the latest in the future.

5

Makes sense. I will provide some updated logs.

I have seen the OneDrive page that you referenced, and I have already tried the suggestions there (even though most are for Sharepoint).

6

When I run this command line:

rclone sync                                                  \
       --ignore-size                                         \
       --ignore-checksum                                     \
       --progress                                            \
       -vvv                                                  \
       --log-file=$LOGFILE                                   \
       --exclude "/.dtrash/**"                               \
       /home/kevin/Pictures/2009/10\ October OneDrive:ryzen/Pictures/2009/10\ October

I see that the transfer is because of modification times:
https://pastebin.com/xmRX5Bqh

OneDrive_upload_20201228181937.log:2020/12/28 18:19:45 DEBUG : Grandpa's 90th Birthday 023.jpg: Modification times differ by 23h42m43.140835608s: 2020-12-27 16:24:44.402164392 -0700 MST, 2020-12-28 23:07:27.543 +0000 UTC
OneDrive_upload_20201228181937.log:2020/12/28 18:19:45 DEBUG : Grandpa's 90th Birthday 023.jpg: SHA-1 = b8f44082d3c8105558bd05d53e699d65225a609c (Local file system at /home/kevin/Pictures/2009/10 October)
OneDrive_upload_20201228181937.log:2020/12/28 18:19:45 DEBUG : Grandpa's 90th Birthday 023.jpg: SHA-1 = 799444213f108404db267d0aded8c079589a7ac1 (One drive root 'ryzen/Pictures/2009/10 October')
OneDrive_upload_20201228181937.log:2020/12/28 18:19:45 DEBUG : Grandpa's 90th Birthday 023.jpg: SHA-1 differ
OneDrive_upload_20201228181937.log:2020/12/28 18:19:46 DEBUG : Grandpa's 90th Birthday 023.jpg: Starting multipart upload
OneDrive_upload_20201228181937.log:2020/12/28 18:19:47 DEBUG : Grandpa's 90th Birthday 023.jpg: Uploading segment 0/3675918 size 3675918
OneDrive_upload_20201228181937.log:2020/12/28 18:19:48 INFO  : Grandpa's 90th Birthday 023.jpg: Copied (replaced existing)

This was run immediately after the first.
https://pastebin.com/TqGay12W

OneDrive_upload_20201228182157.log:2020/12/28 18:22:04 DEBUG : Grandpa's 90th Birthday 023.jpg: Modification times differ by 25h56m24.673835608s: 2020-12-27 16:24:44.402164392 -0700 MST, 2020-12-29 01:21:09.076 +0000 UTC
OneDrive_upload_20201228182157.log:2020/12/28 18:22:04 DEBUG : Grandpa's 90th Birthday 023.jpg: SHA-1 = b8f44082d3c8105558bd05d53e699d65225a609c (Local file system at /home/kevin/Pictures/2009/10 October)
OneDrive_upload_20201228182157.log:2020/12/28 18:22:04 DEBUG : Grandpa's 90th Birthday 023.jpg: SHA-1 = 799444213f108404db267d0aded8c079589a7ac1 (One drive root 'ryzen/Pictures/2009/10 October')
OneDrive_upload_20201228182157.log:2020/12/28 18:22:04 DEBUG : Grandpa's 90th Birthday 023.jpg: SHA-1 differ
OneDrive_upload_20201228182157.log:2020/12/28 18:22:05 DEBUG : Grandpa's 90th Birthday 023.jpg: Starting multipart upload
OneDrive_upload_20201228182157.log:2020/12/28 18:22:06 DEBUG : Grandpa's 90th Birthday 023.jpg: Uploading segment 0/3675918 size 3675918
OneDrive_upload_20201228182157.log:2020/12/28 18:22:07 INFO  : Grandpa's 90th Birthday 023.jpg: Copied (replaced existing)

The second run of the same command shows that the same file has different hashes for the local and remote files, and modification times after being updated. From what I understand, the parameters I used above should skip the hash check. That leaves the modification times. They do not seem to get updated correctly in OneDrive. The modification time was 23 hours different in one case, and almost 26 hours in the other case.

Any thoughts on how/why this happens?

Thanks.

7

This looks like onedrive messing with your images.

What seems to be happening is that onedrive is altering the image (changing the EXIF data maybe) which changes both the modification time, the checksum and the size.

If onedrive changes all those things, then all we've got to go on really is existence of the file which is a pretty poor syncing method. You'll have to add --size-only to your existing flags.

Does onedrive do this to all photos? I've heard complaints about this before. Is there any chance you could email me nick@craig-wood.com a photo which onedrive changes on upload? I can't find any examples of onedrive doing this in my own photos.

8

I tried the --size-only flag, and a bunch of files were synced. I then reran using the original command line, it was fine. No additional files were sent the first time, but the second time, some files were sent.

As an additional test, I ran
rclone sync --progress --log-file=$LOGFILE --exclude "/.dtrash/**" /home/kevin/Pictures/2009/10\ October OneDrive:temp
to sync to a new, empty directory. This seems to work properly. Each additional sync that I tried resulted in no files being sent (since nothing changed, that is what I expected).

When I do the same command, but to a OneDrive folder that already had files in it, files are transferred each time that I execute the command.

This makes me think that the issue might lie with how I started to sync with one drive. The folder was originally part of my Pictures folder (at the root level). I copied the folder to a new location (ryzen/Pictures), and then tried to sync from a new server to OneDrive.

I have read that OneDrive does something special with the Pictures folder so that it display the images. This does not seem to happen automatically when images are stored elsewhere. Maybe that has something to do with the issues?

I am going to try to sync all of my files to a clean OneDrive folder, and see if it works on a larger scale.

Thanks for the help.

9

Not sure exactly what the root cause was, but a sync to a clean directory on OneDrive, and all is working as expected. I have now synced several times using this command:

 rclone sync                                                 \
       --progress                                            \
       -v                                                    \
       --log-file=$LOGFILE                                   \
       --exclude "/.dtrash/**"                               \
       /home/kevin/Pictures OneDrive:ryzen/Pictures

No more messing about with --size-only, or any other options to make it work.

Let me know if want to troubleshoot this any further.

10

That would make sense with the whole onedrive changning the images thread.

Great.

If you have the time, if you could verify that writing to Pictures does indeed change the images then we can put a bit in the docs warning about that.

11

I have done a little more testing, and here is what I have found.

Case 1

  • Folder on OneDrive is Pictures/2019
  • Folder on server is Pictures/2019

Syncing appears to work as expected. After the initial sync, there were no more syncs unless I actually changed something on the server.

Case 2

  • Folder on OneDrive is Pictures/ (2019 does not exist in OneDrive)
  • Folder on server is Pictures/2019

Syncing appears to work as expected. After the initial sync, there were no more syncs unless I actually changed something on the server.

Case 3 -This is the one that replicates the original issue in this post

  • Copy OneDrive folder from Pictures/2019 to NewDir/2019
  • Folder on server is NewDir/2019
  • Before the first rclone sync, the contents of the two directories are the same. The contents on the server were downloaded from OneDrive manually

This time, the first sync worked as expected. However, the second and subsequent syncs still send files from the server to OneDrive. The files have modification dates that are weeks or months different that what is on the local server, and the server modification dates never seem to get updated to what they are on my server.

I hope that this helps someone else in the future if they come across this behavior.

12

Thanks for writing that up @bkn

Any suggestions for things to add to the rclone docs?

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.