SharePoint Migration Assistance

Help and Support
1

Good afternoon,

I am attempting to figure out why my sync commands are not actually copying all information over to my SharePoint Tenant.

Background

  • Migrating 2.5 TB from onpremise storage to SharePoint Online tenant
  • Went Live with SharePoint Tenant last week
  • Users have been using SharePoint tenant and noticing that some files are up to date and others are not

Goal

  • Get on premise storage to SharePoint with most up to date files, while keeping the weeks worth of changes that have now taken place

Setup

    • All commands are using rclonev1.34
    • SharePoint Site collection using O365
    • All connections for rclone are using WebDAV connection with my credentials for logging in
    • I am a domain admin locally
    • I am the O365 admin with an M365 E5 license
    • Each local folder was replicated to its own folder under the same site collection
    • rclone command string
      - rclone sync --filter “- /Forms/**” “F:\BACTALK” projects: --checksum --ignore-size --user-agent “ISV| rclone.org|rclone/v1.43” --progress --low-level-retries 200 --retries 1 --fast-list -q --log-file “F:\BACTALK\rclone-v1.44-windows-amd64\rclone-v1.44-windows-amd64.csv”
  • The only thing that changes is of course the source and destination to the specific folders

Thoughts

  • I think my issue is related to the checksum switch in the command line and my files are ignoring the changes with either the checksum and/or the ignore-size

Concerns

  • removing those in the command line will cause the files that have changed within the last week to be overwritten if there is a file with the same name that exists in the directory on SharePoint
  • Running “rclone sync” without switches mentioned above in thoughts will delete new files that were created on destination site.

I think I have explained enough to elicit assistance, but if not please ask. I am new to using this, and I am still learning how to write the switches.

  • List item
2

v1.34 is a really old version - 1.46 is the most recent - suggest you update to that first then try again. See the latest release. I think you are actually using 1.43 - there have been quite a few fixes to webdav since then.

WebDAV doesn’t implement checksums in the general case so --checksum won’t be doing anything for you. I think --ignore-size may be causing the problem.

Yes that is what sync does - it aims to make the source the same as the destination. You don’t want users changing the destination of a sync as rclone will just overwrite the changes on the next sync.

3

ok thank you for the response. is there a way to do a true delta sync? where only the destination changes if the files are “newer” than the ones at the source? I really just need to do one sync, getting all the data to the cloud and then everyone will be working from the cloud.

4

rclone does a delta sync, but the deltas are whole files rather than parts of them. So if you change one byte in one file between runs, rclone will upload that whole file again, but none of the others.

5

ok is there a way to designate which side wins? meaning if the byte difference is on the destination side, the file will be ignored?

6

rclone sync source: destination: always makes the source identical to the destination by changing the destination only.

rclone copy will just copy stuff to the destination which isn’t there already.