Sync to SharePoint site giving corruption & IO errors

What is the problem you are having with rclone?

When trying to sync files from the local Windows Server 2019 VM (running in vCenter) to a SharePoint site, I receive errors that the files are corrupted due to different sizes and there were IO errors. Checking on the web GUI of SharePoint and downloading the files, they seem ok, but I haven't done a full check. The file sizes are different on a few files.

The error happens on both the C: drive and the E: drive, the latter of which is an iSCSI target to a QNAP NAS which shows in the VM as a mounted drive. The datastore for the C: is also over iSCSI, which is managed by the vCenter cluster.

Logs of the error below.

Run the command 'rclone version' and share the full output of the command.

Yes, using 1.60.1. I just downloaded and set this up today so as far as I can tell it's the latest one.

Which cloud storage system are you using? (eg Google Drive)

MS SharePoint/OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

PS C:\rclone> .\rclone.exe sync C:\'Test Folder' MarCom-SP:Test -v

The rclone config contents with secrets removed.

PS C:\rclone> .\rclone.exe config show
[MarCom-SP]
type = onedrive
token = {"access_token":"*****"}
drive_id = *****
drive_type = documentLibrary

A log from the command with the -vv flag

The log with -vv was too long, so I put the -v "first pass" in here, and included a pastebin link to the same command ran after with the -vv flag.

PS C:\rclone> .\rclone.exe sync C:\'Test Folder' MarCom-SP:Test -v
2022/11/17 12:50:46 INFO  : rclone/git-log.txt: Copied (new)
2022/11/17 12:50:47 INFO  : rclone/rclone.1: Copied (new)
2022/11/17 12:50:48 INFO  : rclone/README.txt: Copied (new)
2022/11/17 12:50:48 ERROR : rclone/README.html: corrupted on transfer: sizes differ 2069115 vs 2069333
2022/11/17 12:50:48 INFO  : rclone/README.html: Removing failed copy
2022/11/17 12:50:51 INFO  : rclone/rclone.exe: Copied (new)
2022/11/17 12:50:51 ERROR : OneDrive root 'Test': not deleting files as there were IO errors
2022/11/17 12:50:51 ERROR : OneDrive root 'Test': not deleting directories as there were IO errors
2022/11/17 12:50:51 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 2069115 vs 2069333
2022/11/17 12:50:55 ERROR : rclone/README.html: corrupted on transfer: sizes differ 2069115 vs 2069333
2022/11/17 12:50:55 INFO  : rclone/README.html: Removing failed copy
2022/11/17 12:50:55 ERROR : OneDrive root 'Test': not deleting files as there were IO errors
2022/11/17 12:50:55 ERROR : OneDrive root 'Test': not deleting directories as there were IO errors
2022/11/17 12:50:55 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 2069115 vs 2069333
2022/11/17 12:50:59 ERROR : rclone/README.html: corrupted on transfer: sizes differ 2069115 vs 2069333
2022/11/17 12:50:59 INFO  : rclone/README.html: Removing failed copy
2022/11/17 12:50:59 ERROR : OneDrive root 'Test': not deleting files as there were IO errors
2022/11/17 12:50:59 ERROR : OneDrive root 'Test': not deleting directories as there were IO errors
2022/11/17 12:50:59 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 2069115 vs 2069333
2022/11/17 12:50:59 INFO  :
Transferred:       54.852 MiB / 54.852 MiB, 100%, 3.916 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:                 8 / 8, 100%
Transferred:            4 / 4, 100%
Elapsed time:        16.0s

2022/11/17 12:50:59 Failed to sync: corrupted on transfer: sizes differ 2069115 vs 2069333

Both -v and -vv logs here

Documented here:

Ah got it, thank you!

Is there an option to use the remote as the "source" and pull down / delete files on the local storage so that it matches the remote directories?

Sure. Just flip the order.

rclone copy remote: X:/SomeDir

I usually test with --dry-run to ensure it's going to do what I want if I'm not sure.

Ok great that works.

And just to make sure I understand correctly, if I used the sync command and flipped the order like you were saying, that would then delete the local files to match the remote?

Yes, sync is destructive as it makes destination look like source.

Make sure to use --dry-run before you do anything so you are positive it's doing what you want before anything gets deleted.

1 Like

Yes, I'll definitely use --dry-run first. I don't think it will be doing much deleting (and I'll most likely use copy more often) but just want to have the option if I need it.