Onedrive error on changed file

Suspected Bug
1

Hi,

I’m using rclone to sync from a Windows device to a Sharepoint 365 share using the OneDrive type (using the latest beta to get this to work).

It looks like the initial upload has worked, but when i’m running a sync to push up any changes, I get errors where the file has changed locally and an upload is required (see below for error).

I think i’m only seeing this for docx and xlsx files

I had to ignore checksums and sizes of files because of an error where docx files are always modified in some way by Sharepoint (adding metadata?), and I wonder if this is contributing to the problem.

2018/09/18 19:53:01 DEBUG : rclone: Version “v1.43-085-ge022ffce-beta” starting with parameters [“C:\TempPath\rclone-beta” “sync” “c:\Shares\Company” “sharepoint:Company” “–ignore-checksum” “–ignore-size” “–log-file” “c:\temppath\sync1.log” “–retries” “1” “-v” “-v”]
2018/09/18 19:53:01 DEBUG : Using config file from “C:\Users\admin\.config\rclone\rclone.conf”
2018/09/18 19:53:07 DEBUG : General Info/current tenders.docx: Modification times differ by 178h17m49.3394733s: 2018-09-11 09:28:57.6605267 +1000 AEST, 2018-09-18 09:46:47 +0000 UTC
2018/09/18 19:53:07 DEBUG : General Info/current tenders.docx: QuickXorHash = 34c1ed01e32d7837e08a8909463bbb5dbc2c97d7 (Local file system at \?\c:\Shares\Company)
2018/09/18 19:53:07 DEBUG : General Info/current tenders.docx: QuickXorHash = 7cbec8b41f60dc8c2f4e8f3ec36e1aacc5bea01a (One drive root ‘Company’)
2018/09/18 19:53:07 DEBUG : General Info/current tenders.docx: QuickXorHash differ
2018/09/18 19:53:07 DEBUG : General Info/current tenders.docx: Starting multipart upload
2018/09/18 19:53:08 DEBUG : General Info/current tenders.docx: Uploading segment 0/16406 size 16406
2018/09/18 19:53:09 DEBUG : General Info/current tenders.docx: Cancelling multipart upload: itemNotFound: The resource could not be found.
2018/09/18 19:53:10 NOTICE: General Info/current tenders.docx: Failed to cancel multipart upload: itemNotFound: The resource could not be found.
2018/09/18 19:53:10 ERROR : General Info/current tenders.docx: Failed to copy: itemNotFound: The resource could not be found.

thanks

James

2

It sounds like that is what sharepoint is doing - modifying the docx files in some way.

@Cnly any ideas what is going on?

3

It pretty much seems like a known problem of SharePoint which can be found mixed in this GitHub issue: https://github.com/ncw/rclone/issues/1770

Maybe we should open a separate issue for that :confused: Maybe also add it to the docs.

4

The way we solve this issue in google drive is to set the length of the docs to -1. This causes their sizes to be ignored. We’d need to set the quickxorhash to “” also (google docs don’t have hashes).

Probably both! I’d like to find a work-around as it sounds unusable for the moment!

5

The way we solve this issue in google drive is to set the length of the docs to -1.

Sounds good! This should be a possible workaround. Our current code knows what kind of OneDrive backend it’s on.

And I have opened an issue here: https://github.com/ncw/rclone/issues/2599

6

The original files upload just fine, once I turn off checksum and size checks. It’s when the file has changed locally that the upload fails to replace the onedrive/sharepoint file.

Do you think this is causing the itemNotFound error, and would the proposed fix cause the problem?

Is there any other logging I capture that would confirm this?

1 Like
7

Thanks for your reminder! Sorry that I’ve neglected that specific error type. It does seem a little strange now, and I’ll do some test to see what the problem is.

If you would like to help by providing more info, you can add --dump requests,responses to the command line. This will not expose your credentials, but arguments like file paths and organization names may be included in the output. So check before posting!

EDIT: It would also be helpful if you could provide a list of file types that you see are causing this error. Thanks!

8

The main file type that stands out is docx and xlsx.

I have created a test folder with a single docx file and done a sync and created 3 logfiles - initial sync log, sync with no changes, and a sync with the changed file. I can’t seem to paste it below because new users can only have 2 links and the logfile obviously contains more than 2!

How much overhead would there be if you (optionally) stamped the original file checksum and size into the uploaded file as metadata (for office files)? You could use these to compare against what the file was originally when it was uploaded.

James

9

rclone with a changed file

2018/09/22 10:52:18 DEBUG : rclone: Version "v1.43-085-ge022ffce-beta" starting with parameters ["C:\\TempPath\\rclone-beta" "sync" "c:\\TempPath\\TestData" "sharepoint:TestData" "--ignore-checksum" "--ignore-size" "--log-file" "c:\\temppath\\test.log" "--log-level" "DEBUG" "--retries" "1" "--dump" "requests,responses"]
2018/09/22 10:52:18 DEBUG : Using config file from "C:\\Users\\admin\\.config\\rclone\\rclone.conf"
2018/09/22 10:52:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:18 DEBUG : HTTP REQUEST (req 0xc0002d9500)
2018/09/22 10:52:18 DEBUG : GET /v1.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/root HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.43-085-ge022ffce-beta
Authorization: XXXX
Accept-Encoding: gzip

2018/09/22 10:52:18 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:20 DEBUG : HTTP RESPONSE (req 0xc0002d9500)
2018/09/22 10:52:20 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Client-Request-Id: 7f8e3939-0b91-4567-8b40-1b990beff4ec
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Sat, 22 Sep 2018 00:52:19 GMT
Duration: 773.7048
Odata-Version: 4.0
Request-Id: 7f8e3939-0b91-4567-8b40-1b990beff4ec
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"East Asia","Slice":"SliceC","Ring":"5","ScaleUnit":"002","Host":"AGSFE_IN_8","ADSiteName":"EAS"}}

291
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#drives('b%21KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u')/root/$entity","createdDateTime":"2018-06-17T15:14:50Z","id":"01KTFZYU56Y2GOVW7725BZO354PWSELRRZ","lastModifiedDateTime":"2018-09-22T00:36:57Z","name":"root","webUrl":"https://tenant.sharepoint.com/Shared%20Documents","size":268395870570,"parentReference":{"driveId":"b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u","driveType":"documentLibrary"},"fileSystemInfo":{"createdDateTime":"2018-06-17T15:14:50Z","lastModifiedDateTime":"2018-09-22T00:36:57Z"},"folder":{"childCount":2},"root":{}}
0

2018/09/22 10:52:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:20 DEBUG : HTTP REQUEST (req 0xc0000be500)
2018/09/22 10:52:20 DEBUG : GET /v1.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/root:/TestData HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.43-085-ge022ffce-beta
Authorization: XXXX
Accept-Encoding: gzip

2018/09/22 10:52:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:21 DEBUG : HTTP RESPONSE (req 0xc0000be500)
2018/09/22 10:52:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Client-Request-Id: 07eabee2-af79-4157-9ecb-a11426e4fc27
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Sat, 22 Sep 2018 00:52:21 GMT
Duration: 845.4823
Odata-Version: 4.0
Request-Id: 07eabee2-af79-4157-9ecb-a11426e4fc27
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"East Asia","Slice":"SliceC","Ring":"5","ScaleUnit":"002","Host":"AGSFE_IN_8","ADSiteName":"EAS"}}

532
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#drives('b%21KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u')/root/$entity","createdDateTime":"2018-09-22T00:36:52Z","eTag":"\"{9E6D72C4-78A3-4C6C-8D1A-320CA5921062},1\"","id":"01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC","lastModifiedDateTime":"2018-09-22T00:36:52Z","name":"TestData","webUrl":"https://tenant.sharepoint.com/Shared%20Documents/TestData","cTag":"\"c:{9E6D72C4-78A3-4C6C-8D1A-320CA5921062},0\"","size":22510,"createdBy":{"application":{"id":"b15665d9-eda6-4092-8539-0eec376afd59","displayName":"rclone"},"user":{"email":"user@company","id":"cd00a58e-10d9-4874-8749-fd3cbf215e8c","displayName":"Mick Hassett"}},"lastModifiedBy":{"application":{"id":"b15665d9-eda6-4092-8539-0eec376afd59","displayName":"rclone"},"user":{"email":"user@company","id":"cd00a58e-10d9-4874-8749-fd3cbf215e8c","displayName":"Mick Hassett"}},"parentReference":{"driveId":"b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u","driveType":"documentLibrary","id":"01KTFZYU56Y2GOVW7725BZO354PWSELRRZ","path":"/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/root:"},"fileSystemInfo":{"createdDateTime":"2018-09-22T00:36:52Z","lastModifiedDateTime":"2018-09-22T00:36:52Z"},"folder":{"childCount":1}}
0

2018/09/22 10:52:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:21 DEBUG : HTTP REQUEST (req 0xc0002d8d00)
2018/09/22 10:52:21 DEBUG : GET /v1.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/items/01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC/children?$top=1000 HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.43-085-ge022ffce-beta
Authorization: XXXX
Accept-Encoding: gzip

2018/09/22 10:52:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:21 DEBUG : HTTP RESPONSE (req 0xc0002d8d00)
2018/09/22 10:52:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Client-Request-Id: 5e8260bf-7f69-4eb4-94bc-66c9810a3401
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Sat, 22 Sep 2018 00:52:21 GMT
Duration: 451.1345
Odata-Version: 4.0
Request-Id: 5e8260bf-7f69-4eb4-94bc-66c9810a3401
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"East Asia","Slice":"SliceC","Ring":"5","ScaleUnit":"002","Host":"AGSFE_IN_8","ADSiteName":"EAS"}}

b2b
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#drives('b%21KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u')/items('01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC')/children","value":[{"@microsoft.graph.downloadUrl":"https://tenant.sharepoint.com/_layouts/15/download.aspx?UniqueId=db74cd07-53e4-4003-a338-89ddc9220c09&Translate=false&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvMm1oY29uc3VsdGluZzEuc2hhcmVwb2ludC5jb21AZTAyMzA4Y2EtZjlmOC00MGQ4LWJlMTYtNGRmYWUyMmVlMWM2IiwiaXNzIjoiMDAwMDAwMDMtMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwIiwibmJmIjoiMTUzNzU3NzU0MSIsImV4cCI6IjE1Mzc1ODExNDEiLCJlbmRwb2ludHVybCI6IjdpNXZoNHFRdW9KQkl2QmZpYXo5U2F5NFlYRUZkcDRCOFM0aEtGeEhMcGs9IiwiZW5kcG9pbnR1cmxMZW5ndGgiOiIxMjUiLCJpc2xvb3BiYWNrIjoiVHJ1ZSIsImNpZCI6Ik5XVTRNall3WW1ZdE4yWTJPUzAwWldJMExUazBZbU10Tmpaak9UZ3hNR0V6TkRBeCIsInZlciI6Imhhc2hlZHByb29mdG9rZW4iLCJzaXRlaWQiOiJaRGt3TVRNM01tSXRZamd3TUMwME1EZzRMVGhsT0dJdE1EUTRPVEZoWVdRMk1USXciLCJhcHBfZGlzcGxheW5hbWUiOiJyY2xvbmUiLCJhcHBpZCI6ImIxNTY2NWQ5LWVkYTYtNDA5Mi04NTM5LTBlZWMzNzZhZmQ1OSIsInRpZCI6ImUwMjMwOGNhLWY5ZjgtNDBkOC1iZTE2LTRkZmFlMjJlZTFjNiIsInVwbiI6Im1pY2tAMm1oY29uc3VsdGluZy5jb20uYXUiLCJwdWlkIjoiMTAwMzdGRkVBRDc0NTU5RCIsInNjcCI6Im15ZmlsZXMucmVhZCBhbGxmaWxlcy5yZWFkIG15ZmlsZXMud3JpdGUgYWxsZmlsZXMud3JpdGUiLCJ0dCI6IjIiLCJ1c2VQZXJzaXN0ZW50Q29va2llIjpudWxsfQ.aVJSZm9OQTdMUjlpdGxzMGh2WG9DVkM0b1NwOEpBbEdDVFVIUm4zcFJ6OD0&ApiVersion=2.0","createdDateTime":"2018-09-20T02:02:40Z","eTag":"\"{DB74CD07-53E4-4003-A338-89DDC9220C09},6\"","id":"01KTFZYUYHZV2NXZCTANAKGOEJ3XESEDAJ","lastModifiedDateTime":"2018-09-20T02:02:40Z","name":"test.docx","webUrl":"https://tenant.sharepoint.com/_layouts/15/Doc.aspx?sourcedoc=%7BDB74CD07-53E4-4003-A338-89DDC9220C09%7D&file=test.docx&action=default&mobileredirect=true","cTag":"\"c:{DB74CD07-53E4-4003-A338-89DDC9220C09},7\"","size":22510,"createdBy":{"application":{"id":"b15665d9-eda6-4092-8539-0eec376afd59","displayName":"rclone"},"user":{"email":"user@company","id":"cd00a58e-10d9-4874-8749-fd3cbf215e8c","displayName":"Mick Hassett"}},"lastModifiedBy":{"application":{"id":"b15665d9-eda6-4092-8539-0eec376afd59","displayName":"rclone"},"user":{"email":"user@company","id":"cd00a58e-10d9-4874-8749-fd3cbf215e8c","displayName":"Mick Hassett"}},"parentReference":{"driveId":"b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u","driveType":"documentLibrary","id":"01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC","path":"/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/root:/TestData"},"file":{"mimeType":"application/vnd.openxmlformats-officedocument.wordprocessingml.document","hashes":{"quickXorHash":"Dkux1GYe6J8vwj74/TQ83mtShHs="}},"fileSystemInfo":{"createdDateTime":"2018-09-20T02:02:40Z","lastModifiedDateTime":"2018-09-20T02:02:40Z"}}]}
0

2018/09/22 10:52:21 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:21 INFO  : One drive root 'TestData': Waiting for checks to finish
2018/09/22 10:52:21 DEBUG : test.docx: Modification times differ by 6596h45m42.4295781s: 2017-12-19 16:16:57.5704219 +1100 AEDT, 2018-09-20 02:02:40 +0000 UTC
2018/09/22 10:52:21 DEBUG : test.docx: QuickXorHash = 43aac543f237ab363820dcfd81483c24714e0da7 (Local file system at \\?\c:\TempPath\TestData)
2018/09/22 10:52:21 DEBUG : test.docx: QuickXorHash = 0e4bb1d4661ee89f2fc23ef8fd343cde6b52847b (One drive root 'TestData')
2018/09/22 10:52:21 DEBUG : test.docx: QuickXorHash differ
2018/09/22 10:52:21 INFO  : One drive root 'TestData': Waiting for transfers to finish
2018/09/22 10:52:21 DEBUG : test.docx: Starting multipart upload
2018/09/22 10:52:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:21 DEBUG : HTTP REQUEST (req 0xc0002d9600)
2018/09/22 10:52:21 DEBUG : POST /v1.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/items/01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC:/test.docx:/createUploadSession HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.43-085-ge022ffce-beta
Content-Length: 116
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"item":{"fileSystemInfo":{"createdDateTime":"2017-12-19T05:16:57Z","lastModifiedDateTime":"2017-12-19T05:16:57Z"}}}
2018/09/22 10:52:21 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:23 DEBUG : HTTP RESPONSE (req 0xc0002d9600)
2018/09/22 10:52:23 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Client-Request-Id: 22a9771e-8d3a-4858-98f2-501f0962ab01
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Sat, 22 Sep 2018 00:52:23 GMT
Duration: 1136.3882
Location: https://graph.microsoft.com
Odata-Version: 4.0
Request-Id: 22a9771e-8d3a-4858-98f2-501f0962ab01
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"East Asia","Slice":"SliceC","Ring":"5","ScaleUnit":"002","Host":"AGSFE_IN_8","ADSiteName":"EAS"}}

5f6
{"@odata.context":"https://graph.microsoft.com/v1.0/$metadata#microsoft.graph.uploadSession","expirationDateTime":"2018-09-22T01:07:22.672Z","nextExpectedRanges":["0-"],"uploadUrl":"https://tenant.sharepoint.com/_api/v2.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/items/01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC/uploadSession?guid='4f048c6c-7ea9-4b6c-bf16-9ab207ada04e'&path='test.docx'&overwrite=True&rename=False&dc=0&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvMm1oY29uc3VsdGluZzEuc2hhcmVwb2ludC5jb21AZTAyMzA4Y2EtZjlmOC00MGQ4LWJlMTYtNGRmYWUyMmVlMWM2IiwiaXNzIjoiMDAwMDAwMDMtMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwIiwibmJmIjoiMTUzNzU3NzU0MiIsImV4cCI6IjE1Mzc2NjM5NDIiLCJlbmRwb2ludHVybCI6IlRLOHpwc2RVQW5XNWZQbXBZRGZRVnl1MDlXaEROT3gyV084MTY2ZFFOTG89IiwiZW5kcG9pbnR1cmxMZW5ndGgiOiIyNzAiLCJpc2xvb3BiYWNrIjoiVHJ1ZSIsImNpZCI6Ik1qSmhPVGMzTVdVdE9HUXpZUzAwT0RVNExUazRaakl0TlRBeFpqQTVOakpoWWpBeCIsInZlciI6Imhhc2hlZHByb29mdG9rZW4iLCJzaXRlaWQiOiJaRGt3TVRNM01tSXRZamd3TUMwME1EZzRMVGhsT0dJdE1EUTRPVEZoWVdRMk1USXciLCJhcHBfZGlzcGxheW5hbWUiOiJyY2xvbmUiLCJhcHBpZCI6ImIxNTY2NWQ5LWVkYTYtNDA5Mi04NTM5LTBlZWMzNzZhZmQ1OSIsInRpZCI6ImUwMjMwOGNhLWY5ZjgtNDBkOC1iZTE2LTRkZmFlMjJlZTFjNiIsInVwbiI6Im1pY2tAMm1oY29uc3VsdGluZy5jb20uYXUiLCJwdWlkIjoiMTAwMzdGRkVBRDc0NTU5RCIsInNjcCI6Im15ZmlsZXMucmVhZCBhbGxmaWxlcy5yZWFkIG15ZmlsZXMud3JpdGUgYWxsZmlsZXMud3JpdGUiLCJ0dCI6IjIiLCJ1c2VQZXJzaXN0ZW50Q29va2llIjpudWxsfQ.MnlQR3M5cjFJWk1Cbk41dW5paVRpSkEycXRoRE8zYWJRRFVOVWhyNFJ6VT0"}
0

2018/09/22 10:52:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:23 DEBUG : test.docx: Uploading segment 0/48904 size 48904
2018/09/22 10:52:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:23 DEBUG : HTTP REQUEST (req 0xc0000be800)
2018/09/22 10:52:23 DEBUG : PUT /_api/v2.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/items/01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC/uploadSession?guid='4f048c6c-7ea9-4b6c-bf16-9ab207ada04e'&path='test.docx'&overwrite=True&rename=False&dc=0&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvMm1oY29uc3VsdGluZzEuc2hhcmVwb2ludC5jb21AZTAyMzA4Y2EtZjlmOC00MGQ4LWJlMTYtNGRmYWUyMmVlMWM2IiwiaXNzIjoiMDAwMDAwMDMtMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwIiwibmJmIjoiMTUzNzU3NzU0MiIsImV4cCI6IjE1Mzc2NjM5NDIiLCJlbmRwb2ludHVybCI6IlRLOHpwc2RVQW5XNWZQbXBZRGZRVnl1MDlXaEROT3gyV084MTY2ZFFOTG89IiwiZW5kcG9pbnR1cmxMZW5ndGgiOiIyNzAiLCJpc2xvb3BiYWNrIjoiVHJ1ZSIsImNpZCI6Ik1qSmhPVGMzTVdVdE9HUXpZUzAwT0RVNExUazRaakl0TlRBeFpqQTVOakpoWWpBeCIsInZlciI6Imhhc2hlZHByb29mdG9rZW4iLCJzaXRlaWQiOiJaRGt3TVRNM01tSXRZamd3TUMwME1EZzRMVGhsT0dJdE1EUTRPVEZoWVdRMk1USXciLCJhcHBfZGlzcGxheW5hbWUiOiJyY2xvbmUiLCJhcHBpZCI6ImIxNTY2NWQ5LWVkYTYtNDA5Mi04NTM5LTBlZWMzNzZhZmQ1OSIsInRpZCI6ImUwMjMwOGNhLWY5ZjgtNDBkOC1iZTE2LTRkZmFlMjJlZTFjNiIsInVwbiI6Im1pY2tAMm1oY29uc3VsdGluZy5jb20uYXUiLCJwdWlkIjoiMTAwMzdGRkVBRDc0NTU5RCIsInNjcCI6Im15ZmlsZXMucmVhZCBhbGxmaWxlcy5yZWFkIG15ZmlsZXMud3JpdGUgYWxsZmlsZXMud3JpdGUiLCJ0dCI6IjIiLCJ1c2VQZXJzaXN0ZW50Q29va2llIjpudWxsfQ.MnlQR3M5cjFJWk1Cbk41dW5paVRpSkEycXRoRE8zYWJRRFVOVWhyNFJ6VT0 HTTP/1.1
Host: tenant.sharepoint.com
User-Agent: rclone/v1.43-085-ge022ffce-beta
Content-Length: 48904
Authorization: XXXX
Content-Range: bytes 0-48903/48904
Accept-Encoding: gzip

__CONTENT REDACTED__

2018/09/22 10:52:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:24 DEBUG : HTTP RESPONSE (req 0xc0000be800)
2018/09/22 10:52:24 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Sat, 22 Sep 2018 00:52:23 GMT
Expires: -1
Microsoftsharepointteamservices: 16.0.0.8105
Ms-Cv: npEPBBCAAAASJq9C8RoNsQ.0
Odata-Version: 4.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Pragma: no-cache
Request-Id: 040f919e-8010-0000-1226-af42f11a0db1
Spclientservicerequestduration: 310
Sprequestduration: 418
Sprequestguid: 040f919e-8010-0000-1226-af42f11a0db1
Strict-Transport-Security: max-age=31536000
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: CC67143C525644CEA92D547610B10278 Ref B: MEL01EDGE0416 Ref C: 2018-09-22T00:52:24Z
X-Powered-By: ASP.NET
X-Sharepointhealthscore: 0
X-Sp-Serverstate: ReadOnly=0

4e
{"error":{"code":"itemNotFound","message":"The resource could not be found."}}
0

2018/09/22 10:52:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:24 DEBUG : test.docx: Cancelling multipart upload: itemNotFound: The resource could not be found.
2018/09/22 10:52:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:24 DEBUG : HTTP REQUEST (req 0xc00017c200)
2018/09/22 10:52:24 DEBUG : DELETE /_api/v2.0/drives/b!KzcB2QC4iECOiwSJGq1hIGSx1CGqaV9BiVhP7yCQSQffeys2-A4yT4170PSzzM-u/items/01KTFZYU6EOJWZ5I3YNRGI2GRSBSSZEEDC/uploadSession?guid='4f048c6c-7ea9-4b6c-bf16-9ab207ada04e'&path='test.docx'&overwrite=True&rename=False&dc=0&tempauth=eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJhdWQiOiIwMDAwMDAwMy0wMDAwLTBmZjEtY2UwMC0wMDAwMDAwMDAwMDAvMm1oY29uc3VsdGluZzEuc2hhcmVwb2ludC5jb21AZTAyMzA4Y2EtZjlmOC00MGQ4LWJlMTYtNGRmYWUyMmVlMWM2IiwiaXNzIjoiMDAwMDAwMDMtMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwIiwibmJmIjoiMTUzNzU3NzU0MiIsImV4cCI6IjE1Mzc2NjM5NDIiLCJlbmRwb2ludHVybCI6IlRLOHpwc2RVQW5XNWZQbXBZRGZRVnl1MDlXaEROT3gyV084MTY2ZFFOTG89IiwiZW5kcG9pbnR1cmxMZW5ndGgiOiIyNzAiLCJpc2xvb3BiYWNrIjoiVHJ1ZSIsImNpZCI6Ik1qSmhPVGMzTVdVdE9HUXpZUzAwT0RVNExUazRaakl0TlRBeFpqQTVOakpoWWpBeCIsInZlciI6Imhhc2hlZHByb29mdG9rZW4iLCJzaXRlaWQiOiJaRGt3TVRNM01tSXRZamd3TUMwME1EZzRMVGhsT0dJdE1EUTRPVEZoWVdRMk1USXciLCJhcHBfZGlzcGxheW5hbWUiOiJyY2xvbmUiLCJhcHBpZCI6ImIxNTY2NWQ5LWVkYTYtNDA5Mi04NTM5LTBlZWMzNzZhZmQ1OSIsInRpZCI6ImUwMjMwOGNhLWY5ZjgtNDBkOC1iZTE2LTRkZmFlMjJlZTFjNiIsInVwbiI6Im1pY2tAMm1oY29uc3VsdGluZy5jb20uYXUiLCJwdWlkIjoiMTAwMzdGRkVBRDc0NTU5RCIsInNjcCI6Im15ZmlsZXMucmVhZCBhbGxmaWxlcy5yZWFkIG15ZmlsZXMud3JpdGUgYWxsZmlsZXMud3JpdGUiLCJ0dCI6IjIiLCJ1c2VQZXJzaXN0ZW50Q29va2llIjpudWxsfQ.MnlQR3M5cjFJWk1Cbk41dW5paVRpSkEycXRoRE8zYWJRRFVOVWhyNFJ6VT0 HTTP/1.1
Host: tenant.sharepoint.com
User-Agent: rclone/v1.43-085-ge022ffce-beta
Authorization: XXXX
Accept-Encoding: gzip

2018/09/22 10:52:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/09/22 10:52:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:24 DEBUG : HTTP RESPONSE (req 0xc00017c200)
2018/09/22 10:52:24 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Cache-Control: no-cache, no-store
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Date: Sat, 22 Sep 2018 00:52:24 GMT
Expires: -1
Microsoftsharepointteamservices: 16.0.0.8105
Ms-Cv: npEPBC5AAAAUWA5IUtrF3w.0
Odata-Version: 4.0
P3p: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
Pragma: no-cache
Request-Id: 040f919e-402e-0000-1458-0e4852dac5df
Spclientservicerequestduration: 318
Sprequestduration: 447
Sprequestguid: 040f919e-402e-0000-1458-0e4852dac5df
Strict-Transport-Security: max-age=31536000
X-Aspnet-Version: 4.0.30319
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Ms-Invokeapp: 1; RequireReadOnly
X-Msedge-Ref: Ref A: 21C3E973DAB44BF1A70B7F46AAAD6286 Ref B: MEL01EDGE0416 Ref C: 2018-09-22T00:52:24Z
X-Powered-By: ASP.NET
X-Sharepointhealthscore: 0
X-Sp-Serverstate: ReadOnly=0

4e
{"error":{"code":"itemNotFound","message":"The resource could not be found."}}
0

2018/09/22 10:52:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/09/22 10:52:24 NOTICE: test.docx: Failed to cancel multipart upload: itemNotFound: The resource could not be found.
2018/09/22 10:52:24 ERROR : test.docx: Failed to copy: itemNotFound: The resource could not be found.
2018/09/22 10:52:24 ERROR : One drive root 'TestData': not deleting files as there were IO errors
2018/09/22 10:52:24 ERROR : One drive root 'TestData': not deleting directories as there were IO errors
2018/09/22 10:52:24 ERROR : Attempt 1/1 failed with 1 errors and: itemNotFound: The resource could not be found.
2018/09/22 10:52:24 Failed to sync: itemNotFound: The resource could not be found.
10

Thanks for your help. If you can’t paste it on the forum, maybe you can try for example https://pastebin.com/ ?


How much overhead would there be if you (optionally) stamped the original file checksum and size into the uploaded file as metadata (for office files)? You could use these to compare against what the file was originally when it was uploaded.

This may not work the way you think. Office files should be able to hold metadata, but we cannot guarantee that if another user does change the file, the checksum that we put in the file gets changed accordingly, unless all users stick to using rclone to sync files all the time.

11

I got it pasted using preformatted. Just waiting on moderator approval for it to appear. In case it takes a while here it is in pastebin https://pastebin.com/f83Wbu4E

My use-case is a migration to sharepoint, doing the bulk upload then periodic syncs before cutover, so for me my idea would work fine. The idea is to pin the file on sharepoint to the source, so if the source file is modified it no longer matches the checksum value stored in the metadata. In my case the file won’t ever be modified in sharepoint until the cutover. I can see that this idea wouldn’t have such a broad appeal though :slight_smile:

Another option (again, very use-case specific) would be for rclone to operate in one mode where all uploaded files get a (say) .rclone extension, until the final cutover where the extension is removed.

Thanks

James

12

I suspect this is some other kind of SharePoint’s “evil” act. I did some test and found that this seems to only affect Office files. But if I upload a .docx file, then modifies it with Office Online or single-part upload, rclone can replace it again.


Another option (again, very use-case specific) would be for rclone to operate in one mode where all uploaded files get a (say) .rclone extension, until the final cutover where the extension is removed.

Currently rclone doesn’t provide name mapping or batch renaming directly, so this approach may not be very elegant. :confused:

13

What happens if you subsequently modify the local copy and sync? Does the sharepoint “brokenness” only happen when rclone first uploads the file, or when rclone last uploaded the file?

14

I haven’t got time to perform further tests, but I thought of something that you might want to try:

You can use rclone sync with the --backup-dir flag. That way, instead of overwriting files, rclone moves old files away. Maybe you can supply an argument such as a timestamp for that flag.

P.S. Don’t set the destination of sync to the root (Failed to sync: source and parameter to --backup-dir mustn’t overlap)
P.S.2 The docs say When using sync, copy or move any files... but this flag doesn’t seem to work with commands other than sync. Is this expected? @ncw

15

Yes that works, and will solve the problem for me for now. Even a second time when I change and sync my test.docx file and there is already a test.docx file in the backup folder.

Would it be worth adding a flag that uploads to a temporary name (with a bogus extension), does the checksum check, then renames to the final location?

Another options would be to simply delete the file before uploading. Obviously that creates a hole where the delete might happen but then the upload fails, leaving no file, but it would still be useful in my case.

Finally, it might also be useful to pass a list of extensions to --ignore-size and --ignore-checksum, so that the size and checksum check can still work for non-office files, eg --ignore-checksum=.docx,.xlsx,.pptx

16

Finally, it might also be useful to pass a list of extensions to --ignore-size and --ignore-checksum, so that the size and checksum check can still work for non-office files, eg --ignore-checksum=.docx,.xlsx,.pptx

Maybe I should go to check if this is a bug of the API first. If it’s fixed on their side, it’ll be great.


Would it be worth adding a flag that uploads to a temporary name (with a bogus extension), does the checksum check, then renames to the final location?

Another options would be to simply delete the file before uploading. Obviously that creates a hole where the delete might happen but then the upload fails, leaving no file, but it would still be useful in my case.

I hope we don’t have to, because this slows down the process when there’re a lot of small files.