Error reading destination directory

Hello.
Got a problem when syncing with rclone a folder in my computer with the corresponding folder at One Drive.

rclone sync --dry-run --checksum -v /home/eduardo/herschel/h2osubmm/models/regulargrid/ \ onedrive:/herschel/h2osubmm/models/regulargrid

rclone works OK with almost any other folder, but the error with this folder is:
2018/10/23 20:21:53 ERROR : : error reading destination directory: couldn’t list files: UnknownError:
2018/10/23 20:21:53 INFO : One drive root ‘herschel/h2osubmm/models/regulargrid’: Waiting for checks to finish
2018/10/23 20:21:53 INFO : One drive root ‘herschel/h2osubmm/models/regulargrid’: Waiting for transfers to finish
2018/10/23 20:21:53 ERROR : One drive root ‘herschel/h2osubmm/models/regulargrid’: not deleting files as there were IO errors
2018/10/23 20:21:53 ERROR : One drive root ‘herschel/h2osubmm/models/regulargrid’: not deleting directories as there were IO errors
2018/10/23 20:21:53 ERROR : Attempt 1/3 failed with 1 errors and: not deleting files as there were IO errors

and so on til the 3rd attempt.
This folder has up to 106300 files, is there any problem with such number?
On the other hand, the folder at onedrive was filled with files from another computer, and a number of them were empty and thus renamed in onedrive as ~tmpXXname,
where XX are 2 numbers and name is the original name. In the local computer that i’m trying to sync with onedrive, these empty files still have their original names. May there be a problem with this?
Thank you in advance.

Sorry for the next mistake: the ~tmpXX_name are not empty, and i do not know why the name was changed. I have just checked the original name of one of them:
td60tau100um0p338nh2o1e17vtur60r120core1env0col10kabs1_.h2o
and when uploading from the other computer, it was renamed to:
~tmp14_td60tau100um0p338nh2o1e17vtur60r120core1env0col10kabs1_.h2o
Why was this file (and others) renamed? Has it anything to do with the above error?

This is the root cause of the problem.

Can you try just doing rclone lsf onedrive:/herschel/h2osubmm/models/regulargrid and see if that works?

I suspect it will fail in the same way.

I’d debug this with rclone lsf onedrive:/herschel/h2osubmm/models/regulargrid -vv --dump-bodies --log-file rclone.log and look through the HTTP transactions in the log. If you put it on pastebin and paste a link here that would be perfect, of if it has too much confidential stuff in then email it to me nick@craig-wood.com - put a link to this forum post so I’ve got some context - thanks.

Thanks!

I have emailed you the rclone.log file. As you suspected,
rclone lsf onedrive:/herschel/h2osubmm/models/regulargrid
gives the same error:
2018/10/25 09:45:16 ERROR : : error listing: couldn’t list files: UnknownError:

Also,

  • rclone --version gives:
    rclone v1.44
  • os/arch: linux/amd64
  • go version: go1.11.1
  • the same command works OK when applied to any other folder
    (e.g. rclone lsf onedrive:/herschel)
  • the folder onedrive:/herschel/h2osubmm/models/regulargrid exists as i have access to it via web browser
  • In onedrive:/herschel/h2osubmm/models/regulargrid, there are up to 23 files of the type ~tmpXX_originalname; i understand that these are temporary files and that the originalname would be recovered once the file has been uploaded; this suggest that for some reason the process for these 23 files was interrumpted. Nevertheless, these files have apparently all the information they should.

BTW: The rclone sync and copy proceed very slowly with just the --checksum flag; would you recommend some of the following flags to speed up the sync/copy (I have to run it every day)?
–checksum --tpslimit 10 --transfers=32 --fast-list --drive-chunk-size=16384k --drive-upload-cutoff=16384k --checkers 16 -v
Any other numbers for the flags? Any other flag?
(usually, the folders have many files, and some of these files have ~20MB while many others only ~kB)

Digging into that error, it is caused by this which is rclone trying to list the directory in question.

So what it looks like is a timeout, so the operation at microsoft took longer than some timeout - which looks like 20s according to the rclone log and the Duration header.

2018/10/25 09:46:11 DEBUG : HTTP REQUEST (req 0xc00011a500)
2018/10/25 09:46:11 DEBUG : GET /v1.0/drives/XXX/items/XXX/children?$top=1000 HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.44
Authorization: XXXX
Accept-Encoding: gzip

2018/10/25 09:46:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/10/25 09:46:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/10/25 09:46:31 DEBUG : HTTP RESPONSE (req 0xc00011a500)
2018/10/25 09:46:31 DEBUG : HTTP/1.1 504 Gateway Timeout
Transfer-Encoding: chunked
Cache-Control: private
Content-Type: application/json
Date: Thu, 25 Oct 2018 07:46:31 GMT
Duration: 20023.7962
X-Ms-Ags-Diagnostic: {"ServerInfo":{"DataCenter":"North Europe","Slice":"SliceC","Ring":"3","ScaleUnit":"003","Host":"AGSFE_IN_50","ADSiteName":"NEU"}}

c6
{
  "error": {
    "code": "UnknownError",
    "message": "",
    "innerError": {
      "request-id": "XXXX",
      "date": "2018-10-25T07:46:31"
    }
  }
}
0

In the beta below I added a parameter --onedrive-list-chunk - the default is 1000 - can you try the lsf again with this parameter - try halving each time from 1000 until it works. It may not work at all, but it may help!

https://beta.rclone.org/branch/v1.44-022-ge606d456-fix-onedrive-list-beta/ (uploaded in 15-30 mins)

If the files are created wth modification times that increase, then you could so an interim copy like this just to copy all the files created within the last day (say). This will be very much quicker. You should do a full sync every now and again too.

rclone lsf --absolute --files-only --max-age 1d /path/to/local > new_files
rclone copy --files-from new_files /path/to/local remote:path

Note that the parameters starting with --drive- are only for the Google drive backend and onedrive doesn’t support --fast-list so you don’t need that either. You can try increasing the tpslimit to make things go faster - this will make it more likely you get blocked though.

Things will run a bit faster without the --checksum flag, but will rely on the modification time or size of a file changing when it changes.

I have used the beta with --onedrive-list-chunk of 1000, 500, 250, 125, 60, 30, 15, 7, 5, 2, and 1, and in all cases i had the same error as before. For the case --onedrive-list-chunk 1, i saved the log, which gives at some point the same message as above:
2018/10/25 20:52:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/10/25 20:53:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/10/25 20:53:13 DEBUG : HTTP RESPONSE (req 0xc000418f00)
2018/10/25 20:53:13 DEBUG : HTTP/1.1 504 Gateway Timeout
Transfer-Encoding: chunked
Cache-Control: private
Client-Request-Id: XXXX
Content-Type: application/json
Date: Thu, 25 Oct 2018 18:53:13 GMT
Duration: 20012.3596
Request-Id: XXXX
Strict-Transport-Security: max-age=31536000
X-Ms-Ags-Diagnostic: {“ServerInfo”:{“DataCenter”:“North Europe”,“Slice”:“SliceC”,“Ring”:“3”,“ScaleUnit”:“001”,“Host”:“AGSFE_IN_12”,“ADSiteName”:“NEU”}}

c6
{
“error”: {
“code”: “UnknownError”,
“message”: “”,
“innerError”: {
“request-id”: “XXXX”,
“date”: “2018-10-25T18:53:13”
}
}
}
0

Great, thanks for the recommendation, will follow that strategy.

Two more points that could help:

  • I have in onedrive another folder with many files, 20302, but the lsf gives no error for this folder. Since the regulargrid folder has more than 100000 files, could there be any problem going from 5 to 6 digits in number of files?
  • I also tried the following:
    rclone copy --checksum -v /home/eduardo/herschel/h2osubmm/models/regulargrid/
    onedrive:/herschel/h2osubmm/models/regulargrid0
    i.e. i have generated another folder in onedrive (regulargrid0) identical to regulargrid. I did this with the home-computer (rather than with the office-computer that i used to generate regulargrid).
    After 31 hours of smoothly uploading files (~106300), and just after uploading the last file, i got the same error:

    2018/10/25 23:27:36 INFO : reservoir/td95tau100um8p0nh2o96e18vtur60r120core1env0col24kabs1_2rec.tbdv: Copied (new)
    2018/10/25 23:27:38 INFO : reservoir/td95tau100um8p0nh2o96e18vtur60r120core1env0col24kabs1_2rec.tmb: Copied (new)
    2018/10/25 23:27:38 INFO : reservoir/td95tau100um8p0nh2o96e18vtur60r120core1env0col24kabs1_2rec.tex: Copied (new)
    (COMMENT: THE ABOVE IS THE LAST FILE, SUCCESSFULLY COPIED)
    2018/10/25 23:27:38 ERROR : Attempt 1/3 failed with 21 errors and: invalidRequest: One of the provided arguments is not acceptable.
    2018/10/25 23:27:46 INFO :
    Transferred: 98.240G / 98.240 GBytes, 100%, 915.521 kBytes/s, ETA 0s
    Errors: 0
    Checks: 0 / 0, -
    Transferred: 106281 / 106281, 100%
    Elapsed time: 31h15m17.3s

2018/10/25 23:28:46 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 915.033 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h16m17.3s

2018/10/25 23:29:46 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 914.546 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h17m17.3s

2018/10/25 23:30:46 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 914.060 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h18m17.3s

2018/10/25 23:31:01 ERROR : : error reading destination directory: couldn’t list files: UnknownError:
2018/10/25 23:31:01 INFO : One drive root ‘herschel/h2osubmm/models/regulargrid0’: Waiting for checks to finish
2018/10/25 23:31:01 INFO : One drive root ‘herschel/h2osubmm/models/regulargrid0’: Waiting for transfers to finish
2018/10/25 23:31:01 ERROR : Attempt 2/3 failed with 1 errors
2018/10/25 23:31:46 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 913.573 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h19m17.3s

2018/10/25 23:32:46 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 913.087 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h20m17.3s

2018/10/25 23:33:46 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 912.602 kBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h21m17.3s

2018/10/25 23:34:24 ERROR : : error reading destination directory: couldn’t list files: UnknownError:
2018/10/25 23:34:24 INFO : One drive root ‘herschel/h2osubmm/models/regulargrid0’: Waiting for checks to finish
2018/10/25 23:34:24 INFO : One drive root ‘herschel/h2osubmm/models/regulargrid0’: Waiting for transfers to finish
2018/10/25 23:34:24 ERROR : Attempt 3/3 failed with 1 errors
2018/10/25 23:34:24 INFO :
Transferred: 98.240G / 98.240 GBytes, 100%, 912.295 kBytes/s, ETA 0s
Errors: 1 (retrying may help)
Checks: 0 / 0, -
Transferred: 106281 / 106281, 100%
Elapsed time: 31h21m55.2s

:frowning:

Any chance you could show the request that generated that reponse?

@Cnly any ideas here? Is there a way of increasing that 20s timeout?

Any chance you could show the request that generated that reponse?

Do you mean the command? Of course, it was
rclone lsf --onedrive-list-chunk X \ onedrive:/herschel/h2osubmm/models/regulargrid
where X was decreased from 1000 to 1.
Is this what you meant? Thanks.

No sorry, I meant the bit of the log that said “HTTP REQUEST” the bit before this “HTTP RESPONSE” - thanks!

Sorry, i’m not familiar with all this; the request is:
2018/10/25 20:52:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/10/25 20:52:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/10/25 20:52:53 DEBUG : HTTP REQUEST (req 0xc000418f00)
2018/10/25 20:52:53 DEBUG : GET /v1.0/drives/b!CH9zXxo7_0-yIIW1ytT7K5M4tHsnNaBLlJ_4SMgKwo7mI7JMR62KRYIDFkMv2FIk/items/01Q4XPA4QDOWQUBP2KERB3TVRSMZ24TZA2/children?$top=1 HTTP/1.1
Host: graph.microsoft.com
User-Agent: rclone/v1.44-022-ge606d456-fix-onedrive-list-beta
Authorization: XXXX
Accept-Encoding: gzip

2018/10/25 20:52:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/10/25 20:53:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/10/25 20:53:13 DEBUG : HTTP RESPONSE (req 0xc000418f00)
2018/10/25 20:53:13 DEBUG : HTTP/1.1 504 Gateway Timeout

I have emailed you the full .log file.
Many thanks!

Thanks. I just wanted to check rclone was using the correct parameters to the command which it is.

I think the next step for this problem would be for you to make a new issue on github - put a link to the forum discussion - and we can get some more developers looking at the problem.

Thank you! I have just posted it.

1 Like