Rclone Sync onedrive Killed after every 9 mins( 1.5 to 2.5 GB) or so

Suspected Bug
1

I was trying to upload to one-drive about 1TB
few days back it worked fine
but it suddenly it stopped working

I also tried uploading sub directories individually

rclone -v sync “/mnt/HDD_4TB/000_Server/002_Design/” “Share01:/Share01/002_Design/”

tried
–transfers 1
–transfers 2
flags too

3

Did you get this working? If not please post some logs! Thanks

5

This is the problem

socket: too many open files

When rclone has been running for a while, can you do

lsof -c rclone

and

netstat -tuanp | grep rclone

And we can see what files are open.

Which rclone are you using? Have you tried with v1.40?

6

https://pastebin.ubuntu.com/p/pzPYVCrGqB/

https://pastebin.ubuntu.com/p/dthT9wbfMh/
using v1.40

8

Hmm, that is showing 24 sockets open to the remote. Can you share your command line please? Rclone should only use --transfers + --checkers sockets normally.

Are you uploading big files BTW? I wonder if the chunked upload is blowing up the concurrency limit…

There is also a suspicous number of sockets open in lsof which looks strange to me, though these aren’t matched by the number in netstat.

9

I did a bit more analysis of the first paste

$ grep -c 'protocol: TCP' lsof.txt 
131
$ grep -c 'IPv4' lsof.txt 
24
$ grep -c 'ESTABLISHED 22455' lsof.txt 
25

I can see from the lsof that you are using --transfers 4. You are probably using --checkers 8 which is the default so there should be no more than 12 sockets open. However there are 24 open which is odd. Also 131 “protocol: TCP” which I don’t really understand.

Can you do a run of rclone then kill -QUIT it in the middle of the run - either use kill -QUIT or press CTRL-\. This will produce a dump of everything that rclone is doing - can you paste that!

Thanks

10

Hi . i am still trying figure whats the problem… ( i am not super user so hard to figure the problem)
here is is the Code you asked for killed rcloned in the middle of the run using CTRL-\ hope this will be helpfull
https://pastebin.ubuntu.com/p/S7hkP8QJ5g/

11

Thanks that is very useful.

There appear to be ~100 sockets open, but rclone only things it is transferring 4 files.

So there must be a socket leak…

I think I’ve fixed the problem - can you try this beta please?

https://beta.rclone.org/v1.40-020-gf9fcd23b-fix-onedrive-leak/ (uploaded in 15-30 mins)

Thanks

12

Its working … ram usage is also very low…

https://beta.rclone.org/v1.40-020-gf9fcd23b-fix-onedrive-leak/2 (uploaded in 15-30 mins)

I did a test run for 3hours so far everything seems good so far

thank you for the fix

13

Excellent - thank you for testing.

I’ll merge it to master and it will be in the latest beta shortly!

1 Like
14

Hi,

I just updated to 1.41 but I still get an upload loop which possibly relates to this bug. I’m using a OneDrive for Business account. rclone runs on Mac OS 10.13.4.

I have four bigger files which are being uploaded but after about 1 h 40 min the upload fails and the upload starts again.

https://pastebin.com/jvw9ZzK2

The failure part starts like this:
2018/05/10 22:08:02 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2018/05/10 22:08:03 DEBUG : pacer: low level retry 1/10 …

Thanks,
Helge

15

Your sync appeared to retry - which is fine - stuff happens on the internet. The error I can see in the log has this message read tcp 192.168.0.87:61576->13.107.6.151:443: read: connection reset by peer which means that something closed the connection - maybe onedrive, maybe your ISP, maybe your router.

The retry didn’t work properly because of this error

2018/05/10 20:07:54 ERROR : rclone_backup_sync_logfile.txt: Failed to copy: can’t copy - source file is being updated (size changed from 411 to 2152)

So you are syncing the log file of this sync by the look of it… I suggest you --ignore that file otherwise your syncs will never complete properly.

16

Thanks a lot for looking into this, Nick! I’m now trying your suggestion will update you on the outcome. I don’t understand why the error regarding the log file would relate to the continuous interruption of the other file uploads but if the exclusion of the log file would help I’d be more than happy of course. If it doesn’t help I’m afraid that OneDrive for Business may have a upload timeout in place if the upload of a file takes too long. But there’s also a chance that the issue could relate to my ISP (internet via TV cable) since I found a warning message in the router status regarding the WAN connection. If the issue persists I will first look into this potential issue with my ISP. Thanks again, Helge

17

It works, Nick! You were right, it was the exckusion of the log file which probably couldn’t be accessed by rclone for upload. I still don’t get how this caused the upload interruption of the other files. I guess it may be a rsync related thing. Thank you for your help! Helge

18

Great! That will cause the sync to run clean which will help…

I suspect the two are unrelated - The other errors “write: broken pipe” look like they could be caused by the internet connection.

19

So far I didn’t have any more issues. I’m happy! Thanks for the clarification that they’re probably unrelated (what I had assumed as well).

1 Like