For some time, I have been having trouble uploading data to Amazon Cloud Drive. I thought it was resolved in v1.34 as it seemed to be working fine for a while, but the problem continues. It is to the point where it is making it impossible to get my data uploaded to ACD, and I am concerned about using rclone during the day because it causes my Internet to go down while the modem reboots.
At present, I am trying to get my Plex video libraries backed up to ACD. My command is using the option “–transfers=1”.
Here are the errors I am getting now:
2016/12/05 14:56:08 Top Shot/Season 3/s03e10.mkv: Upload error detected but didn’t finish upload: Post https ://content-na.drive.amazonaws.com/cdproxy/nodes?suppress=deduplication: write tcp 192.168.1.72:52764->107.21.3.62:443: wsasend: An existing connection was forcibly closed by the remote host. (“HTTP status UNKNOWN”)
2016/12/05 14:56:08 pacer: Rate limited, sleeping for 287.113937ms (1 consecutive low level retries)
2016/12/05 14:56:08 pacer: low level retry 1/1 (error Post https ://content-na.drive.amazonaws.com/cdproxy/nodes?suppress=deduplication: write tcp 192.168.1.72:52764->107.21.3.62:443: wsasend: An existing connection was forcibly closed by the remote host.)
2016/12/05 14:56:08 Top Shot/Season 3/s03e10.mkv: Received error: Post https ://content-na.drive.amazonaws.com/cdproxy/nodes?suppress=deduplication: write tcp 192.168.1.72:52764->107.21.3.62:443: wsasend: An existing connection was forcibly closed by the remote host. - low level retry 2/10
Any ideas what is causing this? If I can’t get this resolved, I am going to have to find a different solution.
This isn’t strictly speaking an rclone problem - it is a problem in your modem - no Internet traffic should cause your modem to reboot.
However rclone is known for using internet resources quite aggressively. I see you are using --transfers 1 which is good. You can also try setting a --bwlimit to limit the total speed.
I think the ultimate solution will involve upgrading the firmware on your modem though and that is what I’d be looking at if it were my modem.
Thank you for the response. I have tried multiple times to get Comcast to upgrade the firmware on my current modem, but they have said multiple times they cannot do it, which I know is bogus because a technician did it once in the past on my previous modem to correct a bug. I have the latest Comcast modem and had it replaced a few months ago due to the problem I was having when using rclone.
No other program causes this issue. For example, I use Arq to backup data to ACD. I do not have any bandwidth limit set as I want it to go as fast as it can. It can run for days and not cause the problem I run into with rclone.
Yesterday, the thought came to me to try setting a bandwidth limit on rclone. I don’t think this should be necessary, but I decided to try it. So far, it has been running fine since yesterday evening.
I am an IT professional and often max out my Internet connection doing many different tasks and often for extended periods of time. rclone is the only program I have used that causes a situation that crashes my router.
rclone uses the Go networking libraries which in turn use the facilities provided by the kernel of your computer to do networking. There should be no possible way rclone can crash your router. Unfortunately we live in the real world and that obviously is happening! I don’t have any idea what particular behaviour of rclone your modem has a problem with though. It is the kind of thing that is extremely hard to track down…
My guess would be that rclone saturates the network quite effectively and that this is overflowing a buffer in your router which is causing it to crash. The fact that --bwlimit seems to help points towards that kind of thing.
You could have a go with Wireshark to see what is happening, but I think your time would be better spent on trying to get a firmware upgrade for the router.
It might be that you could enable or disable features in the router to help - I don’t know how much control you have over it.
Thanks for the response. I agree it is most likely the router not dealing properly with buffering of data. Since the bandwidth limit seems to have resolved the issue, I am back to normal. I miss out on a few more potential GBs per day of data transfer, but the Internet is not going down and my transfers continue to run. All is good.
Since my last post, I have worked out a deal with Comcast and am now at their fastest tier and am getting 30+ Mbps upload speeds consistently in Internet speed tests. I thought this would alleviate the problems and allow me to get faster uploads. Nope. The problems are back. I cannot find any combination of bandwidth and transfers that prevents the issue from occurring. I have even tried Amazon’s own ACD app to see if uploading through the app would resolve the issue. I have the same problems with their app. This is quite frustrating, but proves that it is not an rclone issue. It seems to be an ACD issue. I am losing days of uploads dealing with this problem as I cannot have my Internet crashing constantly as I also work from home.
Does anyone out there have any ideas or is anyone else experiencing these same problems?
Stop paying Comcast tons of money every month to rent a POC modem. Buy your own modem and router. Not only will you not have to deal with this crap, buy you’ll actually save money in the long run.
on this topic i have a few things i used rclone to upload 13TB 2 months nonstop with xfinity modem, it did crash but this was not due to rclone it is due to the crappy hardware that they provide with the buggy firmware they push to the modems. but when they break (the modems) i take them back get a new one. hardware overheats and dies causing more errors i have killed 4 modem/routers from xfinity running a seedbox from home.
xfinity does not like consistent traffic they will reset your modem when they are trying to push a update. if your modem is not a cisco model then its not the latest ver.
also the keyword errors you see are the communication from the modem to comcast has nothing to do with the traffic more to do with the crap firmware.
take your modem into one of the store ask for a replacement they will not fight with you about it. the modems die fast with lot of traffic. due to they are shit hardware. i use about 8-9tb of bandwidth /month so i know exactly how shitty the modems are
factory reset helps sometimes clears the logs which fill up the nvram
