Interesting… I’ve had the suspicion over the years that there is something wrong with the timeout mechanism in rclone, but I’ve never managed to reproduce a problem in any of my tests.
Doubly interesting that
--timeout 1m works but
--timeout 5m doesn’t.
rclone implements an idle timeout on its data channels, so the idea is that it is supposed to timeout after 5 minutes of the channel being idle if it is in the middle of a transfer. That isn’t a standard go feature - it is implemented in rclone.
That is the code I’m suspicious of not always working but I haven’t been able to figure out why.
I agree 100%.
I did a little test to see if the timeout was working…
I started a transfer from s3 then I used netstat to find the IP address in use and put in iptables rules to block the input and output to that IP. After 5 minutes I saw
2018/02/09 09:40:36 NOTICE: 100M: Removing partially written file on error: read tcp 10.x.y.z:43042->184.108.40.206:443: i/o timeout
2018/02/09 09:40:36 DEBUG : 100M: Received error: read tcp 10.x.y.z:43042->220.127.116.11:443: i/o timeout - low level retry 1/10
Commands were these if you want to experiment further
# Start Transfer
rclone copy -vv --buffer-size 0 --bwlimit 1M --stats 1s s3:rclone-big/100M /tmp/big 2>&1 | tee timeout-s3.log
# In another terminal find the IP
netstat -tuanp | grep rclone
# Block it
sudo iptables -I INPUT 1 -s 18.104.22.168 -j DROP ; sudo iptables -I OUTPUT 1 -d 22.214.171.124 -j DROP
# Remove the block when finished
sudo iptables -D INPUT 1 ; sudo iptables -D OUTPUT 1
So the timeout code is working in some situations!
I wonder whether the problem is when the connection never establishes itself. I set all the timeouts in http.Transport and I pass that transport into the s3 sdk.
To debug this further I’d really like to see
- logs (with -vv) of it going wrong.
- tcpdump of the problem (not sure how exactly you’d capture this though in a multithreaded transfer)
- a way of reproducing the problem myself!
If you’d like to continue to debug this then can you please make a new issue on github with as much of the above as you can find!