ReadFileHandle.Read error: low level retry 1/10: unexpected EOF

Suspected Bug
1

rclone mount encrypted ACD gives ReadFileHandle.Read error: low level retry 1/10: unexpected EOF when hashing a large torrent.

Debian 8
go version go1.7.4 linux/amd64
rclone v1.35-DEV (from github)

When hashing a large torrent, the following error appears in the terminal:

ReadFileHandle.Read error: low level retry 1/10: unexpected EOF: read tcp my.ip.address:44670->amazon_ip_address:443: r

If I add more than one torrent, the client (rtorrent) will crash.

i am using the following command, rclone mount --allow-other --no-modtime --read-only

it is on a VPS with 2GB of ram and 1 x 3.5GHz cpu.

2

Some of those errors are to be expected - they are caused by network errors, possibly at amazon’s end. rclone should retry and things should continue as they were.

Crash how? Was there anything it its log or rclone’s log?

3

crash hard with the likes of:

Caught SIGBUS, dumping stack:
rtorrent(+0x1f01d) [0x560a37f0101d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11100) [0x7fa73ea4b100]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2(+0x84463) [0x7fa73c377463]

Error: Success
Signal code '2': Non-existent physical address.
Fault address: 0x7fa7306ac000
The fault address is not part of any chunk.
Aborted

and the log will say N worker_rtorrent: Starting thread.

i think, in my layman’s terms, these delays show up as IO error in the kernal and rtorrent borks. i guess i am wondering if rclone can mitigate this in any way?

4

Are you writing directly on rclone mount ?
Not really a good idea since rclone mount does not support opening file and writing to it what torrent would require. ( need to wait for ncw to implement it so mount would be fully functional )

5

currently mounted as read-only, but behaviour seemed worse when not mounted as read-only.

6

Can you find out what the error was? Look at the rclone mount log (with -v) at the time it crashes - that should give a clue.

Anyway, a crash like that is a bug in rtorrent and you should probably report it to them.

7

Sorry for the delay. Here is the log from a recent crash. It seems to me that the crash occurs when rclone gives up, but I haven’t really tested it.

rtorrent log:

Caught SIGBUS, dumping stack:
/opt/rtorrent/bin/rtorrent() [0x413d0f]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890) [0x7fa3081cf890]
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0(+0x7e91c) [0x7fa306c0091c]

Error: Success
Signal code '2': Non-existent physical address.
Fault address: 0x7fa2fbb9b000
The fault address is not part of any chunk.

rclone log:

2017/01/31 23:07:31 machine/dir1/dir2/album1/15 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:07:37 machine/dir1/dir2/album1/11 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:07:48 machine/dir1/dir2/album1/08 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:08:53 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:11:30 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: read tcp ip-address:443: read: connection reset by peer
2017/01/31 23:11:32 machine/dir1/dir2/album2/01 - file.raw: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%3A%hash3%22+AND+name%3A%hash2%22: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 2/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 3/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 4/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 5/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 6/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 7/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 8/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 9/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 10/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: read tcp ip-address:443: read: connection reset by peer
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 2/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 3/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 4/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 5/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 6/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 7/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 8/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 9/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 10/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: read tcp ip-address:443: read: connection reset by peer
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 2/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 3/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 4/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 5/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 6/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:32 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 7/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 8/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 9/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 10/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:11:33 machine: Dir.Lookup error: Get https://cdws.us-east-1.amazonaws.com/drive/v1/nodes?filters=parents%hash4-zbeA: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:15:18 machine/dir1/dir2/album1/15 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:15:23 machine/dir1/dir2/album1/11 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:15:28 machine/dir1/dir2/album1/08 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:15:40 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:16:43 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: unexpected EOF
2017/01/31 23:18:56 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: read tcp ip-address:443: read: connection reset by peer
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 2/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 3/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 4/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 5/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 6/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 7/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 8/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 9/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 10/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: read tcp ip-address:443: read: connection reset by peer
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 2/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 3/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 4/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 5/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 6/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 7/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 8/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 9/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 10/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 1/10: read tcp ip-address:443: read: connection reset by peer
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 2/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 3/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 4/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 5/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 6/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 7/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 8/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 9/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: low level retry 10/10: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
2017/01/31 23:19:00 machine/dir1/dir2/album1/13 - file.raw: ReadFileHandle.Read error: Get https://content-na.drive.amazonaws.com/cdproxy/nodes/hash1/content: dial tcp ip-address:443: connect: no buffer space available
8

OK, looks like rtorrent isn't coping well with the EIO (IO Error) rclone returns when it gives up. That is a bug in rtorrent :frowning:

As for why rclone went wrong - the "no buffer space available" message is unusual. It means you've run out of some sort of networking buffer. Exactly what I'm not sure, but you might be able to tweak (or untweak) something in /proc or /etc/sysctl.conf

This might help...

9

i think the linked site refers to buffer space error specifically when running ping, which is related to maxed out arp tables, which is the most common issue when searching google for buffer space. i can ping fine.

looks normal to me, i am not sure where to go from here. do these give any ideas?

Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux

cat /proc/sys/net/core/wmem_max 16777216

cat /proc/sys/net/ipv4/route/max_size 2147483647

cat /proc/sys/fs/file-nr 544 0 100000

sysctl -a | grep somaxconn net.core.somaxconn = 128

cat /proc/sys/net/ipv4/tcp_mem 46230 61642 92460

cat /proc/net/sockstat sockets: used 77 TCP: inuse 4 orphan 0 tw 0 alloc 7 mem 1 UDP: inuse 1 mem 1 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0

sysctl -a | grep route net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.route_localnet = 0 net.ipv4.conf.default.accept_source_route = 1 net.ipv4.conf.default.route_localnet = 0 net.ipv4.conf.eth0.accept_source_route = 1 net.ipv4.conf.eth0.route_localnet = 0 net.ipv4.conf.lo.accept_source_route = 1 net.ipv4.conf.lo.route_localnet = 0 net.ipv4.route.error_burst = 1250 net.ipv4.route.error_cost = 250 net.ipv4.route.gc_elasticity = 8 net.ipv4.route.gc_interval = 60 net.ipv4.route.gc_min_interval = 0 net.ipv4.route.gc_min_interval_ms = 500 net.ipv4.route.gc_thresh = -1 net.ipv4.route.gc_timeout = 300 net.ipv4.route.max_size = 2147483647 net.ipv4.route.min_adv_mss = 256 net.ipv4.route.min_pmtu = 552 net.ipv4.route.mtu_expires = 600 net.ipv4.route.redirect_load = 5 net.ipv4.route.redirect_number = 9 net.ipv4.route.redirect_silence = 5120 net.ipv6.conf.all.accept_source_route = 0 net.ipv6.conf.all.router_probe_interval = 60 net.ipv6.conf.all.router_solicitation_delay = 1 net.ipv6.conf.all.router_solicitation_interval = 4 net.ipv6.conf.all.router_solicitations = 3 net.ipv6.conf.default.accept_source_route = 0 net.ipv6.conf.default.router_probe_interval = 60 net.ipv6.conf.default.router_solicitation_delay = 1 net.ipv6.conf.default.router_solicitation_interval = 4 net.ipv6.conf.default.router_solicitations = 3 net.ipv6.conf.eth0.accept_source_route = 0 net.ipv6.conf.eth0.router_probe_interval = 60 net.ipv6.conf.eth0.router_solicitation_delay = 1 net.ipv6.conf.eth0.router_solicitation_interval = 4 net.ipv6.conf.eth0.router_solicitations = 3 net.ipv6.conf.lo.accept_source_route = 0 net.ipv6.conf.lo.router_probe_interval = 60 net.ipv6.conf.lo.router_solicitation_delay = 1 net.ipv6.conf.lo.router_solicitation_interval = 4 net.ipv6.conf.lo.router_solicitations = 3 net.ipv6.route.gc_elasticity = 9 net.ipv6.route.gc_interval = 30 net.ipv6.route.gc_min_interval = 0 net.ipv6.route.gc_min_interval_ms = 500 net.ipv6.route.gc_thresh = 1024 net.ipv6.route.gc_timeout = 60 net.ipv6.route.max_size = 4096 net.ipv6.route.min_adv_mss = 1220 net.ipv6.route.mtu_expires = 600

10

Is rtorrent opening an unusual number of sockets? - Check with lsof …

Worth a look with dmesg to see if there are any complaints from the kernel.

11

would i identify unusual number of sockets like this?
lsof -c rtorrent | wc -l 648

for dmesg - https://0bin.net/paste/zTcK17VIgQr-enu2#wsSl0V0fddLpDu3b-0Ltyk1ZnbVKpxp6yXPDQn9MMhz

12

i have a new symptom. I upgraded to 4.9 kernel Linux debian 4.9.0-0.bpo.1-amd64 #1 SMP Debian 4.9.2-2~bpo8+1 (2017-01-26) x86_64 GNU/Linux

and restarted vm, and set my mounts from root using the usual command. clone mount --allow-other --no-modtime --read-only etc.

while doing an lftp transfer [edit: rtorrent is irrelevant, problem occurs without rtorrent running], the rclone mountpoint drops and the mountpoint is no longer available. there is nothing in the rclone log.

dmesg has
[ 3080.032189] Out of memory: Kill process 2445 (rclone) score 36 or sacrifice child [ 3080.032235] Killed process 2445 (rclone) total-vm:386024kB, anon-rss:76552kB, file-rss:0kB, shmem-rss:0kB [ 3080.042292] oom_reaper: reaped process 2445 (rclone), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
is this just a question of “not enough ram”?

13

Looks like it to me, yes. Your VM only has 384M by the look of it.

14

Same errors with OP here.
rTorrent sometimes crashes when it tries to check files on a mounted remote (ACD).
I have to delete rtorrent.lock or add the torrents one by one. I’m not sure.
Probably not rclone’s fault but someone with more knowledge could check it out, maybe some more optimal mount (or rTorrent) settings?

15

@amaklp I suggest you start a new topic with a snip from the log