Using screen -dmS rclone rclone copy amazon:backup /media/storage/ --exclude "/backup/media/Music/" --size-only --copy-links
I’m getting a lot of 2017/04/01 17:37:19 NOTICE: folder/folder/file.file: Removing partially written file on error: unexpected EOF . Is this because they are hardlinks?
Unlikely - I suspect it is amazon being unreliable 
Hi NCW,
I am recovering 2.7tb from DB for our organisation for the local archive.
rClone has be super…
that said, I have received the EOF file error on large video files <300mb
MP4, MOV, and AVI (and one pptx)
These videos were recorded at various events and with various devices.
They also were uploaded at different times to Dropbox (2010-2018).
The common denominator appears to be the file size, and somehow the timeout.
I am wondering if it relates to the simultaneous downloads? Or the Chunking?
As it seems to get stuck on a file or two in the directory,
much of this is raw footage, and as such there can be 10-12 clips in a directory
the error only seems to affect a few, and especially the first in a sequence…
although it doesn’t have the trouble with the other similar files i.e. those later in the sequence 004+
2018/02/09 09:11:00 NOTICE: Office/ARCHIVE/SANY0002.MP4: Removing partially written file on error: unexpected EOF
2018/02/09 09:11:04 NOTICE: Office/ARCHIVE/SANY0001.MP4: Removing partially written file on error: unexpected EOF
If you need, please PM me, and I can forward a text file with the output copied from my current session; as well as exact file sizes etc.
downloading from Dropbox Business account as Admin/unlimited
installed rClone yesterday using the script method on OSx 10.11.6
iMac 16gb i7-3770; writing to external USB 3.0 Seagate 5tb drive
Connection is xDSL in Belgium
I am wondering if this is something that can be tweaked, as it appears to affect the media files most, and those when it goes into a directory. I have not restarted rClone, as except for about 10 affected files, it continues to chug away.
many thanks for the great tool, and any help or suggestions,
D
What you should find is that when rclone gets to the end of the transfer it will retry those ones that it had a problem with, most likely sucessfully.
So I would wait for the retry and see what happens.
You can always run rclone again to pick up any missing files too.
Hi NCW,
Correct – when rClone comes to an EOF it retries the file.
sometimes, it fails with another EOF, and even sometimes fails.
Running again, seems to get these.
The challenge for my case is time and data.
rClone retries the file from the beginning, and since these can be larger up to 1.5gb each
2-3 retries can be a good amount of time, as well as data.
In my case, I hit the soft cap of my ISP of 500gb, while I had only saved on the disk 350gb.
So far there have been no issues from DropBox as the account is unlimited.
Many people have some data/bandwidth limits, someplace in their workflow.
I have many more examples now.
One interesting observations is that it primarily affects files with images – video, psd, and pptx.
It also seems to affect certain folders. The contents are basically the same.
That was the reason for asking about the chunking. Maybe there is a problem with the way the chunks are being cached? Or maybe it would be necessary for a multi-stream download to launch individual daemons against single files? Is it possible that the pinch is in how the chunks are arriving on the disk and then being checked/assembled.
2018/02/27 02:38:27 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:42:09 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:44:31 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:44:50 NOTICE: Office/MOV03030.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:46:34 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:49:15 NOTICE: Office/MOV03030.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:49:15 ERROR : Office/MOV03030.MPG: Failed to copy: unexpected EOF
2018/02/27 02:50:33 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:54:48 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:56:18 NOTICE: Office/MOV03035.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:57:29 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:59:08 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:00:17 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:00:35 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:01:38 NOTICE: Office/MOV03035.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:05:17 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:09:02 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:12:26 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:13:08 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:15:43 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:16:26 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:18:25 NOTICE: Office/MOV03036.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:19:31 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:24:12 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:24:13 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:28:57 NOTICE: Office/MOV03036.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:29:56 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:30:52 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:30:52 ERROR : Office/MOV03032.MPG: Failed to copy: unexpected EOF
Before I thought that the problem was with really big files only. However as can be seen in the screen shot, the files are actually 30-100mb.
That said, even if a solution to this cannot be found, its not a deal breaker, rClone is really the only solution that really works to do Mass Downloads from Dropbox:
best wishes,
D
This is possible, but not supported by rclone yet…
rclone should write the file out as it goes along and check it for integrity after it has been fully downloaded so this shouldn’t be the problem…
I think those errors are to do with networking problems. It might possibly be your ISP closing long running connections as a form of traffic shaping - I know some ISPs do that.
I’m glad rclone is helpful!
I made an issue which would help you](https://github.com/ncw/rclone) - I haven’t time to work on this right now but I’ll put in in the queue!
First off, thanks for Rclone. I’m posting here because I’m experiencing something somewhat similar and I have not tracked down the cause or resolution yet. I thought I’d post here before GitHub since I’m not sure if this is a bug or not.
I have Rclone serving a Restic REST server with a local folder backend (thanks for the integration!). When backing up across my LAN, I periodically receive errors like the following on the server:
NOTICE: data/e1/e132a1f82f2b1e0d0842c94c521c9d272e4bf956d0e3939f810f58fa3016ed8f: Removing partially written file on error: unexpected EOF
ERROR : data/e1/e132a1f82f2b1e0d0842c94c521c9d272e4bf956d0e3939f810f58fa3016ed8f: Post request put error: unexpected EOF
On the client, the error is like the following:
Save(<data/e132a1f82f>) returned error, retrying after 443.284891ms: client.Post: Post https://user:password@192.168.1.2:8889/data/e132a1f82f2b1e0d0842c94c521c9d272e4bf956d0e3939f810f58fa3016ed8f: read tcp 192.168.1.3:50784->192.168.1.2:8889: read: connection reset by peer
I’m basically running rclone as a systemd service with the following command (I’m using Caddy to reverse proxy the service):
rclone --config /etc/rclone/rclone.conf serve restic --addr 127.0.0.1:8080 --append-only --htpasswd /etc/rclone/htpasswd restic:/Volumes/storage/matt/backup/repos/shared
Given how Restic saves the data, it’s hard to know which files are associated with the errors. The example above is a restic pack file.
One thing I thought might be causing the issue is my usage of --append-only. The pattern seems to be a notice about “Removing partially written file on error: unexpected EOF” followed by “ERROR : … Post request put error: unexpected EOF”. Would append-only prevent the removal of the file and thus lead to the error? I can prune restic repo from the server, but don’t know what’s causing these unexpected EOF errors.
Those errors are likely your cloud provider refusing the upload for some reason (say too busy) or your ISP closing the connection, or the cloud providers load balancer closing the connection or your local router being overloaded or 100 other similar sort of things.
Cloud storage connections often do that sort of thing and rclone does lots of retries to make sure that things go through. However when it is acting as a restic server it doesn’t do those retries - it leaves that to restic. You can see restic doing the retry though so I expect your data is intact. Did restic exit with an error?
I don’t think append-only is relevant here.
Thanks for the reply. The restic repo is a local folder on the same server running rclone so no cloud, ISP, or similar concerns. I suppose it could be my router or LAN. For the time being, I’m stuck with using a power line Ethernet adapter to go from my wireless router to my desktop so my connectivity is far from ideal.
To rule out local network problems, I may try backing up from a VM running on the same server. Since it’ll all be virtual networking on the same box, that will help me determine if it is the wireless router or Power line adapter causing issues between my laptop and my server.
I may also try without append only just to rule that out as well.
I appreciate the insight.
It never exited with an error, but I manually terminated the run after watching it keep failing for a while. I don’t know whether or not it would error out if I let it keep going. It was a fairly large back up job. The smaller ones seem to run ok which gives further credence to the idea of something getting overloaded.
Ah, OK.
This means rclone was reading data and the input ended unexpectedly…
And this means that restic thought rclone killed the connection.
So I think it is quite likely some kind of local networking problem. You may see more stuff if you look in the kernel logs of either machine.
Did it continuously fail the same object or carry on with different objects? My understanding of restic is that it is quite resilient to network failures and will carry on trying.