Removing partially written file on error: unexpected EOF


#1

Using screen -dmS rclone rclone copy amazon:backup /media/storage/ --exclude "/backup/media/Music/" --size-only --copy-links

I’m getting a lot of 2017/04/01 17:37:19 NOTICE: folder/folder/file.file: Removing partially written file on error: unexpected EOF . Is this because they are hardlinks?


#2

Unlikely - I suspect it is amazon being unreliable :frowning:


#3

Hi NCW,
I am recovering 2.7tb from DB for our organisation for the local archive.

rClone has be super…

that said, I have received the EOF file error on large video files <300mb
MP4, MOV, and AVI (and one pptx)

These videos were recorded at various events and with various devices.
They also were uploaded at different times to Dropbox (2010-2018).
The common denominator appears to be the file size, and somehow the timeout.

I am wondering if it relates to the simultaneous downloads? Or the Chunking?

As it seems to get stuck on a file or two in the directory,
much of this is raw footage, and as such there can be 10-12 clips in a directory
the error only seems to affect a few, and especially the first in a sequence…
although it doesn’t have the trouble with the other similar files i.e. those later in the sequence 004+

2018/02/09 09:11:00 NOTICE: Office/ARCHIVE/SANY0002.MP4: Removing partially written file on error: unexpected EOF
2018/02/09 09:11:04 NOTICE: Office/ARCHIVE/SANY0001.MP4: Removing partially written file on error: unexpected EOF

If you need, please PM me, and I can forward a text file with the output copied from my current session; as well as exact file sizes etc.

downloading from Dropbox Business account as Admin/unlimited
installed rClone yesterday using the script method on OSx 10.11.6
iMac 16gb i7-3770; writing to external USB 3.0 Seagate 5tb drive
Connection is xDSL in Belgium

I am wondering if this is something that can be tweaked, as it appears to affect the media files most, and those when it goes into a directory. I have not restarted rClone, as except for about 10 affected files, it continues to chug away.

many thanks for the great tool, and any help or suggestions,
D


#4

What you should find is that when rclone gets to the end of the transfer it will retry those ones that it had a problem with, most likely sucessfully.

So I would wait for the retry and see what happens.

You can always run rclone again to pick up any missing files too.


#5

Hi NCW,

Correct – when rClone comes to an EOF it retries the file.
sometimes, it fails with another EOF, and even sometimes fails.
Running again, seems to get these.

The challenge for my case is time and data.
rClone retries the file from the beginning, and since these can be larger up to 1.5gb each
2-3 retries can be a good amount of time, as well as data.

In my case, I hit the soft cap of my ISP of 500gb, while I had only saved on the disk 350gb.

So far there have been no issues from DropBox as the account is unlimited.

Many people have some data/bandwidth limits, someplace in their workflow.

I have many more examples now.

One interesting observations is that it primarily affects files with images – video, psd, and pptx.
It also seems to affect certain folders. The contents are basically the same.
That was the reason for asking about the chunking. Maybe there is a problem with the way the chunks are being cached? Or maybe it would be necessary for a multi-stream download to launch individual daemons against single files? Is it possible that the pinch is in how the chunks are arriving on the disk and then being checked/assembled.

2018/02/27 02:38:27 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:42:09 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:44:31 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:44:50 NOTICE: Office/MOV03030.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:46:34 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:49:15 NOTICE: Office/MOV03030.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:49:15 ERROR : Office/MOV03030.MPG: Failed to copy: unexpected EOF37
2018/02/27 02:50:33 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:54:48 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:56:18 NOTICE: Office/MOV03035.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:57:29 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 02:59:08 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:00:17 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:00:35 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:01:38 NOTICE: Office/MOV03035.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:05:17 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:09:02 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:12:26 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:13:08 NOTICE: Office/MOV03034.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:15:43 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:16:26 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:18:25 NOTICE: Office/MOV03036.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:19:31 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:24:12 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:24:13 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:28:57 NOTICE: Office/MOV03036.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:29:56 NOTICE: Office/MOV03033.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:30:52 NOTICE: Office/MOV03032.MPG: Removing partially written file on error: unexpected EOF
2018/02/27 03:30:52 ERROR : Office/MOV03032.MPG: Failed to copy: unexpected EOF

Before I thought that the problem was with really big files only. However as can be seen in the screen shot, the files are actually 30-100mb.

That said, even if a solution to this cannot be found, its not a deal breaker, rClone is really the only solution that really works to do Mass Downloads from Dropbox:

best wishes,
D


#6

This is possible, but not supported by rclone yet…

rclone should write the file out as it goes along and check it for integrity after it has been fully downloaded so this shouldn’t be the problem…

I think those errors are to do with networking problems. It might possibly be your ISP closing long running connections as a form of traffic shaping - I know some ISPs do that.

I’m glad rclone is helpful!

I made an issue which would help you](https://github.com/ncw/rclone) - I haven’t time to work on this right now but I’ll put in in the queue!