When I download files from google drive to local (windows 10) it will "cache" the full file size and then starts downloading. This is really bad.
E.g. A full 1gb file will be created before the download starts. And then the download will start.
For me, my drives hit 100% as soon as I enter the copy command.
HOWEVER, when I use ubuntu to download, My drives will never reach above 50%. It's not transferring cache of the full file as temporary storage. Its downloading and saving as it comes.
E.g. The same 1gb file will have this behavior, 0kb>1kb>10kb>1MB>100MB>500MB>1GB.
You get the idea.
Is there a workaround for windows? Or is it a limitation that's making it impossible. Its extremely inefficient, speed wise.
the copy command will use 100% as soon as the copy starts.
you can limit that by tweaking flag --multi-thread-streams
as for the file size behavior,
i am finding that as soon as i start to copy command to download a 40GB file, the free space on my hard drives drops by 40GB immediately.
and what is worse, is that i am not able to stop the rclone process.
if i try to cancel rclone via ctrl+c, it does not respond. i cannot stop rclone
if i type ctr+c over and over, and wait, after a few minutes, rclone will finally exit on it own.
the fact that i cannot stop rclone via task manager is not a good situation.
and this is the log file
This, as noted above, is rclone pre-allocating the space. This speeds up writing, makes for less fragmented disks and makes sure that the file can be written - you don't have to download 1GB of data to get a disk full error at the end.
I'm just wondering why this is causing you a problem - is it taking ages to create the 1GB file initially? It should be instant, so maybe this isn't working as designed.
ran 3 times with bwlimit=10M
ran 3 times with no bwlimit
it seems that
the larger the file, the longer the delay until ctrl+c responds
the larger the bandwidth limit, the longer the delay until ctrl+c responds
bwlimit MB
size GB
delay sec
10
1.7
2
10
6.8
10
10
46
105
0
1.7
0
0
6.8
6
0
46
32
and the progress is not updated accurately.
conclusion:
perhaps that each thread downloads a chunk at a time and each threads is locked until that chunk is downloaded.
once another chunk is about to be downloaded, the thread yields to respond to ctrl+c
@ncw its not instant. This makes the drive write at maximum speed until the 1GB is reached. For example an SSD, around 400MB/s. For HDD, 150MB/s. So, downloading to HDD will be much slower with this. Its very bad for HDD as it will be 100% utilisation for awhile until the 1GB is reached.
But as per the solutions above, --multi-thread-streams=1 fixed it, albeit being slower overall download speed, not local..
That makes me think that maybe rclone should be calling pre-allocate for multi-thread downloads too....
I just checked - it does.
So both --multi-thread-streams > 1 and <= 1 are using pre-allocate, so that probably isn't the problem.
When rclone is doing multithread downloads it is writing in multiple parts of the same file, effectively making a sparse file. In linux, the sparse file is only allocated on demand however that may be different on Windows..
I just had a quick test and it looks like if you seek to the end of a large file and start writing then Windows writes 0s from the start of the file. I can tell this because the larger seek you put in, the longer it takes.
i ran %rcmd% copy %remote%/%testfile% .\ --log-level=DEBUG --log-file=%logfile% --progress --multi-thread-streams=10
the file created was pre-allocated, but not the full amount.
the total file size is 46GB
just after i ran the command, immedialy the file size was 41.7 and creeps up.
after the download was completed, rclone did not exit in a timely manner.
the progress continued to be updated, the download rate failing for a couple of minutes
but the rclone log showed that the download was completed.
Don't know about this, couldn't get it to work --multi-thread-downloads
However, --multi-thread-streams=1 together with --drive-chunk-size=256M is the solution for Windows. 100% didn't know this, you should put it under windows section for PSA.
