What if one big file exceeding the local disk size, Can rclone help somehow?

Help and Support
1

What is the problem you are having with rclone?

Wonder if we can use mount, then save one big file which exceeding the local disk size.
if not, any suggestion please.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0-beta.6072.25146b430

  • os/version: darwin 11.6.5 (64 bit)
  • os/kernel: 20.6.0 (x86_64)
  • os/type: darwin
  • os/arch: amd64
  • go/version: go1.18
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Onedirve

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone remoteOneDrive:tem tem mount --daemon
cd tem
wget ex.com/bigFileGreaterThanlocalDisk.tar.gz

The rclone config contents with secrets removed.

Default

A log from the command with the -vv flag

Not encouter any issue yet since the big file still being downloaded.
2

Not really - rclone doens't make space as if you don't have it.

You can copy directly to a remote though.

rclone copy somebigfile remoteOneDrive:

1 Like
3

Perhaps you are looking for rclone copyurl

Example:

rclone copyurl http://ex.com/bigFileGreaterThanlocalDisk.tar.gz remoteOneDrive:tem
1 Like
4

Seem like this command only delivery the result of the file from the URL after it completely downloaded(or uploaded) it to the remote? (since my process of this command is 65% completed, but the remoteOneDrive has not create anything new).

5

You'd have to share a command you ran and a log file to see what's going on as it's tough to guess without any details what the problem may or may not be.

1 Like
6

Command ran:

rclone copyurl http://ex.com/bigFileGreaterThanlocalDisk.tar.gz remoteOneDrive:newCopy -vv

2022/10/15 08:48:58 DEBUG : newCopy: Uploading segment 25794969600/72189852114 size 10485760
2022/10/15 08:49:21 DEBUG : newCopy: Uploading segment 25805455360/72189852114 size 10485760
2022/10/15 08:49:47 INFO :
Transferred: 24.041G / 67.232 GBytes, 36%, 491.602 kBytes/s, ETA 1d1h35m25s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 14h14m38.7s
Transferring:

  •                                   newCopy: 35% /67.232G, 330.907k/s, 38h1m3s

2022/10/15 08:49:53 DEBUG : newCopy: Uploading segment 25815941120/72189852114 size 10485760
2022/10/15 08:50:27 DEBUG : newCopy: Uploading segment 25826426880/72189852114 size 10485760
2022/10/15 08:50:47 INFO :
Transferred: 24.057G / 67.232 GBytes, 36%, 491.351 kBytes/s, ETA 1d1h35m38s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 14h15m38.7s
Transferring:

  •                                   newCopy: 35% /67.232G, 230.829k/s, 54h28m50s

2022/10/15 08:51:08 DEBUG : newCopy: Uploading segment 25836912640/72189852114 size 10485760
2022/10/15 08:51:41 DEBUG : newCopy: Uploading segment 25847398400/72189852114 size 10485760
2022/10/15 08:51:47 INFO :
Transferred: 24.076G / 67.232 GBytes, 36%, 491.166 kBytes/s, ETA 1d1h35m32s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 14h16m38.7s
Transferring:

  •                                   newCopy: 35% /67.232G, 405.923k/s, 30h58m0s
7

What's failing there? Just looks slow from the output.

1 Like
8

no failing,(as far as can see). but I use this command on multiple devices, just would like to to learn it “normal” expectation.

If that is something expected, would be more patience.

9

That's really hard to answer as well.

Without knowing the URL that's being hit, it could be the source.

OneDrive is generally not great on performance but I don't use it so I really have nothing to compare it to albeit 400k/s is pretty damn slow.

When you use copyurl, it goes through your download/upload as well so if that's slow, that could also be it.

"Normal" is hard to pin down as everyone's setup, use is generally very unique so normal for me, might not be for you and vice versa.

1 Like
10

Really appreciate your quick response with insight detail.
When I said "normal", I was trying to learn how copyurl work behind the scene: 1. Create destination folder if not exits after the command is ran right way, then upload file. 2. Completely download the file first, then create any necessary folder/file if destination not exist.(should have asked a better question for help next time)
While I know my devices is already slow, this speed is somehow known (to me), then will wait for the file completely downloaded first, then upload.
Please correct if the understanding of copyurl is not true.

11

rclone will do the download and upload concurrently at the speed of the slowest remote. The file is down/uploaded in segments of 10MB as you see here:

I guess you are limited by the download speed or the speed of your internet connection. OneDrive is typically much faster than this.

You can verify this by watching the speed of copyurl to you local disk in a few minutes, before stopping again.

I am pretty sure OneDrive doesn't not show the file before the upload is complete. So this is as expected.

12

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.