Disable cache when copying a file on a mounted drive

Help and Support
1

Hi there!

I recently started using RClone to mount a MEGA account as a network drive in Windows 10 using rclone mount mega_nz: M:

My internet speed is about 30 MB/s up and down.

But when I copy a file using Windows' Explorer, the copy goes as fast as about 200 MB/s, and when I check in my MEGA account, the file is not here. I have to wait for a bit to see it appear. The larger the file, the longer I'll have to wait until I see it.

So I guess the file is first copied to a temporary directory on my main drive, before being actually sent to my MEGA account. The problem is that I'll soon have to copy very large files from an external hard drive which will be too big to be stored in my main drive. And I'd also like to be able to know without looking at the console when a transfer is really finished.

I tried to perform the mount with no --vfs-cache-mode option, with --vfs-cache-mode off, with --vfs-cache-mode writes and even with --vfs-cache-mode full (just in case), and I get the exact same behaviour in all these cases.

So my question is: how do I completely disable the cache and force direct copy, which means when I copy a file onto the mounted drive, it is directly copied to my MEGA account (which should show a transfer speed of at most 30 MB/s in Windows' Explorer's copy dialog) ?

I've tried with Windows 10 (20H2, 21H1) and Windows 11, with RClone v1.55.1. I've tried the latest release of WinFSP as well as the latest beta release.

Thanks in advance for your help!

2

And why don't you just use the copy command to make this big/initial transfer?

3

It's not just for the initial transfer, I create or make modifications to very large files daily, so I'd like to make the copy works through Explorer instead of using the command line each time.

EDIT: What I want to do is synchronize a whole drive on my MEGA account, but I have very specific needs for synchronizing which means I have to use a special software for that. Software that only supports synchronization between drives or folders and not through a CLI.

4

by default, there is not a cache, you have to enable it

--vfs-cache-mode CacheMode           Cache mode off|minimal|writes|full (default off)
5

Well, this seems like a bug then, as the data is indeed copied on my local drive first.

Let's say I have a primary drive C:, the data I copy from D:, and the MEGA account mountpoint M:. If I copy a 10 GB file from D: to M:, the C: drive is filled before the actual data is sent to MEGA, even with --vfs-cache-mode off.

When I copy the file, in the logs with the -vv flag there is the following message:

2021/07/02 14:58:41 DEBUG : mega root '': Target remote doesn't support streaming uploads, creating temporary local FS to spool file
2021/07/02 14:58:41 DEBUG : /YYYYYYYYFILE: >Write: n=1048576
2021/07/02 14:58:41 DEBUG : Creating backend with remote "C:/Users/XXXXXXXXXXXX/AppData/Local/Temp/rclone-spool513179859"
6

in what way is this a bug?

as per the documentation, rclone does not do stream uploads for mega.

7

I don't see any mention of this in the docs (Mega or rclone mount).

Also in the end the file is still uploaded to MEGA, the only thing is that it is copied to a local drive first while I would like the file to be uploaded directly from its original location.

8

https://rclone.org/overview/#optional-features

1 Like
9

I see, thanks. But that doesn't answer my question:

Also in the end the file is still uploaded to MEGA, the only thing is that it is copied to a local drive first while I would like the file to be uploaded directly from its original location.

10

That's not how MEGA works:

https://rclone.org/mega/

image
image964Ă—161 21.3 KB

2 Likes
11

Oh I thought that statement was only for the official client.

So that means that the current process is: copy the file to a temporary location, encrypt it locally, then upload it at once on MEGA's servers?

1 Like
12

Yes, that's how it works.

2 Likes
13

Thanks for clarifying this.

One thing I still don't get then is how rclone copy works, as it doesn't create a local copy before sending the file to MEGA's servers.

14

are you sure rclone copy does not create a local copy?
that would be in the debug log.

15

I'm sure indeed, as the available space does not decrease on any drive, unlike when I copy the file using Windows' Explorer.

16

can you copy a single file using a debug log and post that?

17

It says in the debug log as the mount doesn't support streaming uploads so it has to make a local copy

A rclone copy command is not the same a mount command. You can't compare the two.

1 Like
18

Sure:

$ rclone copy some_big_6gb_file mega_nz:/ --progress -vv
2021/07/02 15:59:39 DEBUG : Using config file from "C:\\Users\\XXXXXX\\.config\\rclone\\rclone.conf"
2021/07/02 15:59:39 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone.exe" "copy" "some_big_6gb_file" "mega_nz:/" "-vv" "-P"]
2021/07/02 15:59:39 DEBUG : Creating backend with remote "some_big_6gb_file"
2021/07/02 15:59:39 DEBUG : fs cache: adding new entry for parent of "some_big_6gb_file", "//?/C:/Users/XXXXXX/Downloads/AutoDotFiles"
2021/07/02 15:59:39 DEBUG : Creating backend with remote "mega_nz:/"
2021/07/02 15:59:39 DEBUG : fs cache: renaming cache item "mega_nz:/" to be canonical "mega_nz:"
2021-07-02 15:59:39 DEBUG : some_big_6gb_file: Need to transfer - File not found at Destination
Transferred:      418.500M / 6.305 GBytes, 6%, 3.804 MBytes/s, ETA 26m26s
Transferred:            0 / 1, 0%
Elapsed time:      1m50.7s
Transferring:
 *               some_big_6gb_file.mkv:  6% /6.305G, 2.652M/s, 37m56s

EDIT: The same thing happens with rclone sync by the way

19

hmm, i would have guessed that rclone is not logging the message
Target remote doesn't support streaming uploads
but you wrote that the total amount of free space, as reported by windows explorer, did not shrink by 6GB.

20

I just saw your answer as it seems we've answered at the exact same time ^^'

Ok so the file copy using a mounted drive and a file copy using the builtin copy or sync commands don't work the same.

Thanks for your help :slight_smile: