Moving file to local cache is not instant

Help and Support
1

What is the problem you are having with rclone?

When i upload files to my mount, i first transfer them to a local cache folder. As this local cache folder is on the same drive as the files that i want to move, i expect this transfer to be instant. After the files are in the transfer folder i except that the move to the mounted drive at a slower tempo.

What happens now is that it seems like the files are copied and then removed and after a long of transferring the actual upload to the mount finally starts.

Is this expected behaviour and how could i prevent this?

What i want to do is create a mount command thath is able to move my files with the max amount of bandwidth that i specified.

What is your rclone version (output from rclone version)

rclone v1.55.1

  • os/type: windows
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: dynamic
  • go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

mount gdrive: Y: --volname Drive_Uploaden --config C:\Plex\Rclone\rclone.conf --cache-dir=G:\Uploaden\Cache_normaal --vfs-read-chunk-size=32M --vfs-cache-mode minimal --drive-chunk-size=1G --drive-upload-cutoff=1G  --no-traverse --drive-stop-on-upload-limit --rc --rc-web-gui --rc-user=XXX --rc-pass=XXX --rc-serve --rc-addr :5571 --log-file=D:\Logs\uploadenlog.txt --dir-cache-time=96h --bwlimit 12M

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2021-05-10T13:58:18.2281851+02:00"}

A log from the command with the -vv flag

2021/05/10 13:36:09 DEBUG : /: >Releasedir: errc=0
2021/05/10 13:36:09 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:09 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:09 DEBUG : /test.mkv: Write: ofst=0, fh=0x0
2021/05/10 13:36:09 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=0
2021/05/10 13:36:09 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:09 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:09 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:09 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:09 DEBUG : /test.mkv: Write: ofst=1048576, fh=0x0
2021/05/10 13:36:09 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=1048576
2021/05/10 13:36:09 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:09 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=2097152, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=2097152
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/05/10 13:36:10 DEBUG : /: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/05/10 13:36:10 DEBUG : /: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /: Opendir: 
2021/05/10 13:36:10 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/05/10 13:36:10 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/05/10 13:36:10 DEBUG : /: >Opendir: errc=0, fh=0x1
2021/05/10 13:36:10 DEBUG : /: Releasedir: fh=0x1
2021/05/10 13:36:10 DEBUG : /: >Releasedir: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=3145728, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=3145728
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=4194304, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=4194304
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=5242880, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=5242880
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=6291456, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=6291456
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=7340032, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=7340032
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=8388608, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=8388608
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=9437184, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=9437184
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=10485760, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=10485760
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=11534336, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=11534336
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/05/10 13:36:10 DEBUG : /: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2021/05/10 13:36:10 DEBUG : /: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /: Opendir: 
2021/05/10 13:36:10 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2021/05/10 13:36:10 DEBUG : /: >OpenFile: fd=/ (r), err=<nil>
2021/05/10 13:36:10 DEBUG : /: >Opendir: errc=0, fh=0x1
2021/05/10 13:36:10 DEBUG : /: Releasedir: fh=0x1
2021/05/10 13:36:10 DEBUG : /: >Releasedir: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=12582912, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=12582912
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=13631488, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=13631488
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=14680064, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=14680064
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=15728640, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=15728640
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2021/05/10 13:36:10 DEBUG : /test.mkv: Getattr: fh=0x0
2021/05/10 13:36:10 DEBUG : /test.mkv: >Getattr: errc=0
2021/05/10 13:36:10 DEBUG : /test.mkv: Write: ofst=16777216, fh=0x0
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): _writeAt: size=1048576, off=16777216
2021/05/10 13:36:10 DEBUG : test.mkv(0xc0005f33c0): >_writeAt: n=1048576, err=<nil>
2021/05/10 13:36:10 DEBUG : /test.mkv: >Write: n=1048576
2

It has to copy the file to the cache directory. The files once copied are then uploaded to the remote. The files are removed automatically from the cache in 1 hour (the default value).

You want to use cache mode "writes" instead of minimal.

3

So it is not possible to move the file to the cache directory instead of copying it and then deleting it?

4

Rclone handles the cache itself so you can't interact with the cache.

If you copy something to the mount, it copies it to the cache. I'm not sure how a copy would move something.

5

maybe i'm explaining it incorrectly. A move command on the same drive, could be as easy as changing the link and not the actual position. If you could do this with the cache, then it would only change the link and appear in the cache instantly. I currently need to wait for the file to be copied to the cache which is not instant and then wait for the actual upload.

My question is, is it possible to do a move instead of a copy, which seems like it should easily be possible, so that i can immediatly start the actual uploading part.

6

To move a file, it needs to be copied first.

As I shared earlier, you need to use writes or full. Retest and you can see in the log.

7

I don't understand why this is necessary. It may be how it is implemented in rclone, but a simple changing of the link could perhaps be an option in the future. It seems very handy to me with no drawback if you use it correctly.

Ah i see, thank you. I have since changed it but i had not moved any files yet to notice the difference

8

There doesn't appear to be any difference. I still have to wait for the entire file to copy to the cache and then wait for the upload to complete.

10

The file has been copied and I can move it and it'll cancel the upload and reupload the new named file.

2021/05/11 10:49:32 DEBUG : /: >Lookup: node=<nil>, err=no such file or directory
2021/05/11 10:49:32 DEBUG : /: Rename: oldName="test.file", newName="blah.file", newDir=/
2021/05/11 10:49:32 DEBUG : test.file: vfs cache: cancelling upload
2021/05/11 10:49:33 ERROR : test.file: Failed to copy: Post "https://www.googleapis.com/upload/drive/v3/files?alt=json&fields=id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CexplicitlyTrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%2CshortcutDetails%2CexportLinks&supportsAllDrives=true&uploadType=resumable&upload_id=ABg5-UzQL0RdNUvYbzG7QqAijHBbaoy9vCUc_h5rg5Q_qZmiwMXG2nqu6uwcaa4uO3oV8oTqrdPZ1BLM25WxwTvQ9fY": context canceled
2021/05/11 10:49:33 INFO  : test.file: vfs cache: upload canceled
2021/05/11 10:49:33 DEBUG : test.file: vfs cache: cancelled upload
2021/05/11 10:49:33 INFO  : test.file: vfs cache: renamed in cache to "blah.file"
2021/05/11 10:49:33 DEBUG : blah.file: Updating file with <nil> 0xc00094cfc0
2021/05/11 10:49:33 DEBUG : : Added virtual directory entry vDel: "test.file"
2021/05/11 10:49:33 DEBUG : : Added virtual directory entry vAddFile: "blah.file"
2021/05/11 10:49:33 DEBUG : /: >Rename: err=<nil>
2021/05/11 10:49:33 DEBUG : /: Invalidating "blah.file"
2021/05/11 10:49:38 DEBUG : blah.file: vfs cache: starting upload
2021/05/11 10:49:38 DEBUG : 5qfpimqokf3cr5v56dvrqc04bs: Sending chunk 0 length 1073741824
11

the cache is effectively part of a separate file system, even though its on the same partition. They don't know about each other. their data is not the same. hence, its not just the equivalent of moving a file between file systems, it literally is.

12

No matter, one filesystem or not, rclone will by default create a cached copy to detach uploaded data from its origin before doing a long network upload.

@spotter
AFAICS you want to optimize the first (cache-up) stage, e.g. use a hard link in case (1) the cache and the source file live on the same mount and (2) the fs supports hard links. The optimization would not trigger unless all conditions hold.

Do you want to write a patch implementing your optimization?

13

that's now how the file system interface works in any OS. while, its conceptually possible to try to go underneath the OS interfaces provided to file systems, it would be very fragile.

If you don't want it to copy, it to the cache first, use rclone move / copy directly.

14

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.