File replicated when moving to a VFS-cached mount/Exclude folders in rclone mount

agneev · October 11, 2020, 2:33pm

What is the problem you are having with rclone?

An app is trying to move a file to a mount. The remote has a VFS cache on the same drive.

--vfs-cache-mode writes \
--vfs-cache-max-age 168h \
--vfs-cache-max-size 50G \
drive: /drive

So in theory, it should simply move it right? Except it reads the file and writes it... to the same disk.

Is there to avoid this behavior?

What is your rclone version (output from `rclone version`)

rclone v1.53.1
- os/arch: linux/arm
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspberry Pi OS (Debian Buster) - 32-bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone move file.iso /drive/folder1

The rclone config contents with secrets removed.

N/A

A log from the command with the `-vv` flag

N/A

Animosity022 · October 11, 2020, 2:45pm

A log file would super helpful as you can see what it's doing.

Run the command with -vv and share the output.

You don't list where file.iso is in relation to the copy location.

agneev · October 12, 2020, 5:43am

agneev@server:~ $ rclone move file.iso -vv /drive/ 
2020/10/12 11:11:32 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "move" "file.iso" "-vv" "/drive/"]
2020/10/12 11:11:32 DEBUG : Creating backend with remote "file.iso"
2020/10/12 11:11:32 DEBUG : Using config file from "/home/agneev/.config/rclone/rclone.conf"
2020/10/12 11:11:32 DEBUG : fs cache: adding new entry for parent of "file.iso", "/home/agneev"
2020/10/12 11:11:32 DEBUG : Creating backend with remote "/drive/"
2020/10/12 11:11:32 DEBUG : file.iso: Need to transfer - File not found at Destination
2020/10/12 11:11:32 DEBUG : file.iso: Can't move: rename /home/agneev/file.iso /drive/file.iso: invalid cross-device link: trying copy
2020/10/12 11:11:32 DEBUG : file.iso: Can't move, switching to copy
2020/10/12 11:11:32 DEBUG : file.iso: MD5 = d41d8cd98f00b204e9800998ecf8427e OK
2020/10/12 11:11:32 INFO  : file.iso: Copied (new)
2020/10/12 11:11:32 INFO  : file.iso: Deleted
2020/10/12 11:11:32 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 2 / 2, 100%
Deleted:                1
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2020/10/12 11:11:32 DEBUG : 2 go routines active

I guess this is because /drive is a mount?

Animosity022 · October 12, 2020, 10:37am

It's going across to different mountpoints so it can't do a move operation and it has to copy/delete it.

agneev · October 12, 2020, 10:49am

There is absolutely no workaround huh? Kind of a bummer

Animosity022 · October 12, 2020, 11:26am

It's not really a rclone thing as that's how file systems work.

agneev · October 12, 2020, 2:31pm

Can a folder be created inside the mount that can be set to not be synced?

Animosity022 · October 12, 2020, 2:36pm

Do you mean uploaded?

agneev · October 12, 2020, 5:03pm

Yes, that's what I meant.

Animosity022 · October 12, 2020, 5:05pm

So you want to:

write it to the mount
not upload to your remote
sit in the local cache disk forever ?

agneev · October 12, 2020, 5:07pm

No, a folder in the mount that does not sync to the cloud, sort of like --exclude. rclone mount simply ignores that folder.

That way mv to the cache location can work without a complete read and write.

Nothing to do with the cache, yet.

Animosity022 · October 12, 2020, 5:09pm

I am having a hard time following the use case.

If you have a folder on the mount, how did it get there? Don't you copy it there?

A mv command copies the file form the source to destination and deletes the source.

Can you list out the steps and what you'd want/expect to happen?

agneev · October 12, 2020, 5:12pm

Ok, let me try to paint the big picture.

I create a folder at the root of the mount, local. I set the torrent client's location there.

When radarr/sonarr imports, instead of moving the folders from my disk to the mount, it moves it within the mount, to the proper place, which would be the cache, then rclone backs up the moved file(s).

Animosity022 · October 12, 2020, 5:16pm

So you'd have a mount called /local

You write to /local
You move a file within /local
You want to upload that file to a remote?

cd testfelix@gemini:~$ cd test
felix@gemini:~/test$ ls
felix@gemini:~/test$ cp /etc/hosts .
felix@gemini:~/test$ mv hosts blah
felix@gemini:~/test$ rclone copy blah GD:
felix@gemini:~/test$

That's really how a mount already works so I'm guessing that actually isn't what you mean.

agneev · October 12, 2020, 5:37pm

Uhh no, sorry.

Mount is /drive and the folder that would stay on disk would be /drive/local, where the torrent client would do its thing.

Then Sonarr or Radarr would move it /drive/local/??/??.mkv to the folder on the same drive /drive/media/Movies/??.

Animosity022 · October 12, 2020, 5:39pm

If your remote /drive, what is /drive/local if not the same /drive?

If /drive is the remote, you can move files there without an issue regardless of what the folder is called.

agneev · October 12, 2020, 5:41pm

/drive/local would be locally stored and not synced to the remote.

That's all there is to it; exclude a specific folder from syncing.

Animosity022 · October 12, 2020, 5:44pm

/drive is a mounted rclone remote?
/drive/local is a mounted local disk?

agneev · October 12, 2020, 5:49pm

/drive is the mounted rclone remote, while /drive/local is just a folder inside the mount that resides on the local disk.

Animosity022 · October 12, 2020, 5:53pm

So you run a rclone mount command to mount a remote on /drive like rclone mount remote: /drive.

What do you do so /drive/local gets mounted?