Possible to move files directly into vfs cache dir?

What is the problem you are having with rclone?

Less a problem, more a idea that I figured would be easier and less chance of issue just asking instead of trying myself. I currently have a gdrive encrypted mount going uploading files I push onto it, but have noticed that since Windows thinks the source drive and the mount are two separate drives, that it doesn't just move them, instead duplicating the data in a sense and then deleting the original. Would it be possible to just shutdown the mount, move the files into the cache in the correct location and then start the mount back up and let it catalog the files? Or does rclone have to do some work behind the scenes making this not work how I figure it does

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.22000.556 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "archiveenc:" X: -vvvvvvvv --vfs-cache-mode full --cache-dir Z:\rclone --vfs-read-chunk-size 200M --vfs-cache-max-age 1h --volname 'The Archive' --attr-timeout 1h --drive-skip-gdocs --drive-skip-shortcuts --dir-cache-time 5h --cache-dir 'Z:\rclone' --rc --rc-web-gui --rc-user=admin --rc-pass=admin --transfers 2 --order-by 'size,ascending'

The rclone config contents with secrets removed.

Irrelevant, can provide if necessary but dont figure its needed for just a inquiry 

A log from the command with the -vv flag

Paste  log here

If you run full mode, it'll write locally to the cache and upload it based on your config, 1 hour later.

In general, it's best to leave the cache alone as depending on what is going on, it might not be a full file and a sparse file.

If you are 100% certain you have a complete file, I'd imagine you can move it, but I personally would not as it's asking for trouble/data loss.

I wanted to move things into the cache without going through the mount, not our of the cache. Sorry if that was confusing. I tried it but seemingly without correct vfsmeta files to match the new vfs files, rclone just resets the cache back to before the files were added

If you really want to try this then you'll need to make the vfsmeta files too. That shouldn't be too complicated.

I suppose rclone could have a remote control API to do this to insert this local file into the VFS cache. Maybe moving it, copying in or even using it in place.

What's the problem you are trying to solve though?

I can't figure out why you are trying to do it.

If the file isn't in the mount and you move it to the cache, don't you have an orphan file?

The issue I was trying to solve is less a issue and more an ineffency. Normally if files are moved to another spot on a drive, Windows will just leave the actual data alone and just update the metadata in the filesystem to say the files are now in the new folder, which is almost immediate. But since Windows sees the mount as a different drive, it uses the normal file operation logic of duplicating the data until done and then removing the original file.

Since the files are already on the same drive that I have my vfscache on, I was hoping I could somehow get the files into the cache without having Windows take twice as long to do what is essentially the first situation with a couple of extra steps and then just either manually create the vfsmeta files or find a way to get rclone to accept the files itself and have it create the vfsmeta files programmatically

Not a issue, btw. I absolutely love this program and use it almost constantly, just trying to speed up part of my workflow and this popped into my head yesterday while watching Windows process a large amount of files

Why not instead of copying to the drive location, just upload directly to the remote instead?

Trying to manually manipulate the cache feels like a complex solution to just waiting a little bit.

I use to do what you are doing in a semi similar fashion on Linux using mergerfs, but I've removed that extra layer of software and just let it copy like your scenario.

Does it take a few more cycles? Yes. Does it temporarily double the disk space? Yes.

The big question for me was if those first two were important and I finally landed on removing layers of complexity was better for me so I just let those two slide and have less parts of my process. Disk is cheap and I use a SSD for my cache. If that wears out in 4-5 years, for me, I don't care as I normally replace items like that over time anyway with bigger, faster, better capacity anyway.

Honestly its not much of a issue, just something that popped into my head.

I do have a habit of overcomplicating things lol, gets me in these pickles all the time lol

The reason I have the mount going is I can mostly just forget about it and have it slowly upload things in the background, and I like having the ability to see what all is on the drive just like a normal drive. My upload is only around 4MB a second, so I just let it chug along in the background, letting me hibernate or restart my computer if I need to and wish a single command have it all back up and processing

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.