Can I use rclone api to tell my mount to upload existing files?

Help and Support
1

I have my mount setup to upload every 12h but I'm downloading more than I normally would and I decided I would like to temporarly change this so that those files could just be moved to dropbox, I thought if I run the code below, changing the writeback to the default 5s, that rclone would start to upload those files but it doesn't seem to work as files are still showing up as queued for 12h, so I'm just curious to know if there is a way to tell rclone to upload the files in the mount cache without unmounting.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 4.18.0-425.3.1.el8.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rc options/set --json "{\"vfs\": {\"WriteBack\": 5000}}"
2

I think you've set 5,000 nS.

You can use a string here which is easier to get the right number

rclone rc options/set --json "{\"vfs\": {\"WriteBack\": \"5s\"}}"

I don't think the writeback time will be changed for files which are already waiting for a writeback so you'll have to wait for the 12h to expire then they should start uploading.

3

You're right I did set this to nS as I thought it was ms lol.

The writeback time didn't change for new files added, the log still said new files had to wait 12 hours, I ended up changing the value and remounting it, maybe this was due to me setting the value to 5,000nS, I'll expirement with the writeback through the api so see if it does change.

That didn't work

"failed to write options from block \"vfs\": Reshape failed to Unmarshal: json: cannot unmarshal string into Go struct field Options.WriteBack of type time.Duration"

I had to use the following.

rclone rc options/set --json "{\"vfs\": {\"WriteBack\": 5000000000}}"

So I had 1m WriteBack, and I changed this to 5s then added a new file but it still had a 1m delay, so I don't think this can be changed while a mount is still active.

Thanks for the help.

4

I looked at the code and I thought it could, but I'm obviously wrong!

It would be worth reading the value back via the RC just to check it got set properly.

1 Like
5

The value was changed "WriteBack": 5000000000, it just isn't applying it to any file added.

Here is the latest from the log with another file added.

vfs cache: queuing for upload in 1m0s
6

I traced through the code more carefully and you are right, the vfs options will only apply when you make a new mount.

This is because you've been setting the global VFS options and once started a VFS has its own set of private options.

It would be possible to change the options on a running VFS but it would need a new API call I think.

1 Like
7

I think this would be a useful feature to have in the future for mounts.

When I remounted to change the WriteBack, I also increased my cache size as I don't know what would happen if the size of those files exceeded the size of the cache for that mount. So being able to change the WriteBack and increase cache size without remounting could be very useful, especially for those who may want to avoid any downtime.

8

Do you want to propose that as an enhancement on Github?

Maybe you'd like to help implement it? Or your company might like to sponsor it?

9

I will do this later once I get the chance.

I'm actually in the middle of learning golang or should I say trying lol, right now I wouldn't be capable of implementing anything but in the future I would gladly help.

I'm also not using this within a company (zero profit/personal use) but I would gladly contribute to this project, even if this was never added as an enhancement as I can always fall back to using a script for uploading if my situation became a common occurance.

Anyway, thanks for taking the time to respond to me.

1 Like
10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.