My use case is that i'm trying to download large file contents directly to onedrive (basically using it as a normal storage device) by pointing the program directly to the folder in the mounted drive and despite messing with the flags (using vfs-cache-mode writes or --vfs-cache-max-age 1m --vfs-cache-max-size 1M), it's still using local storage when download is in progress.
A lot of posts suggest to simply use the copy command because it bypasses local storage, but in this case i don't have enough storage to download the files locally to then use the copy command (which is why i'm trying to download directly to the drive)
Any help is appreciated and sorry if I come off as unclear.
rclone mount with -vv output :
2024/05/06 20:03:46 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "mount" "-vv" "--vfs-cache-mode" "writes" "--vfs-cache-max-age" "1m" "--vfs-cache-max-size" "1M" "jme3:" "j:"]
2024/05/06 20:03:46 DEBUG : Creating backend with remote "jme3:"
2024/05/06 20:03:46 DEBUG : Using config file from "C:\Users\hp\AppData\Roaming\rclone\rclone.conf"
2024/05/06 20:03:46 DEBUG : OneDrive root '': Next delta token is: NDslMjM0OyUyMzE7MzthOTAxNTk0Ni1jNDQxLTRmZWEtODdiNC03MjJkNTkzNGM5ZTg7NjM4NTA2MTkwMjU5MTcwMDAwOzQxNjk3NzMzNDslMjM7JTIzOyUyMzA7JTIz
2024/05/06 20:03:46 DEBUG : vfs cache: root is "C:\Users\hp\AppData\Local\rclone"
2024/05/06 20:03:46 DEBUG : vfs cache: data root is "\\?\C:\Users\hp\AppData\Local\rclone\vfs\jme3"
2024/05/06 20:03:46 DEBUG : vfs cache: metadata root is "\\?\C:\Users\hp\AppData\Local\rclone\vfsMeta\jme3"
2024/05/06 20:03:46 DEBUG : Creating backend with remote "C:/Users/hp/AppData/Local/rclone/vfs/jme3/"
2024/05/06 20:03:46 DEBUG : fs cache: renaming cache item "C:/Users/hp/AppData/Local/rclone/vfs/jme3/" to be canonical "//?/C:/Users/hp/AppData/Local/rclone/vfs/jme3"
2024/05/06 20:03:46 DEBUG : Creating backend with remote "C:/Users/hp/AppData/Local/rclone/vfsMeta/jme3/"
2024/05/06 20:03:46 DEBUG : fs cache: renaming cache item "C:/Users/hp/AppData/Local/rclone/vfsMeta/jme3/" to be canonical "//?/C:/Users/hp/AppData/Local/rclone/vfsMeta/jme3"
2024/05/06 20:03:46 DEBUG : Network mode mounting is disabled
2024/05/06 20:03:46 DEBUG : Mounting on "j:" ("jme3")
2024/05/06 20:03:46 DEBUG : OneDrive root '': Mounting with options: ["-o" "attr_timeout=1" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone" "-o" "volname=jme3"]
2024/05/06 20:03:46 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2024/05/06 20:03:46 DEBUG : OneDrive root '': Init:
2024/05/06 20:03:46 DEBUG : OneDrive root '': >Init:
2024/05/06 20:03:46 DEBUG : /: Statfs:
2024/05/06 20:03:46 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:399507456 Bfree:310976649 Bavail:310976649 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2024/05/06 20:03:46 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/05/06 20:03:46 DEBUG : /: >Getattr: errc=0
2024/05/06 20:03:46 DEBUG : /: Readlink:
2024/05/06 20:03:46 DEBUG : /: >Readlink: linkPath="", errc=-40
2024/05/06 20:03:46 DEBUG : /: Getxattr: name="non-existant-a11ec902d22f4ec49003af15282d3b00"
2024/05/06 20:03:46 DEBUG : /: >Getxattr: errc=-40, value=""
The service rclone has been started.
2024/05/06 20:03:46 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/05/06 20:03:46 DEBUG : /: >Getattr: errc=0
2024/05/06 20:03:46 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2024/05/06 20:03:46 DEBUG : /: >Getattr: errc=0
What I meant is :
I mount onedrive using rclone.
When prompted to download and choose a location for where that download will be downloaded to using whatever program, I point to a folder in the onedrive mount.
The expected behaviour is that I want it to directly be downloaded to the onedrive storage, not downloaded locally then transfered.
For the source to destination question, for example I want to download a large file from the browser and when prompted for the save location, choosing a folder in the onedrive mount for it to be downloaded directly to onedrive, using the onedrive storage and not affecting local storage.
onedrive storage is in the cloud, so i think you mean that you want to upload to onedrive?
lol, try --magic.
the machine needs enough free space to cache the entire large file.
once the entire file is in the cache, only then can rclone start to upload to onedrive.
okay I see then, I believe I was mistaken about the point of rclone mount, I always thought its whole point was to use cloud storage as if it was local, thank you for your time and explanation
also I don't know if I should make another post about this, but is it normal that when mounting and archiving a folder in the same location directly through file explorer, for it to use local storage ?
Oh I see, so if I used a cloud storage that have support for it it would work for my use case ? also is this a rclone specific issue or it is a limitation caused by onedrive itself
is there however a way to at least reduce the size of the cache or maybe break it into chunks to upload those chunks separately so you can reduce local storage being used ?
It's a limitation of OneDrive, as @asdffdsa correctly mentioned, OneDrive needs to know the total size of the file before upload can begin. As you are downloading the file, we do not know the size of the file itself.
There was a method of uploading via URL which would bypass this limit. It was limited to OneDrive Personal:
Yet, the feature has now been retired:
Your only real option unfortunately is to get a big enough drive for rclone to cache the file.
that was just an example, in my case I was thinking of downloading an entire YouTube channel for archiving purposes but unfortunately it doesn't seem a viable option to do it directly as I imagined. (I also asked this for other tasks like using using the cloud storage for uploading drive backups directly without using the local storage but as you now explained it won't work as I envisioned). Thank you again for taking the time to provide this much detail, your work is very much appreciated
Not a problem! yeah unfortunately you would need to keep a cache size big enough to accommodate all your concurrent uploading files. YouTube videos are not that big in size (unless very long videos) and if you do one by one and script it, you might be able to make it work. Basically you download a video, use rclone to upload it (maybe using move instead of using mount so it's more script friendly) and move on to the next video. It will not be the fastest method, but definitely doable.
After reading everybody's comments and spending some time thinking about it myself I agree that the realistic solution is to provide enough local storage to properly cache and process whatever it is that you need.