Can you upload to an rclone mount without caching?

I obviously can't claim to know based on such limited data - but from experience (including working as a network engineer professionally), I'd say this is likely not a problem in rclone/settings based on what I see here.

My best guess is that this is either a routing or piping issue in the WAN (i'd guess more likely the latter).
In plain english this means a spesific route that is being chosen for the data to go from you - via your ISP - to the server (wherever it is - maybe Germany or Netherlands) may be overcrowded or improperly provisioned.

This could be temporary (due to Corona changes, as we know this is causing issues right now), or it could be permanent. If it temporary you should note varying and shifting speeds, and probably better speeds late at night. If it is permanent you should contact the ISP - preferably providing a tracert to the affected server so they can verify the problem and fix it by routing traffic differently or provisioning more resources to that link.

@ncw Can you suggest what the easiest way to get hold of the server-ip for a gdrive in use is? Is this picked up in any of he logs?

1 Like

At that speed you are going to start to run into some Google infrastructure limitations pr. connection.
You should be able to easily double that or more if you copy 3-4 files at once.
(entirely unrelated to the issue the OP has...)

that could be the problem,

i drink Guinness Extra Stout

It's not a great time for that brand-name I think lol...

image

1 Like

It's interesting, because I get the same bottleneck with FileZilla Pro and Mountain Duck (i.e. around 2.3MB/s), but Air Explorer can get 4.7MB/s per file.

Is there any way to bypass this at all? Perhaps break up the file and upload it in parallel (just using rclone)? Honestly, I just signed a contract with my ISP for 2 years, so I don't expect them to be very accommodating at this time...

Done. let me know if I missed anything.

Oh absolutely - you can zip or rar files into multiple archive files (which still open as a single unit).
In winrar for example you simply select that in the options, and can set it as default too.

Or if you prefer an automatic solution - rclone has a "chunker" backend that can split files for you as you desire - an a completely transparent way (so they appear as single files in the remote and function normally).

But I would not discount the ISP doing their job out of hand (assuming that it's not just a temporary thing from Corona). ISP admins are humans too - and they miss things, or automatic systems fail to detect problems. You'd be surprised at how many problems rely on user-reporting to be fixed quickly.
This is often a problem that a higher-level admin can fix remotely, so don't just assume the problem is "too big" for them to consider helping you. The worst that can happen is that nothing happens and you wasted a few minutes writing them the email :slight_smile:

I wouldn't bother contacting them without concrete data though. For that we need a tracert and maybe some simple pinglogs (I can help you gather that and also analyze the results) - but I'd need to know from NCW how to grab the relevant IP first, as that's not something I've had a need to do yet.

1 Like

once again, you are correct

C:\data\rclone\scripts\rr\other\test>c:\data\rclone\scripts\rclone.exe sync C:\data\rclone\scripts\rr\other\2GB gdrive:testtest\ --drive-chunk-size=256M --transfers=10 --log-file=C:\data\rclone\scripts\rr\other\test\log2GB.txt --log-level=DEBUG --progress
Transferred:            6G / 6 GBytes, 100%, 97.852 MBytes/s, ETA 0s
Transferred:            3 / 3, 100%
Elapsed time:       1m2.7s

It might be using multipart upload. It's not really possible for me to comment when I don't know the internal mechanisms of the software. Most likely it is due to it leveraging more than one concurrent connection compared to the others.

In that case, I'll give it a try once we get that information.

Thank you both very much for spending so much time today to help me with this issue. I hope that the coronavirus spares you both. I have two quick questions about rclone and caching if that's okay.

(1) How much HDD usage will this flag cause me: --vfs-cache-mode writes. For example, if I were to upload 2TB of data via rclone mount, would that mean that (in total) my HDD (where the cache is saved) would have 2TB go through it?

(2) What local resources does transferring data from one cloud to another cloud use? Would it use my HDD at all, as in download the data from one cloud to my HDD, then upload it again to another cloud? or perhaps only use CPU and RAM?

Can I just send you my HDDs by mail, so that you can upload it for me to the cloud? :slight_smile: This would probably be the fastest way...

so far you have only tested one cloud provider, gdrive.
perhaps get a free trial of wasabi and do some testing.
might learn something...

Yes, 2TB will be written (assuming you use a mount, because this flag only applies to mounts). How much you retain of this data (for faster re-access to those files at a later date) is entirely up to you. You can set a maximum-age after which files get removed from cache, a maximum total size, or both. The minimum size for the cache (temporarily) is the size of the largest file(s) you are currently transferring at any given moment (ie. 4 or 5 files probably).

An upload via CLI or webinterface or basically any other option that isn't a mount will have no ned for a write cache (because they don't try to support complicated read+write commands to begin with - not because they are "better" ). A simple upload script is easy to make if you just want to "dump" files to cloud from a particular folder. For more elegant (and complex) setups you can use the new multiwrite union remote... but I am getting way off topic here. Ask if you want more info on these things :stuck_out_tongue:

You can make do without write-cache if you only do simple copy/move uploads even on a mount. Anything else (like the read+write access that a program might expect to be able to do) is not possible without write-cache. This is not so much a fault of rclone but rather an unfortunate consequence of the way cloud systems are designed. If you have a (spinning) HDD then I'd generally recommend using the write-cache anyway because HDDs don't really get worn out in that sense. Not not to any significant degree at least.

cloud-to-cloud you only use the network, some (low) CPU, and a bit of RAM. No data needs to touch your HDD at all. In fact that is optimal in that scenario - and how it would work by default.

Between certain type of remotes (like Gdrive to Gdrive) you can even do transfers server-side without even using your own bandwidth (I just synced 2 drives at about 4GB/sec .... Google servers be scary).
Ask for more details if you want them. Trying to not ramble on too much here... :slight_smile:

Hell yes, I would like to know more about it, but honestly I'm a little embarrassed to torture you like this for hours and hours... You've been so generous with your time today (and @asdffdsa as well) that I'm also a little bit concerned that you'll want to charge me a pretty sum at the end of all this. :slight_smile: So, now you know the real reason why I didn't want to disclose my location details.

On a more serious note, though, both of the things that you mentioned sound extremely interesting and I would like to know more, but gdrive to gdrive transfers is something that I know that I'll need very soon...

Will using rclone copy gdrive1:/ gdrive2:/ do the server-side only copy you mentioned or is there more to it? I guess the CLI needs to be operational during the transfer, not that it would be problematic at those speeds...

That will work, but if you want to server-side it rather than pipe the data through your local PC then you need to:

either add this line to the rclone.conf file (under both Gdrives I'd recommend, as part of their blocks)
server_side_across_configs = true (this is then on by default)
or as a temporary one-operation flag:
--drive-server-side-across-configs
This will make rclone try to server-side whenever it is possible.

There is also a few small prerequisites you need to meet for server-side - that you need to set up once.
The destination drive needs to have permission to read the source drive.

To say how to do this, I need to know if you use a Gdrive personal (like the free one) or a Teamdrive/shared drive from a Gsuite (a common source for "unlimited" drives).
The methods differ a bit (and teamdrives are much easier to set up for this, but it can be done on both).

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

you need to delete that post, MUST NOT share your private id and secrets.
you need to redact that info NOW!!!!

1 Like

Well, a union (especially the vastly improved multiwrite union currently availiable in the latest beta) lets you do a lot of stuff like:

  • Combine many locations (local, clouds and other locations) into a single logical location.
  • Set up data to be handled exactly like you want. Mirror data to all places for example - or "upload" all data to a local folder where it can be handled by a script (A personal favorite). Multiwrite union basically mirrors the functionality of mergerFS on Linux if are familiar with it at all. The sky's the limit now. This stuff does require a bit of understanding just to know what you really want it to do in the first place so it might be overwhelming if you are a beginner (but hey - feel free to ask if needed).
    Documentation is here when you've covered the easier stuff and want to think about more complex setups later :slight_smile:
    https://tip.rclone.org/union/
1 Like

Yep - directly after the last line (the root folder one).
It doesn't really matter exactly where you put it as long as you don't place it in some other remote's block (which is demarcated by [remotename] if you have several ).

I note you don't have an encryption remote.
If you are security sensitive - you might want to consider this?
There are very few drawbacks to using encryption on rclone from a technical perspective.

rclone will do so automatically. If you don't use the remote for several weeks it could expire, and you might need to re-authorize it via rclone config (edit remote and just keep all settings and do the re-auth step).

If a service-account is set up (authorization baked into a single file, used often on server-systems) it would never expire if that is important to you. But it's not like re-authing is difficult to do if needed.

Then whatever account (email address) you use in rclone must be a member of both teamdrives, and the user must at least have read permissions on the source and write/delete permissions on the destination. In teamdrives this is easy to set up (on google drive website), but ask if you run into any problems.

Happy to help :smiley:

1 Like

It depends entirely on the command used to do the copy... If it doesn't do open, write, close but instead does something fancier like truncate or seek then --vfs-cache-mode writes will be needed.

But failing to truncate is not actually a critical error correct? From my (very limited) understanding of this stuff - it basically means you may get some empty padding at the end of the file right? Not sure if that results in checksums being wrong or not - and thus having more serious run-on consequences.

Also, in my experience - truncate seems to be called in any copy to a mount regardless of if it is a very simple sequential transfer. Even to a mounted local remote. In my testing of this though - all files still CRC checked perfectly afterwards so I'm not really sure what rclone is complaining about in this case?

Whether or not to use encryption is currently my main quandary. The problem is, and please correct me if I'm wrong here, that encryption methods are not cross-compatible with different backup software, so once I make a commitment to encrypt TBs of data I cannot simply change software (e.g., rclone to Air Explorer or vice versa). This might be particularly a problem if I don't manage to solve my slow upload (and as it turns out also download) speed for individual files via rclone CLI.

Would it be possible to somehow decrypt filenames in Chrome, so that I could download via Chrome (and get max download speed)? Obviously, when I look into my gdrive in Chrome all the filenames are encrypted, so I don't know what's what. I can see the decrypted filenames in rclone CLI, rclone Browser or rclone mount, but as you know I get only about 20-25% of my upload/download...

Best I can figure at the moment is to move the files that I want to download to a separate folder with rclone Browser, download the files within that folder via Chrome, copy the files to a folder that I mount in rclone as an encrypted remote, then decrypt to another local folder/remote.

I used rclone mount gdrive: z: --vfs-cache-mode off and then drag&drop.This results in it the following error messages in rclone CLI:

2020/03/25 16:22:08 ERROR : Best settings for GD mount read speed- - question - rclone forum.url: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2020/03/25 16:22:08 ERROR : Best settings for GD mount read speed- - question - rclone forum.url: WriteFileHandle: ReadAt: Can't read and write to file without --vfs-cache-mode >= minimal
2020/03/25 16:22:13 ERROR : Best settings for GD mount read speed- - question - rclone forum.url: WriteFileHandle: Can't open for write without O_TRUNC on existing file without --vfs-cache-mode >= writes

Also, when I try to copy a larger file, like 2GB, with rclone mount gdrive: z: --vfs-cache-mode writes I only get to see the Windows copy window for as long as the file is copied to cache. After that it disappears and when I click anything in Windows Explorer, the window freezes and I have to restart explorer.exe. Using rclone mount gdrive: z: --vfs-cache-mode off doesn't have that problem, although the transfer is slow (but that's probably related to my other issue I mentioned).