Rclone union no longer preferring local copies (Windows)

Help and Support
1

What is the problem you are having with rclone?

Rclone union no longer preferring local copies (Windows)

Starting recently, reads against my Rclone union seem to be grabbing the remote (gdrive) copy of a file instead of the local file (where it exists). I have tried rolling back Rclone versions to 1.57 without any change. When executing the command listed in the command field below, the copy will pull the local file instead of the gdrive version, but the same action executed manually through Windows File Explorer will grab the gdrive copy.

Naturally, this affects Plex playback, as it follows the same pattern is the Windows File Explorer.

This problem is quite challenging to troubleshoot as it does not happen every time.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.0

  • os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
  • os/kernel: 10.0.17763.2928 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.18.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe copy --checksum --verbose --transfers 1 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s --stats-file-name-length 0 --fast-list Files_Union:Movies/2 Fast 2 Furious (2) (2003) D:\Test\2 Fast 2 Furious (2) (2003)

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = <removed>.apps.googleusercontent.com
client_secret = <removed>
scope = drive
token = <removed>
team_drive = <removed>
root_folder_id = 

[Files]
type = crypt
remote = gdrive:Files
filename_encryption = standard
directory_name_encryption = true
password = <removed>
password2 = <removed>

[Files_Union]
type = union
upstreams = Files::ro D:\Shared

A log from the command with the -vv flag

https://pastebin.com/bRjp2T6E
2

The default policy is ff or find first.

It's never been guaranteed that the local remote will be found first as it depends on timing.

I would have thought it should be unless your disk or computer is very busy?

Maybe the union should guarantee that the local disk is used first. That would be more efficient if a bit more complicated to code.

3

Thanks for your quick reply Nick. I was aware of the ff behavior, but I don't understand how gdrive could respond faster than my local drive. My local drive setup is 18x drives on a hardware RAID controller, which seems to perform very well.

Union configuration that could preference local drive instead of ff would be neat, but I imagine that wouldn't be quick to implement.

4

Give this a go - it searches local backends first before anything else which seems like a sensible idea to me, but maybe there are drawbacks I haven't thought of...

v1.60.0-beta.6373.1d1d847f1.fix-union-ff-local-first on branch fix-union-ff-local-first (uploaded in 15-30 mins)

5

Thanks very much. I've swapped the version over to this new test but I'm still seeing the same behavior. Sometimes it grabs local, most of the time it grabs remote. Is there some configuration I need to change to use this new option?

6

No it should "just work"!

Can you share a log with -vv of it going wrong (with the new code) please?

7

Nick, log below. Sample file was verified to exist and match on both local and remote.

Log

8

How do you tell that the file was being downloaded from the internet and not from the local part of the union? I don't see that in the log.

9
  • Performance is poor compared to pulling from local disk. I typically see ~30MB/sec from remote, but ~1000MB/sec from local.
  • I can see heavy reads in Windows Resource Manager from *.1e100.net servers when file access is occurring.
  • I cannot see any reads in Windows Resource Manager from D:*, which is where my local files are located.

image
image1020×89 5.3 KB

10

Any other ideas Nick? My next step is a Linux VM to use mergerfs, which I'd rather not do :frowning:

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.