Rclone ignores cache-tmp-wait-time

Suspected Bug
1

What is the problem you are having with rclone?

After copying a file to a mounted drive, the file is instantly copied to the remote, despite cache-tmp-wait-time. For temporary files (.partial/.inProgress), the files get corrupted during/after write.

I hope this is enough to go on.

What is your rclone version (output from rclone version)

v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 x64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

start "" .\rclone.exe ^
--config ..rclone.conf ^
--fast-list ^
--rc ^
--cache-db-path .\cache mount ^
--cache-tmp-upload-path N:\upload-cache ^
--cache-tmp-wait-time 600m ^
--vfs-cache-mode writes ^
--cache-dir .\vfs-cache ^
--verbose ^
--stats 1m ^
--allow-other ^
--exclude .inProgress ^
GoogleP:/ G:

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

failed to transfer file from cache to remote: corrupted on transfer: MD5 crypted hash differ "ab505369554e31440f5ce8e5072acbc6" vs "64fe0cb37eb559de869c9a39cb9c609c"

####.rclone.config (omitted passwords, tokens etc.)
[Google]
type = drive
scope = drive

[GoogleP]
type = crypt
remote = GooglePCached:/
filename_encryption = standard
directory_name_encryption = true

[GooglePCached]
type = cache
remote = Google:/mounts/p
plex_url = http://127.0.0.1:32400
chunk_size = 10M
info_age = 48h
chunk_total_size = 30G

2

You got a combo of two back end configs going on. You'd probably want to remove that and see if things work. If it only works with that, there might be something else. We'd need to see the whole debug log rather than a clip of it to understand what's going on.

That's also why I use mergerfs and a upload script each night as it's just smoother.

3

When you have cache-mode writes on the files will go into the VFS write-cache and then (rather than uploading to Gdrive directly) will land into the tmp-upload-path of the cache if you have that enabled.

The upload delay will only affect the cache, not the ingest of the VFS write-cache.

I generally do not recommend using the tmp-upload feature of the cache as I found several bugs with it that have not been fixed (and probably never will due to it having no maintainer for a while now).
Keep your eyes pealed in the near future for a major overhaul to the VFS caching system that should make the old cache remote pretty much obsolete. NCW is working on this right now and should have several benefits and better integration to the rest of rclone generally.

For any software that writes partial files (let's say a torrent download for example) the ideal solution is always to see if there is an option in the software to keep incomplete files in a separate folder (and keep that folder local) such that they only get transferred when complete.
The problem is that some software will close the files before they are finished instead of keeping the write-access open. As soon as a file closes rclone is going to assume it's done and start handling it, so this casues an issue in some case. Torrents, some rendering software ect.

Outside of this, right now the only other robust solution I can think of is to have a script do your uploading for you (like Animosity is talking about), and then just use --min-age 10m or something to prevent acting on files that are closed by still being modified. The new multiwrite union remote (currently in beta) can do some nice setups like this and mirror setups like Animosity's that use mergerFS (which only exists on Linux).

4

@Animosity022 @thestigma thank for the quick replies.

Sorry. I completely messed up the copy paste. The --vfs-cache-mode writes shouldn't be there. I tried it as a last minute resort before posting and forgot to remove it.

I'm going to try a union drive. I've been hesitant about this in the past - mainly because I'd overlooked the --min-age parameter.

Would it be possible to maintain a 14 day local cache with a script like this?

rclone copy --min-age 3d
rclone move --min-age 14d --delete-empty-src-dirs

Is move/copy the correct action opposed to moveto/copyto?

For Plex, is a type=cache drive the recommended approach?

5

So I tried the union approach but I can't for the life of me write files to the drive without them being upladed instantly.

I even tried disabling the cached drive, but to no avail.

rclone.conf:

[Google]
type = drive
scope = drive

[GoogleEncrypted]
type = crypt
remote = Google:/mounts/p
filename_encryption = standard
directory_name_encryption = true

[GoogleUnion]
type = union
remotes = "GoogleEncrypted:" "N:\Google"

Command:

.\rclone.exe ^
mount GoogleUnion:/ G: ^
--config .\.rclone.conf ^
--fast-list ^
--rc ^
--verbose ^
--stats 1m ^
--allow-other

If I click optimize on a movie in Plex, it starts transferring both the file to be optimized and the optimized file as it is being built.

Shouldn't all files be written directly to my local drive and not to the Google drive at all?

Here's a log of me trying to copy a file to the union drive and afterwards trying to optimize a movie.

2020/03/21 12:22:33 INFO  : Using --user XXXX--pass XXXX as authenticated user
2020/03/21 12:22:33 NOTICE: Web GUI exists. Update skipped.
2020/03/21 12:22:33 NOTICE: Serving Web GUI
2020/03/21 12:22:33 NOTICE: Serving remote control on http://[::]:5575/
The service rclone has been started.
2020/03/21 12:23:35 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

2020/03/21 12:24:06 ERROR : movies/1917 (2019)/Its.Always.Sunny.in.Philadelphia.S13E01.WEBRip.x264-ION10.mp4: WriteFileHandle: Truncate: Can't change size without --vfs-cache-mode >= writes
2020/03/21 12:24:07 ERROR : movies/1917 (2019)/Its.Always.Sunny.in.Philadelphia.S13E01.WEBRip.x264-ION10.mp4: corrupted on transfer
2020/03/21 12:24:07 ERROR : movies/1917 (2019)/Its.Always.Sunny.in.Philadelphia.S13E01.WEBRip.x264-ION10.mp4: WriteFileHandle.New Rcat failed: corrupted on transfer
2020/03/21 12:24:07 ERROR : movies/1917 (2019)/Its.Always.Sunny.in.Philadelphia.S13E01.WEBRip.x264-ION10.mp4: WriteFileHandle.Flush error: corrupted on transfer
2020/03/21 12:24:07 ERROR : IO error: corrupted on transfer
2020/03/21 12:24:09 ERROR : IO error: can't open file - writer failed
2020/03/21 12:24:09 ERROR : IO error: can't open file - writer failed
2020/03/21 12:24:09 ERROR : IO error: can't open file - writer failed
2020/03/21 12:24:35 INFO  : 
Transferred:   	  219.469M / 221.278 MBytes, 99%, 7.037 MBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Transferred:            3 / 4, 75%
Elapsed time:        31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 27.842k/s, 1m6s

2020/03/21 12:25:35 INFO  : 
Transferred:   	  219.469M / 221.278 MBytes, 99%, 2.407 MBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Transferred:            3 / 4, 75%
Elapsed time:      1m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 593/s, 53m16s

2020/03/21 12:26:35 INFO  : 
Transferred:   	  219.469M / 221.278 MBytes, 99%, 1.452 MBytes/s, ETA 1s
Errors:                 1 (retrying may help)
Transferred:            3 / 4, 75%
Elapsed time:      2m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 12/s, 42h39m58s

2020/03/21 12:27:35 INFO  : 
Transferred:   	  219.469M / 221.278 MBytes, 99%, 1.039 MBytes/s, ETA 1s
Errors:                 1 (retrying may help)
Transferred:            3 / 4, 75%
Elapsed time:      3m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 2050h14m58s

2020/03/21 12:28:35 INFO  : 
Transferred:   	  219.469M / 221.278 MBytes, 99%, 828.709 kBytes/s, ETA 2s
Errors:                 1 (retrying may help)
Transferred:            3 / 4, 75%
Elapsed time:      4m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 98521h17m54s

2020/03/21 12:29:35 INFO  : 
Transferred:   	  219.469M / 221.278 MBytes, 99%, 678.576 kBytes/s, ETA 2s
Errors:                 1 (retrying may help)
Transferred:            3 / 4, 75%
Elapsed time:      5m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 0s

2020/03/21 12:30:35 INFO  : 
Transferred:   	  219.593M / 221.402 MBytes, 99%, 574.821 kBytes/s, ETA 3s
Errors:                 1 (retrying may help)
Transferred:            4 / 5, 80%
Elapsed time:      6m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 2037468h8m9.779728896s

2020/03/21 12:30:51 INFO  : movies/1917 (2019)/Plex Versions/optimized version test 886/dd2a1838-f318-44e3-865a-220bb6a7fcf7: Copied (new)
2020/03/21 12:31:35 INFO  : 
Transferred:   	  607.496M / 2.516 GBytes, 24%, 1.346 MBytes/s, ETA 24m22s
Errors:                 1 (retrying may help)
Transferred:            7 / 10, 70%
Elapsed time:      7m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 2323082h28m49.526403072s
 * movies/1917 (2019)/191…Ray.H264.AAC-RARBG.mp4: 15% /2.267G, 8.574M/s, 3m49s
 * movies/1917 (2019)/Ple…ss/1917 (2019).mp4.886:  0% /off, 869.205k/s, -

2020/03/21 12:32:35 INFO  : 
Transferred:   	    1.187G / 2.565 GBytes, 46%, 2.378 MBytes/s, ETA 9m53s
Errors:                 1 (retrying may help)
Transferred:            7 / 10, 70%
Elapsed time:      8m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 0s
 * movies/1917 (2019)/191…Ray.H264.AAC-RARBG.mp4: 39% /2.267G, 9.732M/s, 2m24s
 * movies/1917 (2019)/Ple…ss/1917 (2019).mp4.886:  0% /off, 868.480k/s, -

2020/03/21 12:33:35 INFO  : 
Transferred:   	    1.682G / 2.613 GBytes, 64%, 3.015 MBytes/s, ETA 5m16s
Errors:                 1 (retrying may help)
Transferred:            7 / 10, 70%
Elapsed time:      9m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 0s
 * movies/1917 (2019)/191…Ray.H264.AAC-RARBG.mp4: 59% /2.267G, 6.919M/s, 2m17s
 * movies/1917 (2019)/Ple…ss/1917 (2019).mp4.886:  0% /off, 777.520k/s, -

2020/03/21 12:34:35 INFO  : 
Transferred:   	    2.198G / 2.660 GBytes, 83%, 3.566 MBytes/s, ETA 2m12s
Errors:                 1 (retrying may help)
Transferred:            7 / 10, 70%
Elapsed time:     10m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 0s
 * movies/1917 (2019)/191…Ray.H264.AAC-RARBG.mp4: 79% /2.267G, 11.196M/s, 42s
 * movies/1917 (2019)/Ple…ss/1917 (2019).mp4.886:  0% /off, 869.775k/s, -

2020/03/21 12:35:35 INFO  : 
Transferred:   	    2.657G / 2.709 GBytes, 98%, 3.936 MBytes/s, ETA 13s
Errors:                 1 (retrying may help)
Transferred:            7 / 10, 70%
Elapsed time:     11m31.1s
Transferring:
 *                movies/1917 (2019)/Procmon.exe: 37% /2.883M, 0/s, 0s
 * movies/1917 (2019)/191…Ray.H264.AAC-RARBG.mp4: 97% /2.267G, 4.951M/s, 10s
 * movies/1917 (2019)/Ple…ss/1917 (2019).mp4.886:  0% /off, 874.897k/s, -

2020/03/21 12:35:54 ERROR : movies/1917 (2019)/Plex Versions/optimized version test 886/.inProgress/1917 (2019).mp4.886: WriteFileHandle.Write: can't seek in file without --vfs-cache-mode >= writes
2020/03/21 12:35:57 ERROR : movies/1917 (2019)/Plex Versions/optimized version test 886/.inProgress/1917 (2019).mp4.886: corrupted on transfer
2020/03/21 12:35:57 ERROR : movies/1917 (2019)/Plex Versions/optimized version test 886/.inProgress/1917 (2019).mp4.886: WriteFileHandle.New Rcat failed: corrupted on transfer
2020/03/21 12:35:57 ERROR : movies/1917 (2019)/Plex Versions/optimized version test 886/.inProgress/1917 (2019).mp4.886: WriteFileHandle.Flush error: corrupted on transfer
2020/03/21 12:35:57 ERROR : IO error: corrupted on transfer
6

Looks like you are using the old union, or more accurately the current union - as multiwrite union is not released yet (but is available as a branch beta and very functional already if you want to test it out).

To write your stuff to the local part of the union - just make sure to put your local folder the last one in the "remotes" line. One of the big annoying problems with union so far is that it only allows one destination for all writes while it reads from the union of all (in the order you layer them). This might sound like what you need, except that it also applies to operations like move, rename, delete - which then makes the whole setup very limited in what you can realistically use it for.

Multiwrite union fixes these limitations by basically mirroring the functionality of mergerFS. One of many possible policies you can set is to make writes mappen to the local union-member, but let stuff like move, rename, delete operations apply to them both. This is then ideal - as you can just have a simple script that periodically moves the local content into the cloud-remote seamlessly.

TLDR: If you go down this route, I think I would spend your time getting to grips with the new multiwrite union rather than working around the limitations of the old union - even if it is not generally released quite yet. The old is going to be obsolete very soon.

But of course - as I said the best solution regardless, whenever possible, is to set temporary local workfilders for incomplete/in-progress files in the software. A lot of torrent clients have this - so do many renderer apps. I can't say about Plex since I am not a daily user of that quite yet :slight_smile:

7

@thestigma

The last entry in the remotes line is my local drive. For now the old union should fit my needs if I could only get it to work. (Though being able to delete across all drives would be very welcome of course.)

I can't specify a temporary folder as Plex allows no such option for the optimize feature.

Is there a reason I'm able to copy to the union drive (and verify the files on my local drive), but not be able to have Plex generate a file on the drive?

Would the new multiwrite beta improve on the ability to write files to the drive?

8

Well, no - your setup does look correct, and this should result in any new or changed files being written to the local remote. Your old file will still exist on the Cloud-remote but will effectively be "hidden" behind the new file since the local remote takes priority when files exist in both places. That is why the layering-order is very important.

I do not see an obvious reason why you have that problem - but I really don't know what Plex does with the files internally during this process. If it relies on moving, renaming or deleting files ect. then it is going to run into problems - and it is very hard to know this for sure without being very familiar with how it is coded.

Can you try a simpler test without involving Plex to start - just to see that the basics are working?
Try:

  • copying a new file to the union-remote
  • overwrite a file on the union remote
    Then go into the local remote manually and check that the files have in fact landed here. They certainly should.

It is a massive overhaul of union, so while some code remains the same I would treat them as separate systems. It certainly removes a ton of limitations from the old union system - and except for it not being as thoroughly tested yet it should only provide benefits.

What I am saying is that if it were me I would rather spend my time at learning how to set up a multiwrite union and testing it well rather than spending that time trying to troubleshoot or work around limitations in the old Union. The setup is certainly a little more technical if you have never worked with mergerFS or similar software before, but it is well worth it IMO. I can provide some guidance in terms of what setting might be appropriate for you, although I'm not an expert at the new system yet.