Caching issue while accessing the files straight after uploading

What is the problem you are having with rclone?

I collect vinyl and CD albums and rip them often to flac files so even if I lose the CD or it breaks, I will still have the same quality. But these files are extremely valuable for me therefore I have decided to store them on gsuite as well. I have tried a simple mount with --vfs-cache-mode=full which worked perfectly. There is only one issue with it. Upon requesting a file it gets downloaded straight and only get served once the complete file has already been downloaded so there is no chunked serving. That's not what I want so I took a look at the caching backend.

I set up the following things (the others not listed here are set to default):
chunk_path: /tmp/music_cache
chunk_no_memory: true (since I am caching into a tmpfs mount, which is already in RAM, there is no need)
chunk_total_size: 19G
db_path: /mnt/SSD_a2/music_db
db_wait_time: 1s
info_age: 2d
tmp_upload_path: /tmp/music_cache_up
writes: true

This works fine. However my ripper code does the following:

  1. A new FLAC file gets written to the mount
  2. After the write is done it immediately opens the file and tries to serve to my media player so I can listen to the end result and see if its broken or not

What I expect is that after the write is done I can immediately open the file for reading too and the music starts to play. However instead I get a file not found back:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/sanic/", line 976, in handle_request
    response = await response
  File "/ripper/", line 197, in media_stream
    headers = {"Content-Length": os.path.getsize(actual_file_path)}
  File "/usr/local/lib/python3.9/", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/music_files/2020-5/VA - Test (2020).flac

The file I just wrote to the file system is not there. Interesting. Even the fuse debug reports that:

2020/05/12 10:29:48 DEBUG : 2020-5/VA - Test (2020).flac: Need to transfer - File not found at Destination
2020/05/12 10:30:08 DEBUG : 2020-5/: Lookup: name="VA - Test (2020).flac"
2020/05/12 10:30:08 DEBUG : 2020-5/: >Lookup: node=<nil>, err=no such file or directory
2020/05/12 10:30:08 DEBUG : fuse: -> [ID=0x23a] Lookup error=ENOENT

Also, for some reason, even if I directly request /music_files/2020-5/VA - Test (2020).flac I can see rclone listing the entire directory like in an ls command. That slows the file serving down as the file doesn't get served until rclone could list the entire directory which is unnecessary in this case, I just want the actual file to be served.

If I wait a bit until the file gets actually uploaded to GSuite and try to play it again, it just works flawlessly.

What should I do in order to have the files served straight?

What is your rclone version (output from rclone version)

rclone v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Alpine Linux 3.11, x64

Which cloud storage system are you using? (eg Google Drive)

Google Drive, GSuite

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount mediacache: /media_files --user-agent "Vinyl Chloride" --config  /conf/rclone.conf

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

It's huge and contains sensitive information. Upon request I can send it in private message/email.

Unfortunately, the cache backend is not maintained at this point so there might be unexpected issues.

You may want to wait until the vfs cache rewrite comes out as if you did find something, it more than likely will not be fixed.

You could try to use cache-tmp-uploads if you are moving things and that may help with the issue. Without seeing the other bits of the debug log, it's hard to guess what the issue was though with that particular request.

Ah I see. I am really sad to hear that. That type of cache was an amazing idea. I hope vfs-cache will have an option to do chuncked serving because currently it's not really optimal to have the whole file fetched first.

I did, it didn't help. Right after writing the file to the mounted cache directory, if I do a read I get a not found error. It's as simple as that. So I can't access the file I just wrote to the mount straight away. Not even or without cache-tmp-uploads.

It will :slight_smile: Chunked serving is one the design goals for the first phase.

1 Like

Amazing. Can't wait to see it happen. :tada:

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.