What is the problem you are having with rclone?
I collect vinyl and CD albums and rip them often to flac files so even if I lose the CD or it breaks, I will still have the same quality. But these files are extremely valuable for me therefore I have decided to store them on gsuite as well. I have tried a simple mount with --vfs-cache-mode=full which worked perfectly. There is only one issue with it. Upon requesting a file it gets downloaded straight and only get served once the complete file has already been downloaded so there is no chunked serving. That's not what I want so I took a look at the caching backend.
I set up the following things (the others not listed here are set to default):
chunk_path: /tmp/music_cache
chunk_no_memory: true (since I am caching into a tmpfs mount, which is already in RAM, there is no need)
chunk_total_size: 19G
db_path: /mnt/SSD_a2/music_db
db_wait_time: 1s
info_age: 2d
tmp_upload_path: /tmp/music_cache_up
writes: true
This works fine. However my ripper code does the following:
- A new FLAC file gets written to the mount
- After the write is done it immediately opens the file and tries to serve to my media player so I can listen to the end result and see if its broken or not
What I expect is that after the write is done I can immediately open the file for reading too and the music starts to play. However instead I get a file not found back:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/sanic/app.py", line 976, in handle_request
response = await response
File "/ripper/main.py", line 197, in media_stream
headers = {"Content-Length": os.path.getsize(actual_file_path)}
File "/usr/local/lib/python3.9/genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '/music_files/2020-5/VA - Test (2020).flac
The file I just wrote to the file system is not there. Interesting. Even the fuse debug reports that:
2020/05/12 10:29:48 DEBUG : 2020-5/VA - Test (2020).flac: Need to transfer - File not found at Destination
<snip>
2020/05/12 10:30:08 DEBUG : 2020-5/: Lookup: name="VA - Test (2020).flac"
2020/05/12 10:30:08 DEBUG : 2020-5/: >Lookup: node=<nil>, err=no such file or directory
2020/05/12 10:30:08 DEBUG : fuse: -> [ID=0x23a] Lookup error=ENOENT
Also, for some reason, even if I directly request /music_files/2020-5/VA - Test (2020).flac I can see rclone listing the entire directory like in an ls command. That slows the file serving down as the file doesn't get served until rclone could list the entire directory which is unnecessary in this case, I just want the actual file to be served.
If I wait a bit until the file gets actually uploaded to GSuite and try to play it again, it just works flawlessly.
What should I do in order to have the files served straight?
What is your rclone version (output from rclone version)
rclone v1.51.0
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Alpine Linux 3.11, x64
Which cloud storage system are you using? (eg Google Drive)
Google Drive, GSuite
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone mount mediacache: /media_files --user-agent "Vinyl Chloride" --config /conf/rclone.conf
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
It's huge and contains sensitive information. Upon request I can send it in private message/email.
Chunked serving is one the design goals for the first phase.