Scanning speed on a mounted drive (crypt + s3) through syncthing is insanely slow. My few olders with maybe 30-60gb in total take literal days until syncthing is able to finish scanning, so I want to optimize that a bit.
Syncthing has to scan all files on startup, so it's a lot of opening of files, traversing directories and repeating. A very bad setup for a remotely mounted block storage like s3.
I am trying to tweak the syncthing side of course as well, but because I'm using rclone to mount the drive, I wanted to see what I can do to help syncthing a bit out.
Run the command 'rclone version' and share the full output of the command.
Which cloud storage system are you using? (eg Google Drive)
I haven't actually used those settings yet, just found them in the linked post as potential things to try. Currently I don't use any additional settings and just have it as default
S3 is a non polling remote I believe so that means any changes outside the mount won't be picked up until after 72 hours expires.
Ah I see, I didn't know that mattered in rclone. Thanks for clarifying!
Can you start with whatever your default mount is and run through it and share a log as that might shed some light. Also, what numbers are you talking about as lots can mean very many things.
Sure I can try that. Currently I have a few synced folders through syncthing with about ~60gb total, not much, but scanning the entire thing takes anywhere between 4~12 hours. Syncthing scanning is already not the fastest, and I guess the combination with rclone to remotely mount s3 is a very bad match
Are there other syncthing users here? Would be curious how you setup rclone
The amount of data isn't so big, just those 30-60 gb I mentioned, but I am using syncthing ontop of rclone. It's a p2p syncing solution, so all machines in the syncthing network can connect with each other and share the files to their peers.
I'm using rclone on the serverside, to have syncthing put a copy of those files into s3 (wasabi), so syncthing is set to use the mounted rclone s3 volume as it's folder backend. That means on restart, syncthing needs to rescan all the files on that storage to figure out if there has been changes to the blocks, and validate the store. That's the part that's uber slow currently.
I asked over at the syncthing forum as well, and maybe mounted block storage is just not a good fit for syncthing. I have 'tweaking rclone' on my list of todos for the weekend to see if I can help speed things up a bit
I'm using digitalocean and wasabi as cloud storage, which are supposedly directly connected for low latency, so it shouldn't be too different to something like hetzner connected to a smb drive. Although of course being in the same data center probably has better performance... it's worth giving a shot
Sadly most block storage is very expensive, so outside of wasabi there isn't much that has high storage options for relatively competitive pricing
Just to update - I've did some changes that made it a bit faster, mainly moving to a different hosting provider. Connection from digitalocean to wasabi, even in the same country (singapore), wasn't that great. Getting much better performance now so latency was probably an issues as well
More importantly I've stopped using wasabi through rclone as the main storage volume directly. The match with syncthing which relies heavily on having a relatively fast file lookup on disk wasn't good. Maybe some other solution like resilio would have been better? Not sure
Instead, while my storage usage allows it, I ended up using a provider that has HDD storage available that's cheaper than SSD storage. Of course not as cheap as wasabi but oh well, at least for now the issues are gone. I'll revisit when it becomes too expensive, but for now I'll sync a backup of my data with rclone into wasabi once a week.
I still have wasabi mounted through rclone for smaller folders that don't have something like syncthing scanning those (like webdav), and I'm happy with that currently.
The Hetzner stuff looks very nice as well, but I live far away from the datacenter locations so latency and speed isn't that great