After mounting a union (made up of google drive and aws s3) it doesn't show new files created on the remote (Google Drive) until stopped and mounted again. Also creating files on this union doesn't create them on the underlying remote.
Run the command 'rclone version' and share the full output of the command.
Thank you, the idea of make union of encrypted remotes instead is brilliant - simplifies a lot of things. I've just tried it and it works fine, but unfortunately, the problems didn't go away. Creating files on that decrypted union still doesn't create files on the gdrive remote.
Thanks, that's good to know. Do you know if union has any other way to reflect changes on the remotes? Would it work if you kept unmounting and remounting it over some short interval?
Yes, it looks the same (log below). I will try to mount it without the options and decryption.
2023/06/27 15:37:00 NOTICE: Serving remote control on http://[::]:8239/
2023/06/27 15:37:21 ERROR : wallpapers/iserugbsielgbl: Failed to copy: object not found
2023/06/27 15:37:21 ERROR : wallpapers/iserugbsielgbl: vfs cache: failed to upload try #1, will retry in 10s: vfs cache: failed to transfer file from cache to remote: object not found
2023/06/27 15:37:38 ERROR : wallpapers/iserugbsielgbl: Failed to copy: object not found
2023/06/27 15:37:38 ERROR : wallpapers/iserugbsielgbl: vfs cache: failed to upload try #2, will retry in 20s: vfs cache: failed to transfer file from cache to remote: object not found
2023/06/27 15:38:05 ERROR : wallpapers/iserugbsielgbl: Failed to copy: object not found
2023/06/27 15:38:05 ERROR : wallpapers/iserugbsielgbl: vfs cache: failed to upload try #3, will retry in 40s: vfs cache: failed to transfer file from cache to remote: object not found
2023/06/27 15:38:50 ERROR : wallpapers/iserugbsielgbl: Failed to copy: object not found
2023/06/27 15:38:50 ERROR : wallpapers/iserugbsielgbl: vfs cache: failed to upload try #4, will retry in 1m20s: vfs cache: failed to transfer file from cache to remote: object not found
2023/06/27 15:40:13 ERROR : wallpapers/iserugbsielgbl: Failed to copy: object not found
2023/06/27 15:40:13 ERROR : wallpapers/iserugbsielgbl: vfs cache: failed to upload try #5, will retry in 2m40s: vfs cache: failed to transfer file from cache to remote: object not found
rclone mount gdrive-aws-union: ./ilsuegrbsierbug/ -vv
tried creating a sample file:
touch ilsuegrbsierbug/sloierbguiseg
This failed with an error:
touch: failed to close 'ilsuegrbsierbug/sloierbguiseg': No such file or directory
listing this ilsuegrbsierbug directory showed that this sloierbguiseg file was there, but after umounting and remounting this union it disappeared. It also didn't show in neither of the two encrypted remotes.
Still testing though - some operations don't seem to work when mounting decrypted union: you can create, delete and copy, but renaming files still doesn't work.
Would you consider it a bug in rclone? AWS may not support free space, but I think that if epmfs is a default policy it woudl be nice if rclone ignored AWS and carried over other remotes that support it?
Sorry, I've initially tagged it as a solution, because it solved 90% of the problems, but the main topic of this thread was meant to be lack of updates between union and its remotes, which is still not solved - union isn't picking up updates if they are made on the remote itself rather than on the union.
It would be really nice to find out if the only solution is to keep unmounting and remounting union via a cron job?
Thank you again, I misunderstood your and Animosity022's replies at first, it seems to work now, but there is one thing that I'm not quite sure of:
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
Are directory entries refer to only "inodes" or file contents as well? That is if you download a 1GB file and set dir-cache-time to 10s does it mean that after 10 seconds that 1GB file will need to be downloaded again?
It means that dir structure (including files) is refreshed every duration time. so new content from remote is picked up.
10s is not good idea as you will be constantly listing both remotes - with Google probably you will hit some quotas. With AWS I am not sure if listing is free... better to check. I normally set it to 15 min for non polling remotes.
There is also possibility to refresh cache on demand using rc I see you have enabled.
Perfect, thank you for clarifying. That's good news, but it's still a bit of a shame that polling doesn't work - I've tried refreshing cache on demand using rc, but it didn't work - union directories still stayed the same after a file on one of their remotes was modified until set dir-cache-time passed.
It didn't happen the other way around - if a file was modified in a union changes were reflected almost immediately on its remote (google drive).
It may be a bad idea, but I've set dir-cache-time temporarily to 30 seconds - will monitor billing and see if it makes a difference. Otherwise smaller dir-cache-time intervals may become problematic in a team environment.
Please ignore that, it was referring to long dir-cache-time times before this fixed it.
You'd have better luck doing different remotes and using mergerfs if you want polling. Otherwise, you'd have to implement, sponsor or find someone to implement polling for the union in rclone.