I have a folder on my GDrive containing ~20,000 folders. I need to organize them into subfolders, each containing 1000 folders. However, this process is going very slow at the moment.
From what I can see in the debug logs, after each mv operation, it’s flushing the parent directory cache too, resulting in it needing to cache the ~20,000 folders again which is taking ~10 minutes or so.
This will take too long to finish. Any way I can speed this up?
PS: RClone Mount:
rclone mount GDrive:GMedia/Movies Movies --dir-cache-time=24h --attr-timeout=1s --log-level=DEBUG --stats=1m --stats-log-level=NOTICE
If I recall right, I didn’t think fast-list worked on a mount but maybe if you test out use rclone commands against the remote directly? That might be more efficient.
You could always break it into chunks too and maybe move like 15k of the folders and do the work on the others and keep moving stuff around to batch it up.
Drive File Stream worked out wonderfully. Whole thing finished in a few minutes. Thanks.
Chunking won’t be of much use since the whole mount gets locked up after a move till the directory cache is refreshed. rclone move could be an alternative, however, after getting a list of the folders.
I see you’ve managed it now, but what you could do is with the latest beta run rclone rcd then use the remote control to move files so rclone rc operations/moveto - that won’t be invalidating the directory in the same way I don’t think.