Hi all, I'm fairly new to rclone however we have access to it at our company and are looking to use it for our purposes - so I do apologise if this is fairly simple. Eventually, we are looking to use it to move from GDrive to an S3 storage, however for the time being we are attempting to pull a list of all the files (and currently just folders) present on GDrive. The issue I'm currently facing is testing a balance between speed and memory usage, as the tree command included below ended up failing due to using up all of the 8gb of RAM on the server when run over a larger dataset (this can be increased but I'd like to understand the ideal usage first), I've been looking at how the command can be altered. So far, I've come to the following conclusions:
--fast-list does not really matter here as from what I can gather, this is mostly RAM intensive but saving on API calls, which is not an issue.
--use-mmap apparently is a good way of managing memory, as go seems to have poor memory management and this can help alleviate moving it back to the available pool.
--checkers int seems to be about running things concurrently, I can't tell if this is usually just for moving or copying files and the like or would work here. On the assumption it is useful here, I imagine I can just bump this up until it gets towards the upper limit of memory and that will be where it can be left. I'm assuming more parallel processing means a faster command.
With this in mind, if anybody could clarify the above points or perhaps suggest the best way to balance memory vs speed then that would be great. Currently I can just leave a command going, and any larger directories can be split up once listing out the actual files within the folders, so longer commands aren't an issue, but I don't want to be taking longer than required due to just doing the wrong thing, or have it fail part way due to running out of RAM.
Run the command 'rclone version' and share the full output of the command.
Currently running, but can come back when done with the version.
Which cloud storage system are you using? (eg Google Drive)
GDrive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
If you only need folders, I would avoid the tree command for the big run. It has to build the pretty hierarchy, so it can be heavier than a plain listing. For a recursive folder list I would try: rclone lsf -R --dirs-only GDrive:/Filepath/ > output.txt. I would also leave --fast-list off while RAM is the problem, since it trades fewer API calls for more memory. --checkers can help concurrency, but I would start low and raise it only after watching memory on a smaller subtree.
Thanks for the advice - is there a way to easily get this into a somewhat user friendly format? Currently with tree I can use the hierarchy to create delimiters that split them up into columns based on folder depth in sheets which is very handy for sharing, along with the visuals of the tree to help with readability. Is tree likely to require much more time/memory than a recursive list?
Then import/split on `/` in Sheets. That gives you the folder depth as columns without rclone needing to build the visual tree. `tree` is convenient for humans, but on a very large remote it can cost more memory because it has to keep enough state to print the hierarchy. `--fast-list` is also the first thing I would leave off for this job; it saves API calls but usually increases RAM. Start with low `--checkers` too, since Google API latency is often the limit here rather than local CPU.
Thanks for the suggestion - I completely missed that I could just use the / as the delimiter! The problem now is figuring out a way to delete all text in cells in a row except the last populated cell... but that is a sheets problem.
In terms of memory - with the tree command the memory seemed to exponentially increase until none was left (I assume it pulls them into memory and then builds out the tree once complete), is this also the case with an LS command (I assume not as this brings back a level by level list that can just be sorted alphabetically), and is there any way to limit this memory usage when running over large data sets where I don't know the amount of files or folders included? Thanks
@Naga yes, for Sheets the slash split is probably the easiest: `rclone lsf -R --dirs-only GDrive:/Filepath/ > folders.txt`, import that as one column, then split on `/`. It will not look as nice as `tree`, but it keeps the rclone side simple. `lsf` should stream the listing rather than building the pretty tree in memory, so it should not grow in quite the same way as `tree` (backend/listing options can still matter). If RAM is the limit I would keep `--fast-list` off, keep concurrency modest, and split the job by subfolder or use `--max-depth` for passes. I do not think there is a hard memory cap flag that makes one giant listing safe; better to make the listing smaller and watch it with a test run.
Gotcha - I can visualise a way to make it more readable in my head but getting sheets to do that I'll have to spend more time on. If LSF is just streaming the listing, does this require memory also? And will that also exponentially increase? I'm not sure how the data is written to the text file, I assume if printing within the terminal then there is no real cached data whereas when writing to the file it may just cache the entire output and then save to while once complete since it seems like the file size doesn't change until completion. If the data stream is pretty consistent with the LS method (which on the current run it seems to be) then I can quite easily just bump up the checkers to use the amount of memory that would be appropriate for the machine.
My understanding is that lsf is much more stream-friendly than building a pretty tree: it can print lines as rclone receives entries, so shell redirection should write them out progressively rather than keeping the whole listing in RAM first. There can still be buffering, and the remote/listing backend may keep some state, so I would not expect zero memory use, but it should not grow like a full tree structure. If the file size looks unchanged until the end, check it from another shell with stat or tail -f; some file managers just do not refresh while the command is running. For a huge Drive I would keep --checkers low, avoid --fast-list if RAM is tight, and test one top-level folder or a --max-depth run before letting it scan everything.
Thanks for the advice. I've been doing some testing and it looks like LSF as expected if cancelled early will still print everything it streamed so far. I have been messing with the checkers and it looks like increasing it seems to slow down the command significantly despite no real increase in system resource usage. This confuses me as I assumed upping the checkers would run more listing in parallel and thus completing quicker, unless I'm misunderstanding checkers and how it works with listing. Either way - currently the command I ran following the advice did get through a folder that was causing errors managed to complete so I'm comfortable leaving it to just run in the background - but I'd like to truly understand how to best optimise for the following listing that will also include files
Just an update, simply listing and then organizing and throwing into sheets was good enough for the use case - it's not as pretty but I've managed to catalogue the entire drive (the folders at least anyways... files will be a lot more work). The main problem has actually been the limitations of sheets! Thanks for the help all.