Is there a way to copy a bunch of nested files to a destination without creating directories in destination?
So what i want is:
"rclone copy /path/folder/ remote:destination --include *.txt"
the /path/folder contains the following
/path/folder/subfolder/text1.txt
/path/folder/subfolder/text2.txt
/path/folder/subfolder/text3.txt
i want text1.txt, text2.txt and text3.txt to be copied to remote:destination without creating the subfolder
Am i making sense?
Thank you
copy it with rclone copy
mount it with rclone mount
run:
$ find /yourdirectory -mindepth 2 -type f -exec mv -i '{}' /yourdirectory ';'
This will recurse through subdirectories of yourdirectory (mindepth 2) and move (mv) anything it finds (-type f) to the top level directory (i.e. yourdirectory)
If you are on windows you will need WSL... I'm sure you could do this without it but I don't speak PowerShell...
You are making sense and this isn't something rclone does at the moment.
Perhaps it should?
Rclone could have a --flatten flag which does what you propose.
What do you think rclone should do if it finds two files with the same name? Or leave that problem to the user?
Without reading any docs, I would expect as default behavior for any move/copy utility a "Replace/Skip/Rename" prompt when there are name collisions. Or just an ERROR: File Exists in Destination
I don't think leaving them in place is a good idea, because you could potentially end up with a single orphaned file beneath many levels of subdirectories, and that sounds like a nightmare haha
Since rclone copy just overwrites name collisions, I'm not sure where else to look in the rclone-verse for consistency...
A "duplicate name" error is a good idea I think.
Rclone copy assumes that you are copying the same file from A->B but with --flatten that assumption might be broken.
I'm also thinking whether --flatten should apply to all the rclone commands move/copy/sync/lsf/ls etc. That would be the obvious way to implement it in rclone.
If implemented as an argument would there be a way to flatten existing directory structures?
Or would one need to decide if they wanted "flat" storage at the outset and stick with that
I guess you could do an rclone move --flatten remote:source remote:flattened to flatten the structure on the server then rename flattened back to source.
My only concern would be:
If you want "flat" storage, can that decision be made at any time, or must one design their storage layout that way from day 1?
I think the ability to "flatten" is of limited use if not designed to be able to retroactively flatten a given remote (i.e. if it only works on the initial upload).
Im glad discussions were started, its not a huge problem for me but it would be nice 
I think I was thinking of a very simple flatten (which is what I've had the most requests for) which would just take the leaf names of the existing files and write them in a single dir
So
/a/file1.txt
/a/b/file2.txt
file3.txt
Would become
file1.txt
file2.txt
file3.txt
It would be relatively easy to swap from one to the other - you could run a sync with --track-renames - that would work.
You aren't the first person to ask for this feature!
Would it be possible to dump the process of --track-renames to a json or csv along the way, so that one has an index/manifest of the flattening process? Would that be simple?
It would also allow tracking of files that were not moved to due name collisions, if the preferred methodology is simply to skip files with duplicate names this could be a serious issue. For example, if one is moving photos, there might be dozens of thumbs.db files that get orphaned. Or index.html files if one is moving websites.
Since there is no "undo," when combined with --dry-run I imagine this would also allow a cursory glimpse of the outcome, in a format that would facilitate easy name changes before running --flatten (albeit changes would need to be made manually).
Or is this more complex than you envisioned?
You could run it with -v then it should log everything it does. Not quite as convenient as CSV but would help.
I think those files would make ERROR logs which would be easy to spot.
Well, I have a few different things I could use to test the --flatten feature once it's implemented (many subdirectories, many duplicate file names, etc.). Just let me know if I can be of assistance.
I need to think how flatten might be implemented...
I think what I could do is make it imply --fast-list so rclone would read the whole source listing into memory. That would give an opportunity to detect duplicate file names at that point before the transfer started. Rclone would then swizzle the directory structure of the objects around.
Doing it like that would mean you'd be able to do rclone ls --flatten which would be useful for testing if nothing else.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.