Split folder with many files into smaller folders

Help and Support
1

I have a folder in my Google Drive that contains many images (+200000). I want to split these files into multiple smaller directories in my Google Drive, containing 30000 images each. Is there any way to do this with rclone?

I have the drive mounted in rclone. I have rclone version v1.51.0 on Windows 10.

2

If you've got the directory mounted, then you can use batch commands, eg this to split them up by initial letter.

cd mountdir
mkdir a
move a* a
mkdir b
move b* b
...
mkdir z
move z* z
3

Thank you for replying! Unfortunately all the files begin with the number 1, and are followed by random numbers. All files are named like this: '1212359897939005441 2020-02-11.jpg'. Can I somehow use the * for the end of the string before .jpg? Then I could do it by date. I tried this, but it doesn't do anything:

rclone move gdrivedu:hkimages gdrivedu:hkimagesfeb --include *2019-02-11.jpg --dry-run -P -vv

Maybe I used the term 'mount' incorrectly. What I mean is that I have my Gdrive set up as a remote in rclone. Sorry, I am a bit new to this, so I just wanted to clear that up.

4

hi,
you might want to update to latest stable at https://rclone.org/install/

filters can be confusing. for testing, try rclone ls
rclone ls gdrivedu:hkimagesfeb --include *2019-02-11.jpg

5

Thanks for replying. I have updated rclone now to newest stable version.

I tried running this:
rclone ls gdrivedu:hkimages --include *2019-02-*.jpg
And succesfully got all images from february 2019 listed in the cmd prompt.

I wish to copy all those images to the 'hkimagesfeb' directory on Gdrive, so I tried:
rclone copy gdrivedu:hkimagesfeb gdrivedu:hkimages --include *2019-02-*.jpg

But it just returns:

2021/03/10 09:50:52 NOTICE:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:      36m0.9s

2021/03/10 09:51:04 INFO  : There was nothing to transfer

Does the filter only work with ls?

6

not sure exactly what you want to do.

you need to post the command with a debug log....

7

Sorry I am bad at communicating what I want to do. Thanks for helping. In the meantime, I have found a solution. In case anyone in the future stumbles upon this post:

I make a text file of all files in the directory that matches my pattern:
rclone lsf gdrivedu:hkimages --include *2019-02-*.jpg > hkimagesfeb.txt

Then I create a directory, and copy all files listed in the txt file to that directory:
rclone mkdir gdrivedu:hkimagesfeb
rclone copy gdrivedu:hkimages gdrivedu:hkimagesfeb --files-from hkimagesfeb.txt -P -v

8

rclone will create the dest folder for you.

rclone copy gdrivedu:hkimages gdrivedu:hkimagesfeb --include *2019-02-*.jpg

9

Oh cool, that is one step less I need to do. Thanks. I do the txt-file step because I want to be able to split it into smaller portions using Python, as some months have too many images for me to transfer them all to one folder. That way I can create sub-folders per month containing a smaller amount of images.

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.