So is there any way to "clean up" those empty, left-behind directories?

Help and Support
1

I’ve been using rclone for more than a year now, to backup a one-terrabyte-and-growing collection of audiobooks, movies, video games, and other types of media to Amazon Cloud Drive. I’ve restructured and reorganized this collection a number of times during this period, changing my mind about which items belong in which folder.

But every time I delete or rename a directory on my local hard drive, the corresponding folder on ACD is left behind. Its contents are deleted, but the folder is still visible in ACD’s web interface. At first it was just a minor annoyance, but it’s gotten to the point where ACD is almost impossible to navigate, empty directories basically everywhere. And there are far too many to delete manually.

This is a well documented limitation of rclone. What I wanted to know is, is there anything that can be done about it? There are far too many empty directories for me to go through and delete all of them by hand. Is there any way I could expedite the process? Maybe generate a list of all the empty folders so that hand deleting them would take hours rather than days? I know that rclone itself doesn’t really understand the concept of folders, but perhaps there’s some other tool I could use alongside it?

I’m VERY close to just wiping ACD and re-uploading everything from scratch, but that would take several months on my slow internet connection.

2

I made a little python3 program which you can used to list all empty directories: http://pub.rclone.org/rclone-show-empty-dirs.py

Download it, run chmod a+x rclone-show-empty-dirs.py and run it ./rclone-show-empty-dirs.py remote:path and it will list all empty directories.

I’ll turn this into a delete all empty directories script in a moment (have to rush off!)

I should probably add another rclone command to do this as it is pretty easy…

1 Like
3

Here is a script which will delete all empty dirs under a remote: http://pub.rclone.org/rclone-delete-empty-dirs.py

Run it with python3.

Use the --dry-run flag with it first to see what it does.

It isn’t very efficient but should get the job done. You may need to run it more than once if you have empty directories within other directories that only have empty directories in.

1 Like
4

Thank you so much for the script! It’s working great!

1 Like
5

Hmm not working for me

plex@flix:/home/flix/scripts$ ./delete-empty-dirs-rclone.py acd-rclone:/encrypted/ --dry-run
Need remote:path argument
Traceback (most recent call last):
  File "./delete-empty-dirs-rclone.py", line 14, in <module>
    os.exit(1)
AttributeError: module 'os' has no attribute 'exit'

Checked with ls

plex@flix:/home/flix/scripts$ rclone ls acd-rclone:/
        0 rclonetest
        0 encrypted/acd-check
etc...
6

I think that is the wrong script - the one I uploaded doesn’t contain os.exit. Try dowloading: http://pub.rclone.org/rclone-delete-empty-dirs.py again and running with python3…

7

Tried it and this one with dry-run shows tons of folders but when i run it without it I receive
2016/10/30 13:19:38 Attempt 1/3 failed with 1 errors and: directory not empty
2016/10/30 13:19:39 Attempt 2/3 failed with 1 errors and: directory not empty
2016/10/30 13:19:39 Attempt 3/3 failed with 1 errors and: directory not empty
2016/10/30 13:19:39 Failed to rmdir: directory not empty
Traceback (most recent call last):
File “./delete-empty-dirs-rclone.py”, line 50, in
subprocess.check_call([“rclone”, “-q”, “rmdir”, dir])
File “/usr/lib/python3.5/subprocess.py”, line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command ‘[‘rclone’, ‘-q’, ‘rmdir’, ‘acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/,v1pv3L1r,KmO3Z7TxBVk8Mi5ZFtfMvsFD03ef6NchG3x1/TCsRs11fNEcXN4rQ7srYP78l’]’ returned non-zero exit status 1

python3 ./delete-empty-dirs-rclone.py acd-rclone:/encrypted --dry-run
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/,v1pv3L1r,KmO3Z7TxBVk8Mi5ZFtfMvsFD03ef6NchG3x1/TCsRs11fNEcXN4rQ7srYP78l
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/,v1pv3L1r,KmO3Z7TxBVk8Mi5ZFtfMvsFD03ef6NchG3x1/TCsRs11fNEcXN4rQ7srYP78l/jVJ5jYxfFpo-dl7rcRdYA6ucY6BR4QFXC2ONuQI4kKu9e1
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/-FxPgWL,c9Dpeilft1ENVVU88GKX7J1j9jTzuHYErRwP8-
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/-HvdZcm7T8YQvVg-HgX9HUkO/JleJ1QXjq8z66LtFsQJhB7zQ
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/-HvdZcm7T8YQvVg-HgX9HUkO/JleJ1QXjq8z66LtFsQJhB7zQ/2gHTSQnjI6x82FLqAbPTMOhVDRbU6zb0UkPYU,YmUbS3u-
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/0-DRYvyo61yNzNqGqtHXjsXDWhuSUdSFr-eIN2HAwEt,F0/L9oyIIMotDfW4HgGEm,ZOWxJ
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/0-DRYvyo61yNzNqGqtHXjsXDWhuSUdSFr-eIN2HAwEt,F0/L9oyIIMotDfW4HgGEm,ZOWxJ/mDMvr4-Lqj86VUMRzfWxI2hShoLELjm3v8EYW8QyCiQIr-
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/0yCMsGmgA4dfK7ASWqs8XGJVboykO-MGLgR5OKH7ds1So-/yydX2ttPolgVsfjydD5kbpXP
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/0yCMsGmgA4dfK7ASWqs8XGJVboykO-MGLgR5OKH7ds1So-/yydX2ttPolgVsfjydD5kbpXP/eLQ6PmgrK71A0gr2IY77gTriHS,KBpSb08vuraNTXbVjy,
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/1jT-fhaJJEOiLOvM1zddkcn6irjpwUmfzA63YkK1hPn8d,/,sndF,VqgasJy5,GTQ52GNTm
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/1jT-fhaJJEOiLOvM1zddkcn6irjpwUmfzA63YkK1hPn8d,/,sndF,VqgasJy5,GTQ52GNTm/G3AoeNponI96lg9A-5SAvWJw2lXNFCAZK7,8RSv6Ylotq-
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/1wXAL0iveXDTqhZISADSaXqK5i6o-s,4y9VbAmN7,5jvj0/HDa4Qxijr-Uveiim71QBTDCo
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/1wXAL0iveXDTqhZISADSaXqK5i6o-s,4y9VbAmN7,5jvj0/HDa4Qxijr-Uveiim71QBTDCo/WS-VQBG-Lcm0OGVYokjorYyheVT9BjqVMXxTapEOZOB3K-
Not deleting: acd-rclone:/encrypted/rz0uXtHDGg5yClWF6O68pTW3/1zPhLTsqdg2QL7Ie,uK8qCuRkefjt9wderKAo8K5McB6z,

and lots. more

8

I suspect you have a mixture of encrypted and non-encrypted files. Run the script on the unencrypted acd remote and it will do more sensible stuff I think.

9

Just as another ‘alternative’ and not to replace the python script as it may be better and would be platform agnostic… but I use bash along with a rclone mount to accomplish this instead on linux. you’d have to run this multiple times depending on if you have empty directories nested in other empy directories (same as the python script).

  1. Mount the rclone somewhere. I’m mounting on robacd: on /data/Media and then running below

  2. Run this:

    MNT=/data/Media
    find $MNT -type d -empty | sed “s~$MNT~robacd:~” | tr ‘\n’ ‘\0’ | xargs -r -t -0 -n1 rclone -v rmdir

or with --dry-run

MNT=/data/Media
find $MNT -type d -empty | sed "s~$MNT~robacd:~" | tr '\n' '\0' | xargs -r -t -0 -n1 rclone --dry-run -v rmdir
10

Cunning using rclone rmdir to remove the directories but finding them with rclone mount.

11

Off topic but I work around the crypt ‘checksum’ issue with this method as well. Works pretty well. Use rsync with --checksum on the mount to determine what files are md5sum different but have the same ‘size’. Produce a list and feed that list into rclone copy --ignore-times.

Media1 is local
Media2 is crypt mount of robacd-crypt:

echo +---------------------------------------+
echo  Creating list off files with diff chksums
echo +---------------------------------------+

TMPFILE=/tmp/rclonefilelist.txt
rm -f $TMPFILE

rsync $OPT --inplace -rlvxi --checksum -n --out-format="%n" \
 --include="*.nfo" \
 --include="*/" \
 --exclude="*" \
 /data/Media1/Videos/ /data/Media2/Videos/ | egrep -v "/$|^sent|^total|^sending" | tee $TMPFILE

echo +---------------------------------------+
echo  Replace files with diff chksums from list
echo +---------------------------------------+
rclone \
  --transfers=8 --checkers=16 \
  --stats 5m \
  --bwlimit=10000 \
  --config="$RCLONECONF" \
  --include-from $TMPFILE \
 copy $OPT -v "/data/Media1/Videos/" "robacd-crypt:/Videos/" \
  --ignore-times 2>&1 | egrep -v "$CRYPTLOGFILTER"

rm -f $TMPFILE
12

Most ingeneous! I could I suppose add another flag into rclone to make it calculate the checksum even if the remote storage system didn’t have it - this would involve dowloading the file to check.

Or maybe a flag which cased rclone to compare the source and dest files byte by byte…

13

In the absence of a md5sum, that would be helpful. It would be even better if it could be stored in acd metadata on the first calculation so it wouldn’t need to be recalculated each time. Even without the metadata, it would be no worse than what I am doing. :slight_smile:

14

If rclone did it with a flag, at least it would be done in parallel. My rsync method is serial since rsync is serial.

15

Which flag do you prefer

--calculate-checksum-if-missing (suggestions for names welcome!) which would

  • download the whole file and calculate the checksum if the remote didn’t have the checksum type requested
  • this wouldn’t use any disk space it would calculate the checksum on the fly
  • it would also be useful for rclone md5sum and rclone sha1sum
  • it would have to download the whole file

--compare-whole-file which would

  • download the two files it is comparing until it found a difference, or right to the end
  • might not have to download the whole file
  • only useful for sync/copy/move

The first is more generally useful so I’d probably prefer that if I can think of a decent name for it :wink:

17

The more I think about it, if both remotes don’t have a chksum stored, it would speed things up to compare the files instead. Most people would use this tor crypt right now… I think the file compare might serve better in this use case… save some bytes.

18

But thinking even more, when I ran a whole library rsync chksum (many TBs, in 99% of case files were the same if size was the same except for things like xml or structured data like that. So I doubt it would save much in most cases. So even checksum would be good. If you prefer the checksum then that is enough I think.

What about --calc-missing-chksum?

19

I made an issue for the feature

Click subscribe if you want updates on it!

1 Like
20

I’ve made a new command rclone rmdirs which does what the above python script does in a more efficient way - remove empty directories (in a nested way)

You can try it in this beta: http://beta.rclone.org/v1.34-25-gf3365dd/ (uploaded in 15-30 mins)

Use it like this

rclone rmdirs --dry-run remote:

When happy then do

rclone rmdirs remote:
21

Oops messed up the build: try http://beta.rclone.org/v1.34-30-g943a093/ (uploaded in 15-30 mins)