Sync only subfolders, or treat folders as a whole item?

Help and Support
1

I'll explain my use case as I feel I don't know enough to ask concisely in one line:

I have contents on my server will have the following layout:
MEDIA
_Media-A
___media-a.mp4
___otherfile.png
___randomthing.txt

I move contents from this server to gdrive.

This works well if all subfiles have the same names of course, as they just get overwritten / updated, as rclone does.

My issue is when, instead of a new media-a.mp4, I have a media-a.avi.
rclone copy media/ gdrive:Media will just add the new file.
rclone sync media/ gdrive:Media will delete the other subdirectories of parent folder Media since they no longer exist on the local system (since I was moving them).

I was hoping the wise minds of experienced rclone users would know an easier solution to my dilemma. Another solution I guess, which just came to me, is I look into telling rclone to delete all folders on gdrive which have the same names as folders I wish to move, and then move my local folders to the gdrive when it's done.

Please, any feedback on this would be splendid!

2

I think I found a decent solution... someone wiser please correct me if this is a naive idea:

for d in */ ; do rclone sync "$d" "gdrive:media/$d" --dry-run ; done

minus dry run of course, and then I'll have to automate testing that sync was successful and delete the local files. Still, would be way more convenient if I could mix folder sync with move.

1 Like
3

Your explanation is pretty unclear... and you are not really stating what the issue is that you want to fix.

What is the relationship between "media-a.mp4" and "media-a.avi" ?
Are these the same file but renamed? Is that the problem here?

If so then --track-renames will be able to identify files as identical even if they are in different places or have different names.

Otherwise I think you need to re-state the problem because I am not understanding you.
I don't see what the point of the script you posted would be. This would sync each folder individually right? ... but why? What would that accomplish exactly?

Maybe it's my understanding here that is at fault rather than your explanation, but in either case we need to clear this up a bit I am to help you :slight_smile:

EDIT: I see you are responding. Heading out for a quick errand though. back in an hour or so.

1 Like
4

I am being unclear. This isn't helpful to those who wish to help, so I'll try again:

I have a seedbox that I move files from, over to my gdrive, because the seedbox is small.

Updates of content I'm following may come to the seedbox, which then needs to go to my gdrive.

If all the filenames are exact, the old ones on the gdrive get replaced.
If, however, one of these updates has a different file extension, it is added to the gdrive, causing duplicates because the old one wasn't removed.

This seems to do what I want:

for path in rque/movies/*/ ; do
     #Get rid of trailing / in folder path
    d="${path%"${path##*[!/]}"}"
     #Get rid of the last / and everything before it (get only folder name)
    d="${d##*/}"
    echo "Delete:"
    rclone delete "remote:Movies/$d" -v
    echo "Move:"
    rclone move "rque/movies/$d" "remote:Movies/$d" -v
done
find rque/movies/* -type d -empty -delete #delete empty folders

Just need to tell it to ignore empty dirs, and to stop if there are no folders to run on.
I think this is what I'll roll with, unless rclone has something built into it that will resolve my edge case issue. It's completely understandable if this is considered out of scope.

5

Hmm ok, that clarified it some - although I'm still not too clear on what the ideal way to handle these "files with changed extensions" would be. Are they identical aside from the filename? If so then --track-renames function would be useful to identify these and just perform a rename operation rather than having to reupload the file - thus both being efficient and solving the problem of leaving an old outdated file behind.

Otherwise it seems like you could probably get some use out of filtering here, so I would advice looking into that:
https://rclone.org/filtering/
I think this would save you from doing most, if not all of that scripting. Not that scripting is a problem, but it's generally preferable and less likely to have bugs to use well-tested code where possible.

This could allow you to perform a sync, but only on spesific folders. (and much much more, as the filtering system is very flexible).
If these folders are static - then you can just define them in a simple filter-file.

If they are dynamic and change all the time, then we will have to generate them on demand.
Thankfully rclone can do most of that work for you.

Let's say for example we are operating on subfolders of gdrive:/media as sompared to sufolders in /home/media ...
we could then run
rclone lsf /home/media --dirs-only > outputfile.txt
(now we should have a list of all the local folders, not including subfolders as -R, recursive is not default on lsf)
Then you can feed this into...
rclone sync /home/media gdrive/media --files-from outputfile.txt
Since filters apply to both sides - this should sync only the folders that exist locally, and ignore the others entirely (ie. not delete them).

This seems to be what you want to do here right? Scripting this would just need 2 lines, so it simplifies things greatly. Note that this is a rough example and it's likely that we may need to tweak the format somewhat to get exactly the result we want here, but I can probably assist you in that if you get stuck.

Good practice is to use --dry-run while testing such things until you are sure it's working as intended. Especially on any sync commands that could result in files being unintentionally deleted.

Does this seem like a solution to you, or have I missed the mark?

EDIT: Changed from --include-from to --files-from . These are subtly, but importantly different and I sometimes confuse them myself. I recommend understanding the difference if you use either. Feel free to ask for such clarifications if needed.

1 Like
6

There are many ways to deal with empty dirs.
rclone rmdirs can remove all empty (and only empty) folders for you easily if you want that.
Also rclone will not copy empty folders by default. You can however force it to with --create-empty-src-dirs

The second I can provide a bash example for.
This example shows more than you need, but I just wanted to show an example with context:

if [ -z "$(ls -A /path/to/your/gdrive/mount)" ]; then
   echo "Empty" && sleep 900
else
   echo "Not Empty" && PUT YOUR FULL RCLONE UPLOAD COMMAND HERE && rm -f /tmp/uploadscript.lock && exit
fi

The relevant part here would be the if statement itself - which you can use to test for if a directory is empty or not - and proceed accordingly. I am by no means a bash guru, but I know some basics.

7

Thanks! That filtering looks like exactly what I was looking for!

I won't need to loop through folder names to send individually to rclone if rclone can already work from a list of the folders!

Additionally, you're absolutely right that working off my own scripts could easily backfire later. I had to do so much research to get handling the empty strings and empty dirs, which I had just finished when you replied. I'm grateful that I got the mental exercise, but even more grateful that rclone can more effectively handle my needs.

I'll work on exploring filtering, now! Thanks so much!

8

Nice to hear that may be the solution :slight_smile:
Knowing what is possible in the filtering system is well worth it as you can effectively do almost anything in terms of automation based on that.

If you need any help on the technicalities - let me know and I will try to assist. I am not an expert on all parts of this, but I have done many similar things to this myself with the filtering system so I have some experience to work with and can probably spare you some of the hassle in finding what you need.

actually, you probably want to use --files-from rather than --include-from in the second command. BOth will work, but the latter will be faster + not have issues when it comes to special characers (as they may otherwise be interpreted as special GLOB patter-characters )

I recommend skimming over
--include / --exclude
--include-from // exclude-from
--files-from (not filtering rules but just a list to work from)
the basic of the GLOB pattern system - a simplified version of REGEX that is popular from Linux, grep ect.

1 Like
9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.