Copying/syncing only new folders/files possible?

Help and Support
1

So, I’m new to seedboxes, I am currently using one for a few torrents.

I discovered rClone and it’s perfect for syncing with pCloud for me, however I have one issue.

I want to either only sync/copy new files/folders, or skip copying/syncing a folder if the name of that folder already exists, even if the contents are different, or only sync/copy files/folders that are new since the last sync/copy. Is there a way to do this?

This is so my torrent can directly upload to the cloud, then I can edit it while its there, and then when I next sync/copy all the folders, it won’t mess with the ones I’ve just changed. If you know another way to do this, please let me know.

I’m new with SSH and Unix commands, so I’d need some serious guidance here.

Thanks,
Zen

EDITS: Grammar and structure

2

there’s a “–max-age 2d” flag . Meaning only upload files that are newer that 2 days. Or a “–ignore-existing” if you’re trying not to clobber existing files on the destination you’ve modified.

If you’re sending torrents though you may find yourself uploading partial downloaded files and then finding they won’t be overwritten when they are complete because of the ignore existing flag.

Maybe I don’t understand what you were trying to do though.

3

Would --ignore-existing ignore existing folders, or the entire folder contents?

Because, if I rearranged files inside the folders and then I did --ignore-existing, and the --ignore-existing flag only recognised if the folder was exactly the same, it’d affect the folder by copying it over again since although the folder exists it’s not exactly the same. Am I correct with that?

4

You’re correct. It looks at files. If you move things it’ll recopy.

5

I see, I guess --max-age might be the only way to do this, unless I'm missing something. I really appreciate your help by the way, thank you!

I also saw this thread on the exact same subject:

But I honestly didn't understand the answers.

6

Frankly you’re best off having your torrent program fire a script off at completion and just copy/move that file to rclone.

7

That doesn’t really solve the problem. To try and clear this up, I’ll do my best to explain:

In my case, I get very messy torrents (which each consist of one folder, with either many files or many subfolders inside of them) directly on a seedbox, and then use rClone to sync it to my preferred cloud system. Then, I want to be able to reorganise the contents of each single folder of the torrent in my cloud system to organise them a bit. Only problem is, next time I copy it will detect a change and just re-copy.

The max age thing makes sense I guess, and I can manage by only syncing every 5 days and doing --max-age 5. I will only be syncing completed torrents by the way.

If there’s a better way you know of, please let me know.

8

If you're trying to only copy new torrents you'd want to copy way more frequently than the days defined. That flag tells rclone to only sync torrents that are newer than X period.

It's 5d btw...

Still think it's best to get it at the source when the torrent completes though be filing a script when the torrent is Moved to the seedbox.

9

Alright, if that's what needs to be done then I suppose I'll do it. Do you have any idea what the replies on this thread are referring to?

10

Particularly the final two replies.

11

Yes. They want to essentially do what you want. You could use a exclude list as well. But you’re going to have to do a good amount of scripting.

12

I am absolutely useless with scripting, I have no idea how or where to start.

13

You could do what the last comment said to do too. I just don’t like dealing with timing things like that. if you change something on your destination within the interval that those dates define then you’re going to screw up your destination with those new files again.

14

I am useless with scripting and I don’t even understand how to do the last reply, honestly.

15

The exclude one sounds perfect but there’s absolutely no way I can script that myself. Do you know anyone who could script it?

16 
cd $SOURCE_DIR
touch $HOME/.rclone.timestamp.new
find . -type f -newer $HOME/.rclone.timestamp -exec rclone .... <stuff goes here>
mv $HOME/.rclone.timestamp.new $HOME/.rclone.timestamp

Are you asking how to make this work in your environment?

17

The only comment i’ll make about that script is if a rclone transfer fails, it’ll ignore it on the next run as that assumes all success between time stamps. The way you really should be doing this is creating a include file based on the contents of the directory and then that script combing the rclone log file to determine success/failure and removing those files from the include file. That include file then serves as a queue.

18

Yes, I also don’t know what goes in the “clever stuff goes here”, I’m assuming just my rclone command. I’m new to all this, but I am a decently quick learner. Would you be able to help me out?

19

I see what you mean, but I’d have no idea how to do it.

20

Yes, your rclone command but the line needs to end with:
{} ;
after your rclone command. You’ll also need to provide the $SOURCE_DIR path

You can also reply with

  1. Your rclone command
  2. The path to where your files are located

and I’ll write something for you.