Sync directory for a specific timeframe?

Help and Support
1

Hi,

I want to sync a folder for a specific time frame. For example run the sync command in the background (screen or something) and kill it as soon as it shouldn’t sync anymore. Is something like that possible with rclone?

Thanks!

2

If you're looking for continuous syncing, it doesn't do it yet. There are a few enhancement requests for it already.

3

Do you mean that you want to rclone to run for 1 hour (say) or until the sync is finished, whichever comes the sooner?

If you do then that would be relatively easy to implement and would be a useful feature.

4

Thanks for your answer.

Let’s imagine I have a script that generates several hundred files within a dynamic timeframe. These files are generated within a directory and I want these files to be synced to my object storage during the period the script runs.

Just to explain it a bit simpler: Start my script that generates files -> Start rclone to sync all files within that directory -> (dynamic time range, can be 1min but also can be several hours) -> stop my script -> stop rclone sync of that directory (or it automatically stops after everything was successfully uploaded)

rclone should be stopped as soon as all files are synced to the object store, but that is not a must have, I could also stop it X mins after my script was stopped simply to be sure that everything got uploaded.

I hope you understood what I want, I am not a native speaker so I am sorry.

Thanks
Nik

5

I see. Your plan at the moment won’t work because rclone will finish and exit when it has done the first sync. The first sync is likely empty.

I could add a new flag to rclone say --continuous which runs the sync/copy//move over and over again until rclone is killed - that would work. Or you could write a small shell script to wrap rclone to do that.

6

The --continuous option would be awesome! I think a lot of people could make use of such a functionality.

7

The only caveat is that it will be continuously scanning and if you select something like chksum, for example, it will be re-scanning files a hell of a lot. Its not a bad idea to have continuous but people should know that it isn’t using something like ‘i-notify’ to tell rclone of changes. It would simply be scanning I suppose. Or did I misunderstand?

Perhaps an scan-interval would be good along with --continuous.

8

good idea! It would be relatively easy to implement but I’m unsure as to the exact user interface. Can you think of any commands which repeat themselves?

9

Nick, rather than have a set of commands to repeat, perhaps we create a blacklist of commands that should never repeat.

Either way, this is a whitelist of ones that I could see some use case to repeat. Some are obviously more important than others but I tried to imagine a reason to not let some of them repeat while considering scripting reasons why someone may want some of these to repeat (like cat or ls for example redirecting output to a file to keep a list of files somewhere up to date).

whitelist
cat Concatenates any files and sends them to stdout.
check Checks the files in the source and destination match.
cleanup Clean up the remote if possible
config Enter an interactive configuration session.
copy Copy files from source to dest, skipping already copied
copyto Copy files from source to dest, skipping already copied
delete Remove the contents of path.
genautocomplete Output bash completion script for rclone.
ls List all the objects in the the path with size and path.
lsd List all directories/containers/buckets in the the path.
lsl List all the objects path with modification time, size and path.
md5sum Produces an md5sum file for all the objects in the path.
mkdir Make the path if it doesn’t already exist.
move Move files from source to dest.
moveto Move file or directory from source to dest.
purge Remove the path and all of its contents.
rmdir Remove the path if empty.
rmdirs Remove any empty directoryies under the path.
sha1sum Produces an sha1sum file for all the objects in the path.
size Prints the total size and number of objects in remote:path.
sync Make source and dest identical, modifying destination only.

Blacklist:
authorize Remote authorization.
mount Mount the remote as a mountpoint.
version Show the version number.
gendocs Output markdown docs for rclone to the directory supplied.
dedupe Interactively find duplicate files delete/rename them.
listremotes List all the remotes in the config file.

Most important whitelist: copy, copyto, move, moveto, sync, rmdirs

Also, i’d do away with having two parameters and simply have a --run-interval and make a zero (default) be the OFF setting. Anything above zero could enable it for that duration. Less switches.

10

Just to add a “me too” for being able to schedule rclone. My use-case is that in the UK we have ridiculously low upstream bandwidth, and, if that bandwidth (10mb) is saturated then it also kills downstream bandwidth (100mbs). Or, too put it another way, I get it in the neck from my wife and 4 kids when “the internet doesn’t work” :-).

Being able to say “run between 11.00pm - 6.00am” and “10.00am - 4.00pm” would be awesome.

11

If @ncw puts in a continuous sync, you could do this pretty easily on linux with the ‘timeout’ command and cron. Timeout allows you to specify how long to run for.

cron::
00 10 * * * timeout -k 5 6h rclone --run-interval 10m sync …
00 23 * * * timeout -k 5 7h rclone --run-interval 10m sync …

Even without the continous command, you could use something like this:

00 10 * * * timeout -k 5 6h watch -n 600 rclone sync …
00 23 * * * timeout -k 5 7h watch -n 600 rclone sync …

Both of these say to run rclone every 10 minutes and terminate them at 6/7 hours and optionally send a KILL if the command doesn’t stop after 5 seconds after timeout signal.

12

Linux is just the gift that keeps on giving :-). I love the Unix philosophy of ‘do one thing and do it well’.

EDIT: Fantastic - it is already installed on unRAID.

13

Maybe I am wrong, but can’t be what the original poster wants be achieved by using something along the following shell script that is started by crond e.g. every 15 minutes or so:

#!/bin/bash

LOCKFILE="/tmp/rclone-upload.lock"
LOGFILE="/tmp/rclone-log"
FROM="<SRC FOLDER>"
TO="<REMOTE FOLDER>"

(
if flock -n 9 ; then
        echo ">>> Starting RCLONE upload script: " `date` >> $LOGFILE 
	rclone -v --log-file=$LOGFILE sync $FROM $TO
	echo "<<< RCLONE upload script has finished: " `date` >> $LOGFILE
else 
	echo "***************************************************************" >> $LOGFILE
	echo "*** ...rclone upload script already running... exiting now. ***" >> $LOGFILE
	echo "***************************************************************" >> $LOGFILE
	echo "" >> $LOGFILE
	exit 1;
fi
	
) 9>$LOCKFILE
1 Like
14

yes it can. Thing is that go can run on many platforms so having some of these things in the program helps be platform agnostic. But you are 100% correct.

15

rclone has such a list built in already to control what the --retries flag does. You have to look at the flags to the Run command in the code, eg cat vs sync - see definition of Run the first parameter bening whether to allow retries, and the second whether to print transfer stats. This agrees mostly with your is.

That sounds like a sensible plan. I’d probably want a --run-cycles flag which when 0 (the default) would run forever, but if non zero you could limit the number of cycles.

1 Like
16

I think this issue is of interest here: https://github.com/ncw/rclone/issues/221

17

Sorry if I misunderstand you here - but actually having a daemonized sync/copy being fed by inotify sounds like a rather good thing to me - I assume you are not arguing against the idea, but just saying that it’s not what --continuous would implement?

Even for a possible daemon mode it would be nice to have “allowed times” or so to help with the case where you need your upstream during office hours (or family-browsing-the-internet times). And as you said - not having to resort to crond magic for this is a huge win in portability…

18

No. not against inotify at all. Just don’t want people setting this to too high an interval and affecting their cloud storage providers with massive and continuous api calls since it doesn’t currently use inotify.

19

Hi,

I made some inotify-wait scripts so as to rclone move/copy files/folders when new files in some source dirs.
Here’s an example:

#!/bin/bash

SDIR='/source/dir'
TDIR='/target/dir'
CMD='mv'

inotifywait -m -r -e close_write "$SDIR" |
    while read file; do
        mv "$SDIR" "$TDIR"
    done

This is used to move a source dir to a target dir on an rclone encrypted mount. But you can use some rclone XXX commands for the CMD.

20

Just used your script, its works fantastically. Thanks for sharing!

One small thing that stopped it working with cron, for me atleast, was that it needed a path line below the #!/bin/bash line…

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/FOLDER

Replacing the FOLDER text with the full folder directory your script is in made it work for cron.