Is it possible to make rclone move to wait x minutes before removing?

[vezel (Gustav Persson Svensson)]

Hi

I'm using google drive as remote. I want to move files to my google drive remote, but before removing on the local I would like to wait for x minutes (so mergerfs has time to find the new file on remote). Is this possible?

I made a bash script that uses rclone copy, then sleeps 2 minutes and uses rm -rf to remove the local files. However I'm having problems when sonarr/radarr moves files to my local directory while the bash script is sleeping, they will also be removed even though they are not in the remote.

[timevortex (Parth Shukla)]

I can't be sure of your exact setup (so a few assumptions here) but you should be able to do something like this in bash:

ls -1 > /tmp/rclone_list_of_files.txt
rclone copy --files-from /tmp/rclone_list_of_files.txt [your other arugments go here]
sleep 120
rm -rf `cat /tmp/rclone_list_of_files.txt`

This way you build a list of files to transfer and only delete those files when you're done sleeping.

[Animosity022]

You can just tune down the default polling to 10 seconds and use rclone move instead. It'll pick up the change in 10 seconds and pretty much, nothing will miss that too much.

[vezel (Gustav Persson Svensson)]

Thanks, this sounds like the way to go. I tried it with this command:
/usr/bin/rclone copy --files-from list_of_files.txt /home/appbox/Media/local crypt: --drive-stop-on-upload-limit --fast-list -vvv --dry-run --verbose

The file contains for example:
"Movie/Thor The Dark World (2013)/Thor The Dark World (2013) Bluray-1080p.mp4"

But when I run it I get:
2021/03/11 13:13:52 DEBUG : Movie: Excluded

Any idea what's causing this?

[timevortex (Parth Shukla)]

Very interesting. I've never used this functionality personally (but plan to soon) and I can replicate your error quite easily with some fake files and very simple --files-from:

timevortex@gallifrey:~/rclone-test$ tree
.
├── dst
└── src
    ├── root_file_1
    ├── sub folder 1
    │   ├── file10.txt
    │   └── file11.txt
    └── sub folder 2
        ├── file20.txt
        ├── file21.txt
        └── file22.txt

4 directories, 6 files
timevortex@gallifrey:~/rclone-test$ 

I started by trying to copy all the files but that didn't work as well with nothing in the debug logs helping:

timevortex@gallifrey:~/rclone-test/src$ find . -type f -printf "\"%p\"\n" > /tmp/file_list_for_rclone.txt
timevortex@gallifrey:~/rclone-test/src$ cat /tmp/file_list_for_rclone.txt 
"./root_file_1"
"./sub folder 1/file11.txt"
"./sub folder 1/file10.txt"
"./sub folder 2/file20.txt"
"./sub folder 2/file21.txt"
"./sub folder 2/file22.txt"
timevortex@gallifrey:~/rclone-test/src$ rclone copy ~/rclone-test/src/ ~/rclone-test/dst/ --files-from /tmp/file_list_for_rclone.txt -vvv --dry-run --dump filters
--- start filters ---
--- File filter rules ---
--- Directory filter rules ---
--- end filters ---
2021/03/11 15:16:22 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "/home/timevortex/rclone-test/src/" "/home/timevortex/rclone-test/dst/" "--files-from" "/tmp/file_list_for_rclone.txt" "-vvv" "--dry-run" "--dump" "filters"]
2021/03/11 15:16:22 DEBUG : Creating backend with remote "/home/timevortex/rclone-test/src/"
2021/03/11 15:16:22 DEBUG : Using config file from "/home/timevortex/.config/rclone/rclone.conf"
2021/03/11 15:16:22 DEBUG : Creating backend with remote "/home/timevortex/rclone-test/dst/"
2021/03/11 15:16:22 DEBUG : root_file_1: Excluded
2021/03/11 15:16:22 DEBUG : sub folder 1: Excluded
2021/03/11 15:16:22 DEBUG : sub folder 2: Excluded
2021/03/11 15:16:22 DEBUG : Local file system at /home/timevortex/rclone-test/dst/: Waiting for checks to finish
2021/03/11 15:16:22 DEBUG : Local file system at /home/timevortex/rclone-test/dst/: Waiting for transfers to finish
2021/03/11 15:16:22 INFO  : There was nothing to transfer
2021/03/11 15:16:22 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

2021/03/11 15:16:22 DEBUG : 2 go routines active
timevortex@gallifrey:~/rclone-test/src$ 

I also tried to only copy one file and that didn't work:

timevortex@gallifrey:~/rclone-test/src$ cat /tmp/file_list_for_rclone.txt 
"root_file_1"
timevortex@gallifrey:~/rclone-test/src$ rclone copy ~/rclone-test/src/ ~/rclone-test/dst/ --files-from /tmp/file_list_for_rclone.txt -vvv --dry-run --dump filters
--- start filters ---
--- File filter rules ---
--- Directory filter rules ---
--- end filters ---
2021/03/11 15:10:05 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "/home/timevortex/rclone-test/src/" "/home/timevortex/rclone-test/dst/" "--files-from" "/tmp/file_list_for_rclone.txt" "-vvv" "--dry-run" "--dump" "filters"]
2021/03/11 15:10:05 DEBUG : Creating backend with remote "/home/timevortex/rclone-test/src/"
2021/03/11 15:10:05 DEBUG : Using config file from "/home/timevortex/.config/rclone/rclone.conf"
2021/03/11 15:10:05 DEBUG : Creating backend with remote "/home/timevortex/rclone-test/dst/"
2021/03/11 15:10:05 DEBUG : root_file_1: Excluded
2021/03/11 15:10:05 DEBUG : sub folder 1: Excluded
2021/03/11 15:10:05 DEBUG : sub folder 2: Excluded
2021/03/11 15:10:05 DEBUG : Local file system at /home/timevortex/rclone-test/dst/: Waiting for checks to finish
2021/03/11 15:10:05 DEBUG : Local file system at /home/timevortex/rclone-test/dst/: Waiting for transfers to finish
2021/03/11 15:10:05 INFO  : There was nothing to transfer
2021/03/11 15:10:05 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

2021/03/11 15:10:05 DEBUG : 2 go routines active
timevortex@gallifrey:~/rclone-test/src$

I feel like I'm missing something obvious

[vezel (Gustav Persson Svensson)]

Yea I get the exact same result.. I'm not sure why it's excluding the file/directories. Do you know if there is any examples on how to use --files-from? I tried to look for a good example but didn't find any

[timevortex (Parth Shukla)]

@vezel

I tried to look for a good example but didn't find any

I found this one but I can't make heads or tails of it at the moment:

Rclone download list of files from txt file?

You need to set up a :http: as the source directory and in your files it should be just the file names relative to the directory in the --http-url. So rclone copy --http-url https://example.com :http: /path/to/dest --files-from files.txt Where files.txt looks like this file.zip dir/file2.zip dir/subdir/file3.zip

[timevortex (Parth Shukla)]

Figure it out! The quotes were the problem!

timevortex@gallifrey:~/rclone-test/src$ find . -type f | sed 's/^\.\///g' > /tmp/file_list_for_rclone.txt 
timevortex@gallifrey:~/rclone-test/src$ cat /tmp/file_list_for_rclone.txt 
root_file_1
sub folder 1/file11.txt
sub folder 1/file10.txt
sub folder 2/file20.txt
sub folder 2/file21.txt
sub folder 2/file22.txt
timevortex@gallifrey:~/rclone-test/src$ rclone copy ~/rclone-test/src/ ~/rclone-test/dst/ --files-from /tmp/file_list_for_rclone.txt -vvv --dry-run --dump filters
--- start filters ---
--- File filter rules ---
--- Directory filter rules ---
--- end filters ---
2021/03/11 15:27:32 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "/home/timevortex/rclone-test/src/" "/home/timevortex/rclone-test/dst/" "--files-from" "/tmp/file_list_for_rclone.txt" "-vvv" "--dry-run" "--dump" "filters"]
2021/03/11 15:27:32 DEBUG : Creating backend with remote "/home/timevortex/rclone-test/src/"
2021/03/11 15:27:32 DEBUG : Using config file from "/home/timevortex/.config/rclone/rclone.conf"
2021/03/11 15:27:32 DEBUG : Creating backend with remote "/home/timevortex/rclone-test/dst/"
2021/03/11 15:27:32 NOTICE: root_file_1: Skipped copy as --dry-run is set
2021/03/11 15:27:32 NOTICE: sub folder 2/file20.txt: Skipped copy as --dry-run is set
2021/03/11 15:27:32 NOTICE: sub folder 2/file21.txt: Skipped copy as --dry-run is set
2021/03/11 15:27:32 NOTICE: sub folder 1/file11.txt: Skipped copy as --dry-run is set
2021/03/11 15:27:32 DEBUG : Local file system at /home/timevortex/rclone-test/dst/: Waiting for checks to finish
2021/03/11 15:27:32 NOTICE: sub folder 2/file22.txt: Skipped copy as --dry-run is set
2021/03/11 15:27:32 NOTICE: sub folder 1/file10.txt: Skipped copy as --dry-run is set
2021/03/11 15:27:32 DEBUG : Local file system at /home/timevortex/rclone-test/dst/: Waiting for transfers to finish
2021/03/11 15:27:32 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Transferred:            6 / 6, 100%
Elapsed time:         0.0s

2021/03/11 15:27:32 DEBUG : 2 go routines active
timevortex@gallifrey:~/rclone-test/src$ 

[vezel (Gustav Persson Svensson)]

Ok so we are not supposed to use quotes? But it can handle the spaces in the text file?

[timevortex (Parth Shukla)]

@vezel

But it can handle the spaces in the text file?

Yup looks like it - you can see in my test I have folders with spaces.

[vezel (Gustav Persson Svensson)]

Yes, just tried it and works for me aswell. Thanks alot!

[timevortex (Parth Shukla)]

Happy to help! This has helped me as well as I probably would've run across the same problem when I start using --files-from flag soon