Can 'rclone move' exclude moving files?

Help and Support
1

I have a folder for downloads, and I run the following commant every 1 minute to move the completed downloads
rclone move /path/of/downloads/ remote:/path/of/completed/ --exclude=**.downloading
But it seems that if the last moving havn't been finished before the next minute, the moving will be repeated again and again.
I can set the time longe enough larger than 1 minute, but I'll get the files late in the remote drive.
Can rclone move exclude moving files? Or can rclone move monite a folder?

2

It's super helpful to use our help and support template.

If you exclude a file, it won't be moved.

If you want more help, use the template and share the details.

3

I know the effect of 'exclude', but it's not my question.

I mean the files are downloaded in a 'downloads' folder, and I run 'rclone move' every 1 minute to move the completed ones to a remote drive.
But if a movement of a file can't be finished in 1 minute, the file will be moved again and again, during the command repeating.
Extending the interval between running commands for more greater than 1 minute can avoid repeated movement, but I don't want to prolong the waiting time. So I want to know how to run 'rclone move' with an exclution of a file which is already in moving.

The title is ambiguous. I want to make it concise. There is no error when runnning the command, but doing the same movement repeatedly will slow down the computer, so I want to improve the command.

4

before my script runs rclone, it creates a filesystem snapshot and uses that as the source.
tho once per minute might be a bit much for that approach.

might try running a rclone mount and use a sync tool such as syncthing.

5

See comment on using the help template as that helps figure out what you are asking as it gives all the details, log files, command, etc.

Does that mean local file system to a remote?

A file being written to will error out and that's evident in the log file. If you repeat it and share a debug log, it's easy to how/explain what happens.

That is definitely true.

Be great to know what OS to help figure out how to answer as that's included in the help template as part of the rclone version output...

6

It's hard to decide the interval, in my case, for most files, once per minute is a comfortable time to display them on the remote drive in time. But I have to be careful of the files of large size. So I think it is the best if rclone has an option to exclude the files which is in moving.

The question is the download tool I use, write the downloading temporary files (like *.downloading), in the same folder, can syncthing exclude them?

7

What is the problem you are having with rclone?

I have a folder for downloads, and I run the following command every 1 minute to move the completed downloads
rclone move /path/of/downloads/ remote:/path/of/completed/ --exclude=**.downloading
But it seems that if the last moving havn't been finished before the next minute, the moving will be repeated again and again.
I can set the time longe enough larger than 1 minute, but I'll get the files late in the remote drive.
Can rclone move exclude moving files? Or can rclone move monite a folder?

Run the command 'rclone version' and share the full output of the command.

~# rclone version
rclone v1.61.1

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.15.0-1029-oracle (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move /path/of/downloads/ remote:/path/of/completed/ --exclude=**.downloading
### and I set this command to run every 1 minute with crontab.

The rclone config contents with secrets removed.

[FebTh]
type = onedrive
client_id = ###id###
client_secret = ###secret###
token = {"access_token":"..."}
drive_id = ###id###
drive_type = business

A log from the command with the -vv flag

log.txt (131.0 KB)

This may run out my CPU and memory in minutes, and I have to restart and stop running the command, so I don't try it often.

8

might try a rclone two-step, something like

#. get a list of fles to be moved
rclone lsf /path/of/downloads/ --format=p --absolute --exclude='*.downloading' > list.txt
#. move the remaining files
rclone move /path/of/downloads/ remote:/path/of/completed/ --include-from=list.txt

edit: i re-edit the script, to simplify it

9

Is this the same effect as ‘--exclude=**.downloading’?
The *.downloading file is generated by the download tool, and won't be moved.

If file F start being moved at time T, and the movement can't be finished at time T+1, then the file F will start being moved again in another movement at time T+1. And I want exclude file F to avoid movements after the first time.

10

yes

then run code in an endless loop. so only one instance of rclone running.

11

I think you might be overcomplicating it a bit.

If your goal is to move items to the cloud, just make sure you aren't executing the script again if it's already running.

I have a very simple one here:

If the script is running, it won't execute again until it is finished.

Files being copied at the time, rclone notices that and won't move them as it'll notice they are changing.

I've never used an exclude, snapshot or anything else and never hit any issues.

12

It's late here, I'll try it tomorrow, thank you!

13

It is really a simple and effective method to judge whether the script will be executed next time according to the last run of the script. I run the following two commands every minute. It work very well now. Thank you!

if [[ $(pidof -x "$(basename "$0")" -o %PPID) ]]; then
echo "Already running, exiting..."; exit 1; fi
rclone move /path/of/downloads/ remote:/finished/ --exclude=**.downloading
14

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.