"rclone rc sync/copy " filters flags ignored

[hyrockbass]

Hello Everbody,

What is the problem you are having with rclone?

When I use the rclone rc sync/copy command with --exclude="*", the exclude filter is ignored.

What is your rclone version (output from rclone version)

rclone v1.49.1

• os/arch: linux/amd64
• go version: go1.12.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

NAS Synology : DSM 6.2.2-24922 Update 3

Which cloud storage system are you using? (eg Google Drive)

PCloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

1. first command
   #> rclone rcd --rc-user=test --rc-pass=test

2. next :
   #> rclone rc sync/copy --exclude="*" --log-file="/tmp/logs/job_copy_photo.log" -v --exclude-if-present="__ignore" --skip-links srcFs="/volume1/photo" dstFs="pcloud_test:photo" --rc-user="test" --rc-pass="test" _async=true --dry-run

output result:
{
"jobid": 16
}

--> Problem : the --exclude="*" is ignored, the dry-run command is ignored,.. all --xxxxx flags are ignored, and the transfer start from srcFs to dstFS

Whereas without "rc" mode, it works and all --xxxxxx flags are well interpreted
#> rclone copy --exclude "*" --log-file="/tmp/logs/job_copy_photo.log" -vv --exclude-if-present="__ignore" "/volume1/photo" "pcloud_test:photo" --rc-user="test" --rc-pass="test" --dry-run

Note that in this example all files are ignored due to --exclude "*". It's just for example to simplify the description of bug.

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

This is the logs of rc sync/copy command :
2019/09/11 23:25:49 DEBUG : rclone: Version "v1.49.1" starting with parameters ["rclone" "rc" "sync/copy" "--exclude=" "--log-file=/tmp/logs/job_copy_photo.log" "-vv" "--exclude-if-present=__ignore" "--skip-links" "srcFs=/volume1/photo" "dstFs=pcloud_test:photo" "--rc-user=test" "--rc-pass=test" "_async=true" "--dry-run"]
2019/09/11 23:25:49 DEBUG : 5 go routines active
2019/09/11 23:25:49 DEBUG : rclone: Version "v1.49.1" finishing with parameters ["rclone" "rc" "sync/copy" "--exclude=" "--log-file=/tmp/logs/job_copy_photo.log" "-vv" "--exclude-if-present=__ignore" "--skip-links" "srcFs=/volume1/photo" "dstFs=pcloud_test:photo" "--rc-user=test" "--rc-pass=test" "_async=true" "--dry-run"]

Have you an idea please ?

Thank you in advance.

Regards
Sam

[ncw (Nick Craig-Wood)]

With the rc you need to set the flags you need using a separate rc command.

https://rclone.org/rc/#options/set

This isn't particularly easy to use!

You can't pass the flags in on the rclone rc command.

[hyrockbass]

Thank you ncw for your reactivity.

I succeding in having more options when I execute the "rc options/get" command

In my case, I would like to add --exclude="*" filter, so I test this command line

first command :

rclone rc sync/copy srcFs="/volume1/photo" dstFs="pcloud_test:photo" --rc-user="test" --rc-pass="test" _async=true

and the second one

rclone rc options/set --json '{"filter": {"ExcludeRule": "*"}}' --rc-user="test" --rc-pass="test"

the second one raise an error

2019/09/12 00:39:08 Failed to rc: Failed to read rc response: 500 Internal Server Error: {
"error": "failed to write options from block "filter": Reshape failed to Unmarshal: json: cannot unmarshal string into Go struct field Opt.ExcludeRule of type []string",
"input": {
"filter": {
"ExcludeRule": "*"
}
},
"path": "options/set",
"status": 500
}

Question 1 : is it the right order to execute the command ?

• "sync/copy" and next "options/set"
• or "options/set" and next "sync/copy" ?

Question 2 : I don't know what is the right syntax

Question 3 : If i have several jobs which are running, how can I have different options for each jobs ?

Thank you.
Sam

[ncw (Nick Craig-Wood)]

and the second one

rclone rc options/set --json '{"filter": {"ExcludeRule": "*"}}' --rc-user="test" --rc-pass="test"

the second one raise an error

It is expecting a list of strings, so this should work I think

rclone rc options/set --json '{"filter": {"ExcludeRule": ["*"]}}' --rc-user="test" --rc-pass="test"

You can use options/set with all the options in, you don't have to do them individually.

Question 1 : is it the right order to execute the command ?

• "sync/copy" and next "options/set"
• or "options/set" and next "sync/copy" ?

"options/set" and then "sync/copy" ?

Question 3 : If i have several jobs which are running, how can I have different options for each jobs ?

That isn't possible at the moment...

With filters in particular there is only one set of filters for the entire programs.

This is something which may get fixed at some point!

[hyrockbass]

Well done!! It Works!! Thank you!!.

For the last point :
**Question 3** : If i have several jobs which are running, how can I have different options for each jobs ?

Maybe I can do like this

• define "option/set" for the job 1
• execute job 1 with sync/copy (running in background)
• define "option/set" for the job 2
• execute job 2 with sync/copy (running in background)
• etc...

I have to check it.

Sam

[ncw (Nick Craig-Wood)]

I have to check it.

I think that will work for some options, but the filter options is a global so if you change it while another sync is running then it will change for that sync too.

This is fixable, it just shows rclone's heritage as a command line program rather than an API driven program

[hyrockbass]

Well, I just checked it before your response and I confirm

My example :
1. define "option/set" for the job 1 (exclude nothing in filter)
2. execute job 1 with sync/copy (running in background)
3. sleep 6
4. define "option/set" for the job 2 (with filter exclude everything *)
5. execute job 2 with sync/copy (running in background)

Result :
the job 2 finished very quickly
the job 1 was still running but after 1 minute (approximately) the job 1 stopped

(I suggest the job 1 is not stopped directly beacuse of check caching or something like that)

As a result, if i well understood, I have two choice :

1. execute my jobs without a remote control and just with "rclone copy"
   --> disavantage : I cannot have information on my running jobs (with job/status)
2. Or wait for a fix (i don't know what is the delay)

Thank you ncw

[ncw (Nick Craig-Wood)]

execute my jobs without a remote control and just with "rclone copy"
--> disavantage : I cannot have information on my running jobs (with job/status)

You can run rclone copy with the --rc command which would give you status. You'd need to pick a different --rc-addr for each rclone though.

Or wait for a fix (i don't know what is the delay)

Re-architecting rclone! It is a big job and it isn't on the roadmap yet though I am thinking about it!

[hyrockbass]

Hello Nick,
Thank you for your answer.

You can run rclone copy with the --rc command which would give you status. You'd need to pick a different --rc-addr for each rclone though.

Good point. Unfortunately, when I run rclone copy with --rc command, there is no "job id" created .

Example :

1. rclone copy --rc --rc-user="test" --rc-pass="test" /volume1/photo pcloud_test:photo --dry-run &
2. rclone rc job/list

Result : no job id returned for the rclone copy --rc command. so i'm not able to use job/status by specifing a jobid

Maybe i forgot something ? note that _async=true param is not possible with rclone copy --rc.

Thank you in advance.
Sam

[ncw (Nick Craig-Wood)]

Good point. Unfortunately, when I run rclone copy with --rc command, there is no "job id" created .

You'd have to read the status of the command by the exit code of rclone in this case.