Remote control api - include files with wild card

What is the problem you are having with rclone?

When using remote control api to copy files from Oracle Cloud Storage to Azure Cloud Storage, the includefrom parameter does not seem to work?

The goal is to just copy CSV files from OCI to Azure. When I run the curl statement CSV and JSON files are copied.

curl --location -g --request POST 'https://rclone-poc.wwt.com/sync/copy?srcFs=oci:/RCLONE/RcloneApiTest&opt={"_filter" : {"IncludeRule":"*.csv"}}&dstFs=devoracldbiccc:rcloneapitest' \

--header 'Authorization: Basic 

Run the command 'rclone version' and share the full output of the command.

{
"arch": "amd64",
"decomposed": [
1,
60,
0
],
"goTags": "none",
"goVersion": "go1.19.2",
"isBeta": false,
"isGit": false,
"linking": "static",
"os": "linux",
"version": "v1.60.0"
}

Are you on the latest version of rclone? You can validate by checking the version listed here:

Which cloud storage system are you using? (eg Google Drive)

Oracle Cloud and Azure

The command you were trying to run (eg rclone copy /tmp remote:tmp)

curl --location -g --request POST 'https://rclone-poc.wwt.com/sync/copy?srcFs=oci:/RCLONE/RcloneApiTest&opt={"_filter" : {"IncludeRule":"*.csv"}}&dstFs=devoracldbiccc:rcloneapitest' \

--header 'Authorization: Basic Z3VpOmFzZGozMmtq'

The rclone config contents with secrets removed.

{
	"azure": {
		"account": "rclonepoc",

		"type": "azureblob"
	},
	"devoracldbiccc": {
		"account": "stdiadev001",

		"type": "azureblob"
	},
	"oci": {
		"access_key_id": "d1c12526cce7c43054a5cec887f5c4bbefc3d4cf",
		"acl": "private",
		"endpoint": "https://wwtsoacs.compat.objectstorage.us-ashburn-1.oraclecloud.com",
		"provider": "Other",
		"region": "us-ashburn-1",
		"type": "s3"
	},
	"prdoracldbiccc": {
		"account": "stdiaprd001",

		"type": "azureblob"
	},
	"tstoracldbiccc": {
		"account": "stdiatst001",
		"type": "azureblob"
	}
}

A log from the command with the -vv flag


Hi Jeremy,

The IncludeRule is a list of filters, here is a working example (from VSCode):

POST http://localhost:5572/sync/sync HTTP/1.1
content-type: application/json

{
    "srcFs": "../testfolder/",
    "dstFs": "../testfoldertarget/",
    "_filter": { "IncludeRule": [ "*.csv" ] },
    "_config": { "DryRun": true }
}

You may also need to escape the * when using curl from the command line (not sure).

Tip: I typically start the server like this to see how my requests are received and handled:

PS> ./rclone rcd --rc-no-auth -vv
2023/01/23 19:13:57 DEBUG : rclone: Version "v1.62.0-DEV" starting with parameters ["rclone.exe" "rcd" "--rc-no-auth" "-vv"]
2023/01/23 19:13:57 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/01/23 19:14:09 DEBUG : rc: "sync/sync": with parameters map[_config:map[DryRun:true] _filter:map[IncludeRule:[*.csv]] dstFs:../testfoldertarget/ srcFs:../testfolder/]
2023/01/23 19:14:09 DEBUG : Creating backend with remote "../testfolder/"
2023/01/23 19:14:09 DEBUG : Using config file from "rclone.conf"
2023/01/23 19:14:09 DEBUG : fs cache: renaming cache item "../testfolder/" to be canonical "//?/C:/Users/olefr/Development/rclone/testfolder"
2023/01/23 19:14:09 DEBUG : Creating backend with remote "../testfoldertarget/"
2023/01/23 19:14:09 DEBUG : fs cache: renaming cache item "../testfoldertarget/" to be canonical "//?/C:/Users/olefr/Development/rclone/testfoldertarget"
2023/01/23 19:14:09 DEBUG : Local file system at .../testfoldertarget: Waiting for checks to finish
2023/01/23 19:14:09 DEBUG : Local file system at .../testfoldertarget: Waiting for transfers to finish
2023/01/23 19:14:09 NOTICE: file1.csv: Skipped copy as --dry-run is set (size 0)
2023/01/23 19:14:09 NOTICE: file2.csv: Skipped copy as --dry-run is set (size 0)
2023/01/23 19:14:09 DEBUG : Waiting for deletions to finish
2023/01/23 19:14:09 DEBUG : rc: "sync/sync": reply map[]: <nil>

Here is the example converted to curl by VSCode for your inspiration:

curl --request POST --url http://localhost:5572/sync/sync --header 'content-type: application/json' --header 'user-agent: vscode-restclient' --data '{"srcFs": "../testfolder/","dstFs": "../testfoldertarget/","_filter": { "IncludeRule": [ "*.csv" ] },"_config": { "DryRun": true }}'

Thank you Ole. This helped tremulously.

You are welcome Jeremy, glad to hear. Happy coding!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.