What is the problem you are having with rclone?
Using --include with the combine backend seems to be excluding the top level directory.
What I'm after:
The webdav source (/eos/webdavpool/data) contains many user directories, one of them being mine (myuser@mydomain.au). The user data is in this format:
$ ls /eos/webdavpool/data/myuser@mydomain.au
files files_trashbin files_versions thumbnails
I'm trying to copy data just for individual users (example, myuser), as shown in the config file, and I'm after only the files folder and nothing else.
But it keeps excluding my entire user (myuser@mydomain.au) rather than myuser@mydomain.au/files_trashbin etc when I run the rclone command below.
I'm observing the same behavior when running the same command against webdav source:
$ rclone -vv --inplace --use-json-log --config .rclone_conf ls webdav:"/" --include "/files/**"
< redacted >
04T07:47:57.763876+00:00"}
{"level":"debug","msg":"Excluded","object":"lee@mydomain.au","objectType":"*fs.Dir","source":"list/list.go:66","time":"2024-01-04T07:47:57.7639+00:00"}
{"level":"debug","msg":"Excluded","object":"lisa@mydomain.au","objectType":"*fs.Dir","source":"list/list.go:66","time":"2024-01-04T07:47:57.764131+00:00"}
{"level":"debug","msg":"Excluded","object":"myuser@mydomain.au","objectType":"*fs.Dir","source":"list/list.go:66","time":"2024-01-04T07:47:57.764113+00:00"}
<redacted>
How can I make this work?
If I do this, it works:
rclone -vv --inplace --use-json-log --config .rclone_conf ls primetenant:"/" --include "/myuser@mydomain./files/**"
However this doesnt seem ideal.
Is this a bug or did I make a mistake somewhere? Is there a better way to do this?
I did try using --exclude and -filter but both appear to include "files" in otherwise excluded directories (eg: myuser@mydomain.au/files_trashbin/files), unless I refine them with more rules. Due to the size and number of files I'm dealing with, this indicates a potential performance issue because it will need to do a walk of directories containing potentially a lot of files (more than 500k) and I worry that I may get a performance impact.
Rclone version:
2024/01/04 07:35:43 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "version"]
rclone v1.63.0
- os/version: redhat 7.9 (64 bit)
- os/kernel: 3.10.0-1160.102.1.el7.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.5
- go/linking: static
- go/tags: none
2024/01/04 07:35:43 DEBUG : rclone: Version "v1.63.0" finishing with parameters ["rclone" "version"]
Which cloud storage system are you using?
Source: EOS webdav
Destination: SFTP (using the combine backend)
The command you were trying to run
rclone -vv --inplace --use-json-log --config .rclone_conf ls sftpcloud:"/" --include "/files/**"
Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
[webdav]
type = alias
remote = /eos/webdavpool/data/
[sftpcloud]
type = combine
upstreams = "myuser@mydomain.au=webdav:/myuser@mydomain.au/"
[source]
type = hasher
remote = sftpcloud:
hashes = md5
max_age = 1s
[destination]
type = sftp
host = sftpcloud.mydomain.au
user = migration
pass = mysuperstrongpass
port = 22
disable_hashcheck = false
chunk_size = 255k
shell_type = unix
A log from the command that you were trying to run with the -vv flag
{"level":"debug","msg":"Version \"v1.63.0\" starting with parameters [\"rclone\" \"-vv\" \"--inplace\" \"--use-json-log\" \"--config\" \".rclone_conf\" \"ls\" \"sftpcloud:/\" \"--include\" \"/files/**\"]","object":"rclone","objectType":"string","source":"cmd/cmd.go:422","time":"2024-01-04T07:43:45.465042+00:00"}
{"level":"debug","msg":"Creating backend with remote \"sftpcloud:/\"","source":"fs/newfs.go:34","time":"2024-01-04T07:43:45.465414+00:00"}
{"level":"debug","msg":"Using config file from \"/home/szahri/rclone_tests/.rclone_conf\"","source":"config/config.go:362","time":"2024-01-04T07:43:45.465513+00:00"}
{"level":"debug","msg":"Creating backend with remote \"webdav:myuser@mydomain.au/\"","source":"fs/newfs.go:34","time":"2024-01-04T07:43:45.465696+00:00"}
{"level":"debug","msg":"Creating backend with remote \"/eos/webdavpool/data/myuser@mydomain.au\"","source":"fs/newfs.go:34","time":"2024-01-04T07:43:45.46574+00:00"}
{"level":"debug","msg":"fs cache: renaming cache item \"webdav:/myuser@mydomain.au/\" to be canonical \"/eos/webdavpool/data/myuser@mydomain.au\"","source":"cache/cache.go:84","time":"2024-01-04T07:43:45.628947+00:00"}
{"level":"debug","msg":"fs cache: renaming cache item \"sftpcloud:/\" to be canonical \"sftpcloud:\"","source":"cache/cache.go:84","time":"2024-01-04T07:43:45.629094+00:00"}
{"level":"debug","msg":"**Excluded","object":"myuser@mydomain.au",**"objectType":"*fs.Dir","source":"list/list.go:66","time":"2024-01-04T07:43:45.629255+00:00"}
{"level":"debug","msg":"4 go routines active\n","source":"cmd/cmd.go:300","time":"2024-01-04T07:43:45.629365+00:00"}
We are generating it programmatically so it should be good from that end.