Unable to filter from root folder of remote storage

peter-olom · June 6, 2022, 3:57pm

What is the problem you are having with rclone?

I am unable to filter files in the root of my storage. It works in any of the sub directories but not in the root

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1
- os/version: Microsoft Windows 11 Pro 21H2 (64 bit)
- os/kernel: 10.0.22000.675 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.17.9
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Next Cloud

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone lsf  remote: --include="Readme.md" -vv

The rclone config contents with secrets removed.

[remote]
type = webdav
url = https://login.practicaldrive.com/remote.php/dav/files/username/
vendor = nextcloud
user = 
pass =

A log from the command with the `-vv` flag

2022/06/06 16:54:54 DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\Tools\\rclone.exe" "lsf" "remote:" "--include=Readme.md" "-vv"]
2022/06/06 16:54:54 DEBUG : Creating backend with remote "remote:"
2022/06/06 16:54:54 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/06 16:54:54 DEBUG : found headers:
2022/06/06 16:54:56 DEBUG : .DS_Store: Excluded
2022/06/06 16:54:56 DEBUG : ._.DS_Store: Excluded
Documents/
OldEmails/
Photos/
Readme.md
Talk/

Animosity022 · June 6, 2022, 4:09pm

What's not working? The output looks like it excluded the files in that directory. What are you expecting to happen that isn't?

peter-olom · June 6, 2022, 5:23pm

You are right Animosity022.

To give full context:

While trying to troubleshoot copying from the root I ended up testing using the --include flag. I was previously adding my filters in a manifest.txt file and passing it in with --include-from.

Running the copy command with --dry-run -P -vv, I can see that the filter is actually being applied. I only just learnt about the -vv flag as I came here to ask for help.

The output from -vv and your comment has now revealed my real problem which is:

rclone is traversing sub folders of other folder within my storage.

The other folders could contain thousands of files which causes the copy command to run for a very long time.

How do I prevent this behavior and copy only the files/folder in the manifest.txt without traversing other directories?

Command:

rclone copy  FilerNextCloud: --include-from="C:\Users\user\manifest.txt" "C:\Dest" --dry-run -P -vv

Content of manifest.txt:

Photos/*
Readme.md

Command Output

2022/06/06 18:16:38 DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\Tools\\rclone.exe" "copy" "remote:" "--include-from=C:\\Users\\user\\manifest.txt" "C:\\Dest" "--dry-run" "-P" "-vv"]
2022/06/06 18:16:38 DEBUG : Creating backend with remote "remote:"
2022/06/06 18:16:38 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2022/06/06 18:16:38 DEBUG : found headers:
2022/06/06 18:16:43 DEBUG : Creating backend with remote "C:\\Dest"
2022/06/06 18:16:43 DEBUG : fs cache: renaming cache item "C:\\Dest" to be canonical "//?/C:/Dest"
2022-06-06 18:16:44 NOTICE: Readme.md: Skipped copy as --dry-run is set (size 1)                                                                                                                                                                                                                                                                                   
2022-06-06 18:16:47 NOTICE: Photos/Coast.jpg: Skipped copy as --dry-run is set (size 800.553Ki)
2022-06-06 18:16:47 NOTICE: Photos/Nut.jpg: Skipped copy as --dry-run is set (size 932.643Ki)                                                                                                                                                                                                                                                                      
2022-06-06 18:16:47 NOTICE: Photos/Hummingbird.jpg: Skipped copy as --dry-run is set (size 571.503Ki)                                                                                                                                                                                                                                                              
2022-06-06 18:16:47 DEBUG : Documents/About.odt: Excluded
2022-06-06 18:16:47 DEBUG : Documents/About.txt: Excluded
2022-06-06 18:16:52 DEBUG : Local file system at //?/C:/Dest: Waiting for checks to finish
2022-06-06 18:16:52 DEBUG : Local file system at //?/C:/Dest: Waiting for transfers to finish                                                                                                                                                                                                                                                                      
Transferred:        2.251 MiB / 2.251 MiB, 100%, 328.279 KiB/s, ETA 0s                                                                                                                                                                                                                                                                                             
Transferred:            4 / 4, 100%                                                                                                                                                                                                                                                                                                                                
Elapsed time:        13.5s
2022/06/06 18:16:52 NOTICE:
Transferred:        2.251 MiB / 2.251 MiB, 100%, 328.279 KiB/s, ETA 0s
Transferred:            4 / 4, 100%
Elapsed time:        13.5s

Notice that the contents of Documents/ is read how do I avoid this because I'm only interested in Readme.md, Photos and whatever subfolders and files it may have?

ncw · June 6, 2022, 5:30pm

This says "Look for a file called Readme.md anywhere under remote:" - that is why it recurses all the directories.

If you want just a file called "Readme.md" in the root then use

rclone lsf  remote: --include="/Readme.md" -vv

And rclone knows it doesn't need to recurse the other directories.

peter-olom · June 6, 2022, 5:36pm

Got it. Thanks

I updated 'manifest.txt' content to

/Photos/*
/Readme.md

system · June 9, 2022, 5:36pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.