Files specified in --include-from are ignored

What is the problem you are having with rclone?

I want to copy some files to my encrypted Google Drive (remote: gcrypt). These files are in different folders, with other files (that I don't want to copy).
I made a textfile with all the filenames of the files I want to copy, and refer to this file with --include-from, but no files are copied. In the verbose print I can't see that files are included or excluded.
I also tried with the non-encrypted Google Drive (gdrive:), but that made no difference.

Run the command 'rclone version' and share the full output of the command.

D:\Program Files\rclone>rclone version
rclone v1.64.2

  • os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
  • os/kernel: 10.0.17763.4974 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy F:\Test gcrypt:\Test --include-from=files.txt --dry-run -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gcrypt]
type = crypt
remote = gdrive:/crypt
password = XXX
password2 = XXX

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive =
#

A log from the command that you were trying to run with the -vv flag

2023/11/26 16:59:45 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "copy" "F:\\Test" "gcrypt:\\Test" "--include-from=files.txt" "--dry-run" "-vv"]
2023/11/26 16:59:45 DEBUG : Creating backend with remote "F:\\Test"
2023/11/26 16:59:45 DEBUG : Using config file from "D:\\Program Files\\rclone\\rclone.conf"
2023/11/26 16:59:45 DEBUG : fs cache: renaming cache item "F:\\Test" to be canonical "//?/F:/Test"
2023/11/26 16:59:45 DEBUG : Creating backend with remote "gcrypt:\\Test"
2023/11/26 16:59:45 DEBUG : Creating backend with remote "gdrive:/crypt/tj67irlirpfegrcev287vlttf8"
2023/11/26 16:59:45 DEBUG : Google drive root 'crypt/tj67irlirpfegrcev287vlttf8': 'root_folder_id = 0AOIL8zKYbJt1Uk9PVA' - save this in the config to speed up startup
2023/11/26 16:59:46 DEBUG : fs cache: renaming cache item "gdrive:/crypt/tj67irlirpfegrcev287vlttf8" to be canonical "gdrive:crypt/tj67irlirpfegrcev287vlttf8"
2023/11/26 16:59:46 DEBUG : fs cache: switching user supplied name "gdrive:/crypt/tj67irlirpfegrcev287vlttf8" for canonical name "gdrive:crypt/tj67irlirpfegrcev287vlttf8"
2023/11/26 16:59:46 DEBUG : fs cache: renaming cache item "gcrypt:\\Test" to be canonical "gcrypt:/Test"
2023/11/26 16:59:46 DEBUG : Encrypted drive 'gcrypt:/Test': Waiting for checks to finish
2023/11/26 16:59:46 DEBUG : Encrypted drive 'gcrypt:/Test': Waiting for transfers to finish
2023/11/26 16:59:46 INFO  : There was nothing to transfer
2023/11/26 16:59:46 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.8s

2023/11/26 16:59:46 DEBUG : 8 go routines active

Directory listing of source:

D:\Program Files\rclone>rclone ls F:\Test
  2159161 Folder3/Test10.bin
 50790400 Folder3/Test7.bin
 35782656 Folder3/Test8.bin
  3792845 Folder3/Test9.bin
164823040 Folder1/Test1.bin
136904704 Folder1/Test2.bin
135987200 Folder1/Test3.bin
 94306304 Folder2/Test4.bin
 78839808 Folder2/Test5.bin
 55967744 Folder2/Test6.bin

files.txt:

Test1.bin
Test4.bin
Test7.bin

welcome to the forum,

what is the output of
rclone ls F:\Test --include-from=files.txt -vv --dump=filters

and what if you specify a single file?
rclone ls F:\Test --include=Test1.bin -vv --dump=filters

D:\Program Files\rclone>rclone ls F:\Test --include-from=files.txt -vv --dump=filters
--- start filters ---
--- File filter rules ---
 $(^|/)��T e s t 1 \. b i n
 $(^|/) T e s t 4 \. b i n
 $(^|/) T e s t 7 \. b i n
+ (^|/) $
- ^.*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
2023/11/26 17:38:32 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "ls" "F:\\Test" "--include-from=files.txt" "-vv" "--dump=filters"]
2023/11/26 17:38:32 DEBUG : Creating backend with remote "F:\\Test"
2023/11/26 17:38:32 DEBUG : Using config file from "D:\\Program Files\\rclone\\rclone.conf"
2023/11/26 17:38:32 DEBUG : fs cache: renaming cache item "F:\\Test" to be canonical "//?/F:/Test"
2023/11/26 17:38:32 DEBUG : 2 go routines active
D:\Program Files\rclone>rclone ls F:\Test --include=Test1.bin -vv --dump=filters
--- start filters ---
--- File filter rules ---
+ (^|/)Test1\.bin$
- ^.*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
2023/11/26 17:39:39 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "ls" "F:\\Test" "--include=Test1.bin" "-vv" "--dump=filters"]
2023/11/26 17:39:39 DEBUG : Creating backend with remote "F:\\Test"
2023/11/26 17:39:39 DEBUG : Using config file from "D:\\Program Files\\rclone\\rclone.conf"
2023/11/26 17:39:39 DEBUG : fs cache: renaming cache item "F:\\Test" to be canonical "//?/F:/Test"
164823040 Folder1/Test1.bin
2023/11/26 17:39:39 DEBUG : 2 go routines active

It looks like it's reading files.txt wrong. It's only a text file with on each row a filename. No spaces or special characters...

what is the character encoding on that file?
convert the file to something like UTF-8 or ANSI

rclone cat list_utf-16.be.bom.txt
��Test1.bin

rclone cat list_utf-8.txt
Test1.bin

I can see in Notepad++ that the encoding of the file is "UTF-16 LE BOM".
The file is created via Out-File cmd-let in PowerShell, and I had to add "-Encoding ASCII" to get a UTF-8 encoded file.

It seems to work now:

--- start filters ---
--- File filter rules ---
+ (^|/)Test1\.bin$
+ (^|/)Test4\.bin$
+ (^|/)Test7\.bin$
- ^.*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
2023/11/26 19:42:35 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "ls" "F:\\Test" "--include-from=files.txt" "-vv" "--dump=filters"]
2023/11/26 19:42:35 DEBUG : Creating backend with remote "F:\\Test"
2023/11/26 19:42:35 DEBUG : Using config file from "D:\\Program Files\\rclone\\rclone.conf"
2023/11/26 19:42:35 DEBUG : fs cache: renaming cache item "F:\\Test" to be canonical "//?/F:/Test"
 50790400 Folder3/Test7.bin
164823040 Folder1/Test1.bin
 94306304 Folder2/Test4.bin
2023/11/26 19:42:35 DEBUG : 2 go routines active
2023/11/26 19:35:24 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "copy" "F:\\Test" "gcrypt:\\Test" "--include-from=files.txt" "--dry-run" "-vv"]
2023/11/26 19:35:24 DEBUG : Creating backend with remote "F:\\Test"
2023/11/26 19:35:24 DEBUG : Using config file from "D:\\Program Files\\rclone\\rclone.conf"
2023/11/26 19:35:24 DEBUG : fs cache: renaming cache item "F:\\Test" to be canonical "//?/F:/Test"
2023/11/26 19:35:24 DEBUG : Creating backend with remote "gcrypt:\\Test"
2023/11/26 19:35:24 DEBUG : Creating backend with remote "gdrive:/crypt/tj67irlirpfegrcev287vlttf8"
2023/11/26 19:35:24 DEBUG : Google drive root 'crypt/tj67irlirpfegrcev287vlttf8': 'root_folder_id = 0AOIL8zKYbJt1Uk9PVA' - save this in the config to speed up startup
2023/11/26 19:35:25 DEBUG : fs cache: renaming cache item "gdrive:/crypt/tj67irlirpfegrcev287vlttf8" to be canonical "gdrive:crypt/tj67irlirpfegrcev287vlttf8"
2023/11/26 19:35:25 DEBUG : fs cache: switching user supplied name "gdrive:/crypt/tj67irlirpfegrcev287vlttf8" for canonical name "gdrive:crypt/tj67irlirpfegrcev287vlttf8"
2023/11/26 19:35:25 DEBUG : fs cache: renaming cache item "gcrypt:\\Test" to be canonical "gcrypt:/Test"
2023/11/26 19:35:25 DEBUG : Folder3/Test7.bin: Need to transfer - File not found at Destination
2023/11/26 19:35:25 NOTICE: Folder3/Test7.bin: Skipped copy as --dry-run is set (size 48.438Mi)
2023/11/26 19:35:25 DEBUG : Folder1/Test1.bin: Need to transfer - File not found at Destination
2023/11/26 19:35:25 NOTICE: Folder1/Test1.bin: Skipped copy as --dry-run is set (size 157.188Mi)
2023/11/26 19:35:25 DEBUG : Folder2/Test4.bin: Need to transfer - File not found at Destination
2023/11/26 19:35:25 NOTICE: Folder2/Test4.bin: Skipped copy as --dry-run is set (size 89.938Mi)
2023/11/26 19:35:25 DEBUG : Encrypted drive 'gcrypt:/Test': Waiting for checks to finish
2023/11/26 19:35:25 DEBUG : Encrypted drive 'gcrypt:/Test': Waiting for transfers to finish
2023/11/26 19:35:25 NOTICE:
Transferred:      295.562 MiB / 295.562 MiB, 100%, 0 B/s, ETA -
Transferred:            3 / 3, 100%
Elapsed time:         1.6s

2023/11/26 19:35:25 DEBUG : 9 go routines active

@asdffdsa I marked your reply as the solution.
Thank you very much!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.