Log files which have been excluded with "--exclude-from"

IT_PI · March 12, 2025, 3:09pm

What is the problem you are having with rclone?

I would like to log files which have been excluded. I found this topic Why rclone output log show "Excluded from sync" using "rclone copy" with a single file? but it is a copy, and I would need this output for a sync. Ideally I could get the list of excluded files without running a dry run afterwards without excluding files to see what still needs to be transferred ...

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.1

os/version: ubuntu 24.04 (64 bit)
os/kernel: 6.8.0-54-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.24.0
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync "/mnt/source/zz_share_cg/"  gw-migr-2025-03-06:bkp12 -c --create-empty-src-dirs --log-file=/mnt/leah-nc/z_logs/testing.log --log-format date,time,shortfile,longfile --log-level DEBUG --exclude-from /mnt/leah-nc/z_logs/exlude.txt

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gw-migr-2025-03-06]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = XXX
root_folder_id = 
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

2025/03/12 15:49:42 log.go:179: DEBUG : 9 go routines active
2025/03/12 15:51:33 log.go:179: DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "sync" "/mnt/source/zz_share_cg/" "gw-migr-2025-03-06:bkp12" "-c" "--create-empty-src-dirs" "--log-file=/mnt/leah-nc/z_logs/testing.log" "--log-format" "date,time,shortfile,longfile" "--log-level" "DEBUG" "--exclude-from" "/mnt/source/z_logs/exlude.txt"]
2025/03/12 15:51:33 log.go:179: DEBUG : Creating backend with remote "/mnt/source/zz_share_cg/"
2025/03/12 15:51:33 log.go:179: DEBUG : Using config file from "/home/user/logs/rclone/rclone.conf"
2025/03/12 15:51:33 log.go:179: DEBUG : fs cache: renaming cache item "/mnt/source/zz_share_cg/" to be canonical "/mnt/source/zz_share_cg"
2025/03/12 15:51:33 log.go:179: DEBUG : Creating backend with remote "gw-migr-2025-03-06:bkp12"
2025/03/12 15:51:34 log.go:179: DEBUG : 2021: Making directory with metadata
2025/03/12 15:51:36 log.go:179: INFO  : 2021: Made directory with metadata (mtime=2023-07-02T11:57:24+02:00)
2025/03/12 15:51:36 log.go:179: DEBUG : HO-XXXX_EN.dotx: Need to transfer - File not found at Destination
2025/03/12 15:51:36 log.go:179: DEBUG : test-file (copy 1).pgp: Need to transfer - File not found at Destination
2025/03/12 15:51:36 log.go:179: DEBUG : test-file.odt: Need to transfer - File not found at Destination
2025/03/12 15:51:36 log.go:179: DEBUG : zz_share_cg.zip: Need to transfer - File not found at Destination
2025/03/12 15:51:36 log.go:179: DEBUG : 2021/1: Making directory with metadata
2025/03/12 15:51:37 log.go:179: INFO  : 2021/1: Made directory with metadata (mtime=2023-07-02T11:57:24+02:00)
2025/03/12 15:51:37 log.go:179: DEBUG : 2021/2021_XXX: Making directory with metadata
2025/03/12 15:51:38 log.go:179: DEBUG : HO-YYYY_EN.dotx: md5 = f0d62730ed9fe9baf4a8facbaea48eff OK
2025/03/12 15:51:38 log.go:179: INFO  : HO-YYYY_EN.dotx: Copied (new)
2025/03/12 15:51:38 log.go:179: DEBUG : test-file.odt: md5 = 0104c9040a4205425b2743b9671485ee OK
2025/03/12 15:51:38 log.go:179: INFO  : test-file.odt: Copied (new)
2025/03/12 15:51:38 log.go:179: DEBUG : test-file (copy 1).pgp: md5 = 0104c9040a4205425b2743b9671485ee OK
2025/03/12 15:51:38 log.go:179: INFO  : test-file (copy 1).pgp: Copied (new)
2025/03/12 15:51:38 log.go:179: DEBUG : zz_share_cg.zip: md5 = 859a1781a617f08a591b383c1b46bd68 OK
2025/03/12 15:51:38 log.go:179: INFO  : zz_share_cg.zip: Copied (new)
2025/03/12 15:51:38 log.go:179: INFO  : 2021/2021_XXX: Made directory with metadata (mtime=2023-07-02T11:57:24+02:00)
2025/03/12 15:51:38 log.go:179: DEBUG : 2021/2021_XXX/2021-03-03_document-X_DP_CG.odt: Need to transfer - File not found at Destination
2025/03/12 15:51:38 log.go:179: DEBUG : 2021/1/2021-03-03_document-X_DP_CG.odt: Need to transfer - File not found at Destination
2025/03/12 15:51:38 log.go:179: DEBUG : Google drive root 'bkp12': Waiting for checks to finish
2025/03/12 15:51:38 log.go:179: DEBUG : Google drive root 'bkp12': Waiting for transfers to finish
2025/03/12 15:51:40 log.go:179: DEBUG : 2021/1/2021-03-03_document-X_DP_CG.odt: md5 = 7bc22cdc681de83e1239120a0ba17506 OK
2025/03/12 15:51:40 log.go:179: INFO  : 2021/1/2021-03-03_document-X_DP_CG.odt: Copied (new)
2025/03/12 15:51:40 log.go:179: DEBUG : 2021/2021_XXX/2021-03-03_document-X_DP_CG.odt: md5 = 7bc22cdc681de83e1239120a0ba17506 OK
2025/03/12 15:51:40 log.go:179: INFO  : 2021/2021_XXX/2021-03-03_document-X_DP_CG.odt: Copied (new)
2025/03/12 15:51:40 log.go:179: DEBUG : Waiting for deletions to finish
2025/03/12 15:51:40 log.go:179: INFO  :
Transferred:      584.325 KiB / 584.325 KiB, 100%, 146.085 KiB/s, ETA 0s
Transferred:            6 / 6, 100%
Elapsed time:         6.7s

kapitainsky · March 12, 2025, 3:24pm

I am not entirely sure what do you need but have a look at sync docs:

at Logger Flags section.

You can get detailed logging what sync is doing.

IT_PI · March 12, 2025, 3:48pm

Thanks for the quick reply, the logger is indeed a better way to log what files have been transferred, updated, deleted or stayed the same.

However, I still cannot find a way to also log files which have been excluded from the sync - it doesn´t seem to work with the logger options

kapitainsky · March 12, 2025, 7:29pm

Interestingly it was apparently "fixed" here:

github.com/rclone/rclone

adding more detailed logging

master ← KDreynolds:master

opened 04:28AM - 06 Jan 24 UTC

KDreynolds

+14 -9

#### What is the purpose of this change? #### Was the change discussed in… an issue or in the forum before? #### Checklist - [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix). - [x] I have added tests for all changes in this PR if appropriate. - [x] I have added documentation for the changes if appropriate. - [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages). - [x] I'm done, this Pull Request is ready for review :-)

but I do not think it is working.

If you feel like investigating and fixing it then go for it:)

Otherwise you can create an issue on github. At least we will have it recorded.

IT_PI · March 13, 2025, 1:56pm

Thanks, it should indeed work, see: More detailed Logging · Issue #7463 · rclone/rclone · GitHub

and it works with e.g. --max-age=XX, I see the exclusions in the logfiles.

....
2025/03/13 14:50:08 log.go:179: DEBUG : 2021/2021_Human-resources: Directory modification time the same (differ by 0s, within tolerance 1ms)
2025/03/13 14:50:08 log.go:179: DEBUG : 2021/1/2021-03-03_document-X_DP_CG.odt: Excluded (ModTime Filter)
2025/03/13 14:50:08 log.go:179: DEBUG : 2021/1/2021-03-03_document-X_DP_CG.odt: Excluded
2025/03/13 14:50:08 log.go:179: DEBUG : 2021/2021_XXXX/2021-03-03_document-X_DP_CG.odt: Excluded (ModTime Filter) 
2025/03/13 14:50:08 log.go:179: DEBUG : 2021/2021_XXXX/2021-03-03_document-X_DP_CG.odt: Excluded
2025/03/13 14:50:08 log.go:179: DEBUG : Google drive root 'bkp19': Waiting for checks to finish
2025/03/13 14:50:08 log.go:179: DEBUG : Google drive root 'bkp19': Waiting for transfers to finish
2025/03/13 14:50:08 log.go:179: DEBUG : Waiting for deletions to finish
...

so I will file a bug report on github that somehow line 339 is never reached: fs: add more detailed logging for file includes/excludes · rclone/rclone@7835991 · GitHub