Excluding files and including some excluded files at the same time

Help and Support
1

What is the problem you are having with rclone?

I use rclone to sync from an external hard drive to Google Drive. I would like to stop rclone from syncing ._ files (generated by my Mac) to the Google Drive. I have already run rclone so the ._ files are already on Google Drive, so I would like to delete the ._ files that are already there and for rclone not to sync ._ files in the future. Furthermore I have ._init.py files that I do want to keep syncing. My first thought was to combine --exclude with --include:

--exclude '._*' --include '._init.py'

hoping that the --include flag would pull out the .init.py files that were excluded by the --exclude flag. However this gave me warning that I should use the filter command instead. It's not clear to me what the correct syntax should be for the filter command in my case. I tried

--filter '- ._*' '+ ._init.py'

but this threw an error that I have provided 3 non flag arguments

Run the command 'rclone version' and share the full output of the command.

rclone v1.55.1

  • os/type: darwin
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo rclone sync /sourcelocation remote:/desitnationlocation --filter '- ._*' '+ .___init__.py' --progress --delete-excluded --dry-run

The rclone config contents with secrets removed.

I'm not sure how to get this, hopefully not required, if it is, can someone advise?

A log from the command with the -vv flag

Usage:
  rclone sync source:path dest:path [flags]

Flags:
      --create-empty-src-dirs   Create empty source dirs on destination after sync
  -h, --help                    help for sync

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
Command sync needs 2 arguments maximum: you provided 3 non flag arguments: ["/sourcelocation" "remote:/destinationlocation" "+ .___init__.py"]
2

Should be:

rclone sync /sourcelocation remote:/desitnationlocation --filter '- ._*,+ .___init__.py' --progress --delete-excluded --dry-run

I find it easier to test with rclone ls first as well you can see what's printing out and validate it's doing what you want.

3

Many thanks for you help. Can you clarify what you mean by testing rclone with ls first? Do you mean replace 'rclone sync ....' with 'rclone ls .....' (with the .... being the flags as suggested in your post)?

4

Sure, just use rclone ls like this instead to see if it lists out what you would expect and it filters properly.

rclone ls /sourcelocation --filter '- ._*,+ .___init__.py'

I think your filter is backwards though as it should be from most restrictive to least.

So if try

rclone ls  --filter '+ .___init__.py,- ._*'

Instead as I think that should do it.

5

Thanks for explanation. I've tried the code and whilst it's running, it is not deleting previously uploaded '._' files from my Google Drive (I mean the sync code). Any advice?

6

I've got no idea what you've run or what the output was so you'd have to share the details.

7

I ran:

sudo rclone sync /sourcelocation remote:/destinationlocation --filter '+ .___init__.py, - ._*' --delete-excluded --progress

The output was:

Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:             99999 / 99999, 100%
Elapsed time:      8m24.5s

The ._ files that were present in Google Drive have remained there.

8

Run the same command with -vvv and share the full debug log.

9

I assume you don't need a list of all the files, at the end it gave this output:

Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:             99999 / 99999, 100%
Elapsed time:      8m24.3s
2022/02/20 15:20:06 NOTICE: 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:             99999 / 99999, 100%
Elapsed time:      8m24.3s

2022/02/20 15:20:06 DEBUG : 20 go routines active

Note, I had a look through the list of files for ._ files, and they are treated the same as every other file, e.g. output is like this:

2022-02-20 15:08:06 DEBUG : filelocation/._filename.pdf: Unchanged skipping

It looks like '--delete-excluded' does not see filtered files as excluded (before when I ran sync with --exclude --dry-run it was picking them up and was ready to purge them).

10

I don't know as that's a snippet and need to see a full debug log.

11

When you say 'full debug log' do you mean the thousands of lines showing the status of each individual file? Or do you mean something else?

12

Yes, the full / complete / entire / nothing missing log.

13

Can you advise if there is a way to automatically redact file and folder names? Obviously there are some privacy considerations sharing that information.

14

hi,
why not use rclone delete --include='._*
and that would simply your sync command.

15

Thanks for the advice, I used your suggested code get rid of the ._ files from the Google Drive, but the issue remained that I didn't know what command to use to sync my local files without uploading ._ files whilst syncing .___init__.py files. In the end I found that the following worked:

sudo rclone sync /sourcelocation remote:/destinationlocation --filter '+ .___init__.py' --filter '- ._*' --delete-excluded --progress --dry-run

The difference to the previously suggested command is that there is a separate --filter for the two conditions.

16

That's why the debug log is key as it's much easier to figure out what the issue is.

The "," solution I gave works for include/exclude but not filter filter as that would have been evident in the debug log :slight_smile:

If you follow what @asdffdsa was saying and use an include, it simplies things as the exclude all is automatically implied as seen by the filtering dump.

2022/02/20 10:54:13 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
--- start filters ---
--- File filter rules ---
+ (^|/)\.___init__\.py$
- ^.*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
2022/02/20 10:54:13 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "DB:test" "-vvv" "--dump" "filters" "--include" ".___init__.py"]
2022/02/20 10:54:13 DEBUG : Creating backend with remote "DB:test"
2022/02/20 10:54:13 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/02/20 10:54:14 DEBUG : file_0002.txt: Excluded
2022/02/20 10:54:14 DEBUG : file_0003.txt: Excluded
2022/02/20 10:54:14 DEBUG : file_0005.txt: Excluded
2022/02/20 10:54:14 DEBUG : .___init_notexclude.py: Excluded
       10 .___init__.py
2022/02/20 10:54:14 DEBUG : 6 go routines active
2022/02/20 10:54:14 INFO  : Dropbox root 'test': Commiting uploads - please wait...
17

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.