Sync ignoring excludes file

I'm wanting to sync one google drive account to another google drive account, the command I am using is as follows: -

rclone command
/usr/bin/rclone sync GD1:'/D2' GD2:'/D1/D2' --dry-run --exclude-from '/path/to/excludes.txt' --fast-list --log-level DEBUG --progress

But in the destination folder is an extra directory which I don't wish to have deleted, but it seems no matter what I put in the excludes file rclone wants to delete this folder. This directory is in the excludes.txt file.

Output Log
The output log doesn't have any DEBUG entries that are specific to the directory I am wanting to exclude from the sync, the log has the following as the last entry, entries further up in the list the sub directories.

2020-02-09 13:02:41 NOTICE: 1. D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS: Not deleting as dry run is set

Contents of the excludes.txt
D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS
D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
D1/D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/

rclone version
rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7

OS version
Debian Stretch x64

What is your rclone version (output from rclone version)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Can you share the exclude file too?

Most of what you asked for was already in the initial post, but have added the extra bits.

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp )

There are no DEBUG entries that are specific to the directory I am wanting to exclude, all the DEBUG enteries are merely stating that files in other directory have the same checksum.

Can you please include a full debug log?

Is that a file or directory since I don't have a full log?

Oh it's a directory, the log merely lists that directory as the last entry, and all the sub directories it contains further up in the log.

You need to put a / at the end if it's a directory.

D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
D1/D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
felix@gemini:~/test$ rclone ls --exclude-from exclude /home/felix/test/ -vv
2020/02/09 08:47:51 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "ls" "--exclude-from" "exclude" "/home/felix/test/" "-vv"]
2020/02/09 08:47:51 DEBUG : Using config file from "/opt/rclone/rclone.conf"
      221 blah
      150 exclude
        6 files
       13 filter
      227 hosts
      227 test
        0 test.pdf
      221 test2
2020/02/09 08:47:51 DEBUG : D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS: Excluded
2020/02/09 08:47:51 DEBUG : 3 go routines active
2020/02/09 08:47:51 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "ls" "--exclude-from" "exclude" "/home/felix/test/" "-vv"]
felix@gemini:~/test$

I tried that but the problem remains, also I am doing a sync from one google drive to another google drive, and not local storage to google drive, is that why it's not working?

2020-02-09 13:59:24 NOTICE: D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS: Not deleting as dry run is set
2020-02-09 13:59:24 DEBUG : Google drive root 'D1/D2': deleted 85 directories

No directories should have been deleted.

Just do a simple ls command like I have and share the exclude file you are using the debug log of the ls.

It's because there is something incorrect as the full command and logs are shown in the debug output.

felix@gemini:~/test$ rclone ls --exclude-from exclude gcrypt:test -vv
2020/02/09 08:59:46 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "ls" "--exclude-from" "exclude" "gcrypt:test" "-vv"]
2020/02/09 08:59:46 DEBUG : Using config file from "/opt/rclone/rclone.conf"
      221 blah
      150 exclude
        6 files
       13 filter
      227 hosts
      227 test
        0 test.pdf
      221 test2
2020/02/09 08:59:47 DEBUG : D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS: Excluded
2020/02/09 08:59:47 DEBUG : 7 go routines active
2020/02/09 08:59:47 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "ls" "--exclude-from" "exclude" "gcrypt:test" "-vv"]

and the exclude.

felix@gemini:~/test$ cat exclude
D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
D1/D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/

Without the slash, it thinks it is a file and does not work.

felix@gemini:~/test$ cat exclude
D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS
D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
D1/D2/D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/
felix@gemini:~/test$ rclone ls --exclude-from exclude gcrypt:test -vv
2020/02/09 09:00:56 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "ls" "--exclude-from" "exclude" "gcrypt:test" "-vv"]
2020/02/09 09:00:56 DEBUG : Using config file from "/opt/rclone/rclone.conf"
      221 blah
      150 exclude
        6 files
       13 filter
      227 hosts
      227 test
        0 test.pdf
      221 test2
      227 D3 - Misc/1st - D4/3 - SCC.442 - Misc/4 - VMS/hosts
2020/02/09 09:00:58 DEBUG : 7 go routines active
2020/02/09 09:00:58 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "ls" "--exclude-from" "exclude" "gcrypt:test" "-vv"]

Ok is that everything?

rclone ls --exclude-from '/path/to/excludes.txt' GD2:'/Test/1. Test' -vv
2020/02/09 14:08:20 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "ls" "--exclude-from" "/path/to/excludes.txt" "GD2:/Test/1. Test" "-vv"]
2020/02/09 14:08:20 DEBUG : Using config file from "/root/.rclone.conf"
2020/02/09 14:08:28 DEBUG : 3. Test: Excluded
    0 New Text Document.txt
    0 2. Test/New Text Document.txt
2020/02/09 14:08:23 DEBUG : 7 go routines active
2020/02/09 14:08:23 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "ls" "--exclude-from" "/path/to/excludes.txt" "GD2:/Test/1. Test" "-vv"]

Contents of the exclude.txt file
3. Test/

So the testing directory seems to work, but when I go back to my real directory and have the exclude directory path from the end of the remote like in the working example, it is still wanting to delete the directory.

The directory I am wanting to exclude doesn't exist in the source google drive, but does in the destination google drive.

Edit.

@Animosity022 It seems the excludes file works only on the source google drive, so as to exclude that directory from being copied/moved/synced to the other google drive, which is great, but not what I need the destination has an extra directory in one of the directories that also is in the source, this directory however doesn't fit in the source google drive due to having only a 15GB space limit, so only exists in the destination google drive, and when I want to sync them rclone is wanting to delete the directory as it doesn't exist in the source folder.

sync makes the source look like the destination.

If you exclude, it's on the source.

If you don't want to delete, you can use copy and you can always use a filter to delete on the destination I suppose.

Did you try

  --create-empty-src-dirs   Create empty source dirs on destination after sync

If the filter excludes the contents of the source directory then rclone will think it is empty and want to delete it on the destination.

I'm wanting to keep the contents of one drive (max 15GB space) synced with my GSuite drive, but a folder exists that is too large for the source google drive, and so this only exists in the destination, and sync is wanting to delete this folder.

@ncw I didn't try --create-empty-src-dirs but have just now, and the same problem exists in that it still wants to delete the folder in the destination.

It's a shame the excludes file wasn't able to be used for excluding both in the source and in the destination to prevent folders that don't exist from being deleted.

I just looked at the logs and in the first 10 lines this appears.

2020/02/15 07:08:36 DEBUG : Folder1/Folder2/Folder3/Folder4: Excluded from sync (and deletion)

But at the very end of the log this still appears as normal.

2020/02/15 07:08:54 NOTICE: Folder1/Folder2/Folder3/Folder4: Not deleting as dry run is set

Could you make a very small example of the problem? If you set up two local directories called src and dst then you should be able to make some files/directories in src and dst to demonstrate.

I tried this

.
├── dst
│   ├── dir
│   └── file1.txt
└── src
    ├── dir
    │   └── file2.txt
    └── file1.txt

With rclone sync --exclude "/dir/**" -vv src dst it didn't delete the dir directory on the dst so I obviously haven't quite understood your scenario yet.

@ncw hows this?

├── dst
│   ├── dir
│   │   └── file2.txt
│   └── file1.txt
└── src
   └── file1.txt
2020/02/16 16:24:54 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "sync" "--exclude" "/dir/**" "GD1:/src" "GD2:/dst" "-vv"]
2020/02/16 16:24:54 DEBUG : Using config file from "/root/.rclone.conf"
2020/02/16 16:24:56 DEBUG : dir: Excluded
2020/02/16 16:24:56 DEBUG : file1.txt: Size and modification time the same (differ by 0s, within tolerance 1ms)
2020/02/16 16:24:56 DEBUG : file1.txt: Unchanged skipping
2020/02/16 16:24:56 INFO  : Google drive root 'dst': Waiting for checks to finish
2020/02/16 16:24:56 INFO  : Google drive root 'dst': Waiting for transfers to finish
2020/02/16 16:24:56 INFO  : Waiting for deletions to finish
2020/02/16 16:24:56 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2020/02/16 16:24:56 DEBUG : 8 go routines active
2020/02/16 16:24:56 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "sync" "--exclude" "/dir/**" "GD1:/src" "GD2:/dst" "-vv"]

Oh that example seems to have worked, I think my exclude files is wrong as I didn't have the ** after the directory name.

That is a well known gotcha...

I keep meaning to make "dir/" equivalent to "dir/**" which would be what most people would expect.