Filter-file contents does not exclude 3 specific files

Help and Support
1

What is the problem you are having with rclone?

I am new to ubuntu and rclone and google drive. Just trying to learn

My "filter-from" file does not seem to be working.
I run
"sudo rclone copy --filter-from=$HOME/rclone_filter.txt -L /etc gdrive: linux_files/backup/etc"

I use sudo because many of the files are owned by root and copying under my userid causes security errors.

I have run it with --dry-run. Each time I get 2 errors.
2 Symlinks in /etc sub directories point to non-existent files.
In my rclone_filter.txt file I have these enteries:

- /NetworkManager/
- /etc/network/if-post-down.d/avahi-daemon
- /etc/pulse/client.conf.d/01-enable-autospawn.conf
- /etc/mtab

I don't know if the first entry is working there is no error.

I have tried commenting out all combinations but nothing changes.

The last three lines try to exclude 3 specific files from being copied.
The second and third lines are smylinks to non-existent files.
mtab should not be copied since it causes corruption on the copy.

so why do I still see the same 3 errors even though I am trying to exclude them
I am running ubuntu linux (lastest edition 21.x). I just installed rclone two days ago

  • os/arch: linux/amd64
  • go version: go1.13.6

both the rclone_filter.txt and the rclone_log.txt in my home directory have 777 permissions.

What is your rclone version (output from rclone version)

I am running ubuntu linux (lastest edition 21.x). I just installed rclone two days ago

  • os/arch: linux/amd64
  • go version: go1.13.6

both the rclone_filter.txt and the rclone_log.txt in my home directory have 777 permissions.

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

sudo rclone copy -L --filter-from=$HOME/rclone_filter.txt --log-file=$HOME/rclone_log.txt /etc gdrive:_Ubuntu_files/backup/etc -v

The rclone config contents with secrets removed.

[gdrive]
scope = drive
token = {"access_token":"ya29.a0ARrdaM9G8PXBK1yy1L0uPe1tfGLlGjiUr4wdt6PgudPN4dy3bnKkk6Xoc9GgyHrBwWFWeRYx0RHdQV2tYnysU9LAHYTRQAy33sXZfteWQRk8uJ56Ue8RPOYCg3_a5kUpQ8qdMwbXz_fZvMguAzZOJBTdQ91u","token_type":"Bearer","refresh_token":"1//04Q5i2xyfgql3CgYIARAAGAQSNwF-L9IrcL0efo8ThqItWbUKt70RXJ26llRl7QnLQMNoM6Lfk-cqzMOK4FvhOhAKLSmLV9z4rYY","expiry":"2021-09-17T22:47:37.590504476-05:00"}

A log from the command with the -vv flag

2021/09/17 21:48:42 ERROR : mtab: corrupted on transfer: sizes differ 0 vs 4290
2021/09/17 21:48:42 INFO  : mtab: Removing failed copy
2021/09/17 21:48:46 INFO  : cups/subscriptions.conf: Copied (replaced existing)
2021/09/17 21:48:46 INFO  : cups/subscriptions.conf.O: Copied (replaced existing)
2021/09/17 21:48:56 ERROR : network/if-post-down.d/avahi-daemon: Listing error: symlink: stat /etc/network/if-post-down.d/avahi-daemon: no such file or directory
2021/09/17 21:48:57 ERROR : pulse/client.conf.d/01-enable-autospawn.conf: Listing error: symlink: stat /etc/pulse/client.conf.d/01-enable-autospawn.conf: no such file or directory
2021/09/17 21:49:00 INFO  : Google drive root '_Ubuntu_files/backup/etc': Waiting for checks to finish
2021/09/17 21:49:00 INFO  : Google drive root '_Ubuntu_files/backup/etc': Waiting for transfers to finish
2021/09/17 21:49:00 ERROR : Attempt 1/3 failed with 5 errors and: corrupted on transfer: sizes differ 0 vs 4290
2021/09/17 21:49:04 ERROR : mtab: corrupted on transfer: sizes differ 0 vs 4290
2021/09/17 21:49:04 INFO  : mtab: Removing failed copy
2021/09/17 21:49:18 ERROR : network/if-post-down.d/avahi-daemon: Listing error: symlink: stat /etc/network/if-post-down.d/avahi-daemon: no such file or directory
2021/09/17 21:49:18 ERROR : pulse/client.conf.d/01-enable-autospawn.conf: Listing error: symlink: stat /etc/pulse/client.conf.d/01-enable-autospawn.conf: no such file or directory
2021/09/17 21:49:27 INFO  : Google drive root '_Ubuntu_files/backup/etc': Waiting for checks to finish
2021/09/17 21:49:27 INFO  : Google drive root '_Ubuntu_files/backup/etc': Waiting for transfers to finish
2021/09/17 21:49:27 ERROR : Attempt 2/3 failed with 5 errors and: corrupted on transfer: sizes differ 0 vs 4290
2021/09/17 21:49:29 ERROR : mtab: corrupted on transfer: sizes differ 0 vs 4290
2021/09/17 21:49:29 INFO  : mtab: Removing failed copy
2021/09/17 21:49:38 INFO  : 
Transferred:   	   15.232k / 15.232 kBytes, 100%, 778 Bytes/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:              5552 / 5552, 100%
Transferred:            2 / 4, 50%
Elapsed time:         20s
Transferring:
 *                       cups/subscriptions.conf:100% /682, 340/s, 0s
 *                     cups/subscriptions.conf.O:100% /682, 340/s, 0s

2021/09/17 21:49:38 INFO  : cups/subscriptions.conf.O: Copied (replaced existing)
2021/09/17 21:49:38 INFO  : cups/subscriptions.conf: Copied (replaced existing)
2021/09/17 21:50:38 INFO  : 
Transferred:   	   15.232k / 15.232 kBytes, 100%, 767 Bytes/s, ETA 0s
Errors:                 2 (retrying may help)
Checks:              5863 / 5863, 100%
Transferred:            4 / 4, 100%
Elapsed time:       20.3s

2021/09/17 21:50:45 ERROR : network/if-post-down.d/avahi-daemon: Listing error: symlink: stat /etc/network/if-post-down.d/avahi-daemon: no such file or directory
2021/09/17 21:50:46 ERROR : pulse/client.conf.d/01-enable-autospawn.conf: Listing error: symlink: stat /etc/pulse/client.conf.d/01-enable-autospawn.conf: no such file or directory
2021/09/17 21:50:57 INFO  : Google drive root '_Ubuntu_files/backup/etc': Waiting for checks to finish
2021/09/17 21:50:57 INFO  : Google drive root '_Ubuntu_files/backup/etc': Waiting for transfers to finish
2021/09/17 21:50:57 ERROR : Attempt 3/3 failed with 5 errors and: corrupted on transfer: sizes differ 0 vs 4290
2021/09/17 21:50:57 Failed to copy with 5 errors: last error was: corrupted on transfer: sizes differ 0 vs 4290
2

This unfortunately is a known gotcha.

The filtering works at a higher level to the backends and it is the local backend which reports the error before the files have been filtered

If you use the -l flag instead of -L then rclone will copy the contents of the link rather than the file it points to, which is probably what you want and I don't think it will give this error in that case.

You could also use

  --skip-links                          Don't warn about skipped symlinks.
3

hi,

that version of rclone seems very old, the latest stable is v1.56.0.
the ubuntu repository, like most linux repositories, use old versions.

you can update your version using
https://rclone.org/downloads/#script-download-and-install

4

actually I am using this as a backup. I would prefer that all smylinks be copied as links rather than as the file they are linking.

That way when I restore the copy it would end up as a link.
copying the file would destroy the link to the real file.

If I don't copy links then the restore will be incomplete. And the system might not work the same as with the original

5

YMMV - I wouldn't use rclone as a Linux backup since you can't get permissions or anything of value. There are more tools that do a much better job imo.

6

ok i am new.
can you name a tool that will backup linux to google drive directly, is free or at low cost (no monthly service) and is not an insync type tool (I have a limited amount of network quota - continuous sync is too expensive - just want to run it on a fixed schedule or on demand) and provides recovery if my machine or OS is wiped out?

thanks for any help

7

A few google results:

Assuming you want a GUI.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.