List all files on local but not bisync remote and why?

Help and Support
1

What is the problem you are having with rclone?

I am using rclone to bisync my desktop and laptop via IDrive with some filters. It seems to work great but on a test of ~9600 local desktop files only ~9000 are synced to my laptop. This is probably just the filters, but I did not think I would have so many files matching those. Is there some way I can actually output every file that is not synced for any reason, and why?

I have tried to do this using diff on a sorted list of files from my local desktop and remote but I realised that even sorted, once a single file is missed all subsequent lines will not match.

find ~/sync -type f > local_files
rclone lsf idrive-sync-crypt:/ --recursive > remote_files

sort local_files > local_files_sorted
sort remote_files > remote_files_sorted
diff local_files_sorted remote_files_sorted

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-101-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.1
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

IDrive S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone bisync "idrive-sync-crypt:/" "/home/salty/sync/" --check-access --fast-list --verbose --filter-from=/home/salty/.config/rclone/filters --links

WIP filters template I found:

- .remmina/**
- **~lock**
- **.tmp**
- **.log**
- .Trash-1000/**

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[idrive-sync]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
acl = private
endpoint = x3k6.ie.idrivee2-50.com
bucket_acl = private
no_check_bucket = true
server_side_encryption = aws:kms

[idrive-sync-crypt]
type = crypt
remote = idrive-sync:sync
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

The log contains a lot of personal file documents, so I would have to create a new directory with other files containing non personal documents to share this, which may not contain the same unsynced file types but I have shared the beginning and end of the output below:

2024/03/30 11:02:07 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "bisync" "idrive-sync-crypt:/" "/home/salty/sync/" "--check-access" "--fast-list" "-vv" "--filter-from=/home/salty/.config/rclone/filters" "--links"]
2024/03/30 11:02:07 DEBUG : Creating backend with remote "idrive-sync-crypt:/"
2024/03/30 11:02:07 DEBUG : Using config file from "/home/salty/.config/rclone/rclone.conf"
2024/03/30 11:02:08 DEBUG : Creating backend with remote "idrive-sync:sync"
2024/03/30 11:02:08 DEBUG : Resolving service "s3" region "us-east-1"
2024/03/30 11:02:08 DEBUG : Creating backend with remote "/home/salty/sync/"
2024/03/30 11:02:08 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2024/03/30 11:02:08 DEBUG : fs cache: renaming cache item "/home/salty/sync/" to be canonical "local{b6816}:/home/salty/sync"
2024/03/30 11:02:08 NOTICE: bisync is IN BETA. Don't use in production!
2024/03/30 11:02:08 INFO  : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
2024/03/30 11:02:08 INFO  : Bisyncing with Comparison Settings: 
{
	"Modtime": true,
	"Size": true,
	"Checksum": false,
	"HashType1": 0,
	"HashType2": 0,
	"NoSlowHash": false,
	"SlowHashSyncOnly": false,
	"SlowHashDetected": true,
	"DownloadHash": false
}
2024/03/30 11:02:08 DEBUG : Lock file created: /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.lck
2024/03/30 11:02:08 INFO  : Synching Path1 "idrive-sync-crypt:/" with Path2 "local{b6816}:/home/salty/sync/"
2024/03/30 11:02:08 DEBUG : : updated backup-dir for Path1
2024/03/30 11:02:08 DEBUG : : updated backup-dir for Path2
2024/03/30 11:02:08 INFO  : Building Path1 and Path2 listings
2024/03/30 11:02:08 DEBUG : &{0xc0005db5f0 0xc000bdc320 false false false /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync /home/salty/.cache/rclone/bisync /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.path1.lst /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.path2.lst /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.path1.lst-new /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.path2.lst-new map[] 0xc000ad8d80 {{}} {{}} false false <nil> <nil>  /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.lck map[] false}: starting to march!
2024/03/30 11:02:09 DEBUG : dev: path2 only
2024/03/30 11:02:09 DEBUG : dev: is Dir
2024/03/30 11:02:09 DEBUG : rsync test: path2 only
2024/03/30 11:02:09 DEBUG : rsync test: is Dir

2024/03/30 10:59:17 DEBUG : Lock file removed: /home/salty/.cache/rclone/bisync/idrive-sync-crypt_..local__home_salty_sync.lck
2024/03/30 10:59:17 INFO  : Bisync successful
2024/03/30 10:59:17 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:             13158 / 13158, 100%
Elapsed time:       1m3.7s

2024/03/30 10:59:17 DEBUG : 16 go routines active
2

reverse your filter rules and exclude everything else:

+ .remmina/**
+ **~lock**
+ **.tmp**
+ **.log**
+ .Trash-1000/**
- *

and now you can list it:

rclone lsf ~/sync --recursive --filter-from=/home/salty/.config/rclone/filters
1 Like
3

Thanks.

This output 3052 lines of mostly directories so I edited it to this:

rclone lsf ~/sync --recursive --filter-from=/home/salty/.config/rclone/filters_test --files-only

Which returned 34 lines, 2 locks and 34 logs.

Is there some other way we could identify the 600 items not being synced? I thought it may be something like Gnome Files counts files from some symlink inside sync on my desktop but not laptop, or my bisync command might be missing some argument to handle certain types of files.

4

As you already have local and remote sorted lists use comm command:

comm -23 local_files_sorted remote_files_sorted

Will print files present in your local file but not in the remote one.

1 Like
5

Those commands generated a list of 6609 files for some reason.

I realised before I was able to post an edit that I could just compare local_files on my laptop to local_files on my desktop. I scanned through both files side by side with Meld and the only files highlighted from the desktop version not as missing from the laptop version matched my filters rules for lock and log.

Further wc -l also confirms the difference in the 2 files is 34, matching the number generated when excluding.

It seems I should not be using the number of 'items' Gnome Files / Nautilus lists a directory having as a barometer of whether a bisync was successful or is complete.

Thanks for your help.

6

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.