I am syncing folders from my Synology NAS to the cloud but for some reason Synology in their wisdom decided to add a hidden folder called @eadir to all subfolders. It contains extra attributes and other small files and I would like to exclude @eadir found in any path. Is that possible? I know I can exclude a specific folder from reading the Rclone Filtering but I was now sure how to apply that to every instance of a folder in all directories/paths
1 Like
Do not anchor this exclusion anywhere and then any occurrence should be excluded:
--filter "- @eadir/**"
1 Like
Thanks very much. I realised I was I had misspelled it, @eaDir is the correct spelling, not @eadir
1 Like
Spoke too soon. I tested my command on a single folder and a few things happened that I did not expect.
rclone bisync Chunk:Archive/ /volume1/Archive/ --exclude "- @eaDir/**" --create-empty-src-dirs --compare size --slow-hash-sync-only --resilient -MvP --fix-case --resync -vv
2024/07/10 12:15:25 DEBUG : decision: copied to src: @eaDir/theme.mp3/SYNOINDEX_MEDIA_INFO
First thing was that it did still copy over the @eaDir folder
Second thing was that things only synced in one direction, ie the @eaDir folder copied to the cloud but the contents of the Cloud folder did not sync back to the NAS.
Read bisync documentation. You are supposed to use --resync flag only once to initialise all setup.
Are you sure that bi-directorial sync is what you want?
Secondly to solve your exclusion issues pick up some small directory and run your command with -vv option and post log here. Doing it from the start would save time and effort - as you talk about sync then use bisyncs etc.
Sorry I was not aware there would be a difference between sync and bisync
Yes I need bisync, I have the two folders and there is data missing on both that needs to be on the other also so I want them to be mirrored eventually.
Yes I ran the command (with --resync) only once on the path but it is a lot of data so I will probably need to sync it on and off over a few months. Does that mean I need to run it with --resync each time I start it again or only the first time even if it does not finish?
This is interesting case:) As --resync should finish before you start using your bisync. @nielash could you comment here?
As per your exclusion not working please provide log file. Then it will be 100% sure what you are doing as for example --exclude "- @eaDir/**" is not correct flag. It should be --exclude "@eaDir/**". There is no point guessing:) Let's see some data.
1 Like
Thanks will do, I am syncing another folder at the moment so waiting for that to finish to run another test. I tried both "- @eaDir/**" and "@eaDir/**" previously but neither worked.
If you could try with some limited dataset it would be helpful. Less data, shorter log to look at.
For quick test please run your command with -vv --dump filters --dry-run flags. And post only beginning of the log. It will tell us how rclone understands your filtering rules.
rclone bisync Chunk:Archive/HomeVideos/Holiday1 /volume1/Media/Archive/HomeVideos/Holiday1 --exclude "- @eaDir/**" -vv --dump filters --resync --dry-run
--- start filters ---
--- File filter rules ---
- (^|/)- @eaDir/.*$
--- Directory filter rules ---
- (^|/)- @eaDir/.*$
--- end filters ---
2024/07/10 13:39:28 DEBUG : rclone: Version "v1.67.0" starting with parameters ["rclone" "bisync" "Chunk:Archive/HomeVideos/Holiday1" "/volume1/Media/Archive/HomeVideos/Holiday1" "--exclude" "- @eaDir/**" "-vv" "--dump" "filters" "--resync" "--dry-run"]
2024/07/10 13:39:28 DEBUG : Creating backend with remote "Chunk:Archive/HomeVideos/Holiday1"
2024/07/10 13:39:28 DEBUG : Using config file from "/var/services/homes/brimur/.config/rclone/rclone.conf"
2024/07/10 13:39:28 DEBUG : Creating backend with remote "Box-Enc:Archive/HomeVideos/Holiday1"
2024/07/10 13:39:29 DEBUG : Creating backend with remote "Box:dsit5a28428qsbmrifpvtog0q4/8qfh1soj61j8d136t7489h110k/27qem5j9levjkdo1fgcstvobjo"
2024/07/10 13:39:33 DEBUG : Reset feature "ListR"
2024/07/10 13:39:33 DEBUG : Creating backend with remote "/volume1/Media/Archive/HomeVideos/Holiday1"
2024/07/10 13:39:33 NOTICE: bisync is IN BETA. Don't use in production!
2024/07/10 13:39:33 INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
2024/07/10 13:39:33 INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"HashType1": 0,
"HashType2": 0,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"SlowHashDetected": true,
"DownloadHash": false
}
2024/07/10 13:39:33 DEBUG : defaulting to --resync-mode path1 as --resync is set
2024/07/10 13:39:33 INFO : Synching Path1 "Chunk:Archive/HomeVideos/Holiday1/" with Path2 "/volume1/Media/Archive/HomeVideos/Holiday1/"
2024/07/10 13:39:33 DEBUG : : updated backup-dir for Path1
2024/07/10 13:39:33 DEBUG : : updated backup-dir for Path2
2024/07/10 13:39:33 INFO : Copying Path2 files to Path1
2024/07/10 13:39:33 NOTICE: - Path2 Resync is copying files to - Path1
2024/07/10 13:39:33 DEBUG : filterSync.Opt.MinSize: off
2024/07/10 13:39:33 DEBUG : : updated backup-dir for Path1
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "/volume1/Media/Archive/HomeVideos/Holiday1/",
"Dst": "",
"Name": "Video1.mov",
"AltName": "",
"Size": 12020034,
"Modtime": "2023-07-29T05:16:43Z",
"Hash": "",
"Flags": "-",
"Sigil": 43,
"Err": null,
"Winner": {
"Obj": null,
"Side": "none",
"Err": null
},
"IsWinner": false,
"IsSrc": true,
"IsDst": false,
"Origin": "sync"
}
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "/volume1/Media/Archive/HomeVideos/Holiday1/",
"Dst": "",
"Name": "Video1.mov",
"AltName": "",
"Size": 12020034,
"Modtime": "2023-07-29T05:16:43Z",
"Hash": "",
"Flags": "-",
"Sigil": 43,
"Err": null,
"Winner": {
"Obj": null,
"Side": "none",
"Err": null
},
"IsWinner": true,
"IsSrc": false,
"IsDst": true,
"Origin": "sync"
}
2024/07/10 13:39:33 DEBUG : Video1.mov: Need to transfer - File not found at Destination
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "@eaDir",
"Dst": "",
"Name": "@eaDir",
"AltName": "",
"Size": -1,
"Modtime": "2024-05-31T17:35:11.972854761Z",
"Hash": "",
"Flags": "d",
"Sigil": 61,
"Err": {},
"Winner": {
"Obj": {
"Directory": {
"Directory": {}
}
},
"Side": "dst",
"Err": {}
},
"IsWinner": false,
"IsSrc": true,
"IsDst": false,
"Origin": "sync"
}
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "@eaDir",
"Dst": "",
"Name": "@eaDir",
"AltName": "",
"Size": -1,
"Modtime": "2024-07-10T11:46:42Z",
"Hash": "",
"Flags": "d",
"Sigil": 61,
"Err": {},
"Winner": {
"Obj": {
"Directory": {
"Directory": {}
}
},
"Side": "dst",
"Err": {}
},
"IsWinner": true,
"IsSrc": false,
"IsDst": true,
"Origin": "sync"
}
2024/07/10 13:39:33 NOTICE: Video1.mov: Skipped copy as --dry-run is set (size 1.146Mi)
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "Day 1",
"Dst": "",
"Name": "Day 1",
"AltName": "",
"Size": -1,
"Modtime": "2024-05-11T01:35:53Z",
"Hash": "",
"Flags": "d",
"Sigil": 61,
"Err": {},
"Winner": {
"Obj": {
"Directory": {
"Directory": {}
}
},
"Side": "dst",
"Err": {}
},
"IsWinner": false,
"IsSrc": true,
"IsDst": false,
"Origin": "sync"
}
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "Day 1",
"Dst": "",
"Name": "Day 1",
"AltName": "",
"Size": -1,
"Modtime": "2024-07-10T11:48:57Z",
"Hash": "",
"Flags": "d",
"Sigil": 61,
"Err": {},
"Winner": {
"Obj": {
"Directory": {
"Directory": {}
}
},
"Side": "dst",
"Err": {}
},
"IsWinner": true,
"IsSrc": false,
"IsDst": true,
"Origin": "sync"
}
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "@eaDir/Video1.mov",
"Dst": "",
"Name": "@eaDir/Video1.mov",
"AltName": "",
"Size": -1,
"Modtime": "2024-05-31T17:35:11.998854863Z",
"Hash": "",
"Flags": "d",
"Sigil": 61,
"Err": {},
"Winner": {
"Obj": {
"Directory": {
"Directory": {}
}
},
"Side": "dst",
"Err": {}
},
"IsWinner": false,
"IsSrc": true,
"IsDst": false,
"Origin": "sync"
}
2024/07/10 13:39:33 DEBUG : writing result:
{
"Src": "@eaDir/Video1.mov",
"Dst": "",
"Name": "@eaDir/Video1.mov",
"AltName": "",
"Size": -1,
"Modtime": "2024-07-10T11:46:42Z",
"Hash": "",
"Flags": "d",
"Sigil": 61,
"Err": {},
"Winner": {
"Obj": {
"Directory": {
"Directory": {}
}
},
"Side": "dst",
"Err": {}
},
"IsWinner": true,
"IsSrc": false,
"IsDst": true,
"Origin": "sync"
}
I had to include --resync because it threw an error saying path1 and path2 were missing
This is wrong syntax. See my earlier post
- or + have only use with --filter flag. With exclude they become explicit name.
1 Like
Success! I just tried with --exclude "@eaDir/**" and now I see
2024/07/10 13:53:59 DEBUG : @eaDir: Excluded
Thanks again for your help!
Now I just need to find and delete @eaDir folders I synced before I noticed they were copying over! 
1 Like
OK. So one issue fixed.
Now as it is bisync I would love to hear @nielash (who created it) comment on --resync which is run like in your case multiple times as it is impossible to finish in one go. I am not sure really how it will behave.
1 Like
I can't take credit for creating it, but I do currently maintain it 
You would definitely need to include --resync until the first time it finishes successfully. (Then omit it after that.)
Note that I don't necessarily recommend intentionally interrupting it this way... for one thing, it could cause deleted/renamed files to inadvertently reappear (because it's being copied from the other side where it still exists.) For your particular use case, you might be better off using rclone copy on one side at a time, until it's close enough that bisync --resync can finish in one go.
See also --resync-mode CHOICE for some other options.
2 Likes
Thanks guys. I appreciate the extra info.
I have run rclone check before this to find any overlap in files and there werent any thankfully but I am using --resync-mode larger just in case which us usually the better option for most recordings. Also luckily the data is just archived data and does not change so no risk of data changes between syncs.
Also saw this...
2024/07/10 18:10:26 NOTICE: No need to gracefully shutdown during --resync (just run it again.)
1 Like
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.