Bisync not working

kimchijuice · November 7, 2023, 8:20am

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I've been using this script to sync my Obsidian on my computer to my phone, and it was working pretty well for a while. Then, I've noticed that it hasn't been syncing my files like it normally do, so i decided to run the script on my terminal to see for any errors.

Here's the error whenever I run the script:

sync
2023/11/07 15:54:23 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/07 15:54:23 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/07 15:54:23 ERROR : Bisync aborted. Must run --resync to recover.
2023-11-07 07:54:24 Loading updated app package /home/nimby/snap/obsidian/x1/.config/obsidian/obsidian-1.4.16.asar
2023-11-07 07:54:24 Checking for update using Github
2023-11-07 07:54:24 Success.
2023-11-07 07:54:24 Latest version is 1.4.16
2023-11-07 07:54:24 App is up to date.
2023/11/07 15:54:34 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/07 15:54:34 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/07 15:54:34 ERROR : Bisync aborted. Must run --resync to recover.

I've already tried running the --resync command again and it works fine. But for some reason when I try to run the script, it always give the same error. "cannot find prior Path1 or Path2 listings, likely due to critical error on prior run"

I've also run this script rclone bisync ~/Documents/rclone remote:rclone --remove-empty-dirs --resync -vv --log-file=rclone.log and this is the result: 2023/11/07 16:01:46 DEBUG : rclone: Version "v1.64.2" starting with parameters [ - Pastebin.com

Tried running the script again and I still get the same error.

I honestly don't know what's happening as this is all new to me. I'm just trying to build my own Obsidian vault with sync involved.

PS. Im using the same exact script as the script on the link above. I didn't change anything, I also did the troubleshooting mentioned on this script, and none have worked.

Any suggestion?

Run the command 'rclone version' and share the full output of the command.

yes

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone bisync ~/Documents/rclone remote:rclone --remove-empty-dirs &

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[remote]
type = drive
scope = drive
token = XXX
team_drive = 
### Double check the config for sensitive info before posting publicly

A log from the command that you were trying to run with the `-vv` flag

rclone bisync ~/Documents/rclone remote:rclone --remove-empty-dirs & -vv
[1] 25475
-vv: command not found

kapitainsky · November 7, 2023, 4:13pm

You have only posted log for rclone bisync --resync - and it looks like this one was OK.

What is missing is log file for normal bisync (without --resync).

Please run:

rclone bisync ~/Documents/rclone remote:rclone --remove-empty-dirs -vv --log-file=rclone_bisync.log

And share the log.

nielash · November 8, 2023, 1:20am

In your bisync command, your Path1 is:

~/Documents/rclone

but in your --resync log it is:

/home/nimby/Documents/rclone

I realize that the first is likely an alias for the second, but I'm not sure if bisync realizes that. If I'm right, it is creating a listing file during --resync that it is unable to find later because the path has changed. (Bisync uses the path for the session name in the listing filename.) I think either version should work, as long as you pick one and use it consistently.

I think this is because the ampersand (&) shouldn't be there. Try removing it?

kimchijuice · November 8, 2023, 2:34am

I've ran this code. This is the output:

2023/11/08 10:33:14 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "bisync" "/home/nimby/Documents/rclone" "remote:rclone" "--remove-empty-dirs" "-vv" "--log-file=rclone_bisync.log"]
2023/11/08 10:33:14 DEBUG : Creating backend with remote "/home/nimby/Documents/rclone"
2023/11/08 10:33:14 DEBUG : Using config file from "/home/nimby/.config/rclone/rclone.conf"
2023/11/08 10:33:14 DEBUG : Creating backend with remote "remote:rclone"
2023/11/08 10:33:14 DEBUG : remote: Loaded invalid token from config file - ignoring
2023/11/08 10:33:15 DEBUG : Saving config "token" in section "remote" of the config file
2023/11/08 10:33:15 DEBUG : remote: Saved new token in config file
2023/11/08 10:33:16 DEBUG : Google drive root 'rclone': 'root_folder_id = 0ALeNXI_NhhKQUk9PVA' - save this in the config to speed up startup
2023/11/08 10:33:17 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/08 10:33:17 DEBUG : Lock file created: /home/nimby/.cache/rclone/bisync/home_nimby_Documents_rclone..remote_rclone.lck
2023/11/08 10:33:17 INFO  : Synching Path1 "/home/nimby/Documents/rclone/" with Path2 "remote:rclone/"
2023/11/08 10:33:17 DEBUG : Lock file removed: /home/nimby/.cache/rclone/bisync/home_nimby_Documents_rclone..remote_rclone.lck
2023/11/08 10:33:17 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/08 10:33:17 ERROR : Bisync aborted. Must run --resync to recover.

kimchijuice · November 8, 2023, 2:40am

I've already tried changing the script to this path /home/nimby/Documents/rclone but for some reason I'm still getting the same error.

New script:

#!/usr/bin/env bash

# to simplify: remove this function
notify() {
  notify-send -i $1 "Obsync" "$2"
}

obsync() {
  # start the synchronization process
  rclone bisync /home/nimby/Documents/rclone remote:rclone --remove-empty-dirs &

  # to simplify: remove this loop (incl. the "&" above)
  while [[ -n $(jobs -r) ]]; do
    notify $1 "$2"
    sleep 4
  done
}

# to simplify: replace everything below with "obsync && obsidian && obsync"
obsync process-working "Preparing to launch Obsidian..."
obsidian
obsync drive-harddisk "Synchronizing with Google Drive..."
notify face-wink "Synchronization successful."

Error message from terminal:

6:08 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/08 10:36:08 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/08 10:36:08 ERROR : Bisync aborted. Must run --resync to recover.
2023-11-08 02:36:14 Loading updated app package /home/nimby/snap/obsidian/x1/.config/obsidian/obsidian-1.4.16.asar
2023-11-08 02:36:15 Checking for update using Github
2023-11-08 02:36:15 Success.
2023-11-08 02:36:15 Latest version is 1.4.16
2023-11-08 02:36:15 App is up to date.
[5726:1108/103624.127817:ERROR:gl_surface_presentation_helper.cc(260)] GetVSyncParametersIfAvailable() failed for 1 times!
[5726:1108/103639.320151:ERROR:gl_surface_presentation_helper.cc(260)] GetVSyncParametersIfAvailable() failed for 2 times!
[5726:1108/103815.651933:ERROR:gl_surface_presentation_helper.cc(260)] GetVSyncParametersIfAvailable() failed for 3 times!
2023/11/08 10:38:18 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/08 10:38:18 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/08 10:38:18 ERROR : Bisync aborted. Must run --resync to recover.

Obsidian launches perfectly and syncs any changes coming from the Google drive. But when I close the app, the changes I made on the local file doesn't upload to the Google drive. It only does when I run --resync.

nielash · November 8, 2023, 3:29am

Ok. A few things we can try to narrow this down:

Firstly, let's take a look at your bisync cache directory -- this will shed light on whether the listing file exists, and what it is named. Try this command and share the output:

rclone lsl /home/nimby/.cache/rclone/bisync

Secondly, it would be helpful if we could try running the rclone commands manually by themselves (without the rest of the bash script), to see if the problem is in rclone or in the script. Try running these commands, in this order, and share the whole output (including the command itself would be very helpful):

rclone bisync /home/nimby/Documents/rclone remote:rclone --remove-empty-dirs -vv --resync

rclone lsl /home/nimby/.cache/rclone/bisync

rclone bisync /home/nimby/Documents/rclone remote:rclone --remove-empty-dirs -vv

rclone lsl /home/nimby/.cache/rclone/bisync

kimchijuice · November 13, 2023, 12:57am

Thanks, I've ran these codes. Here are the outputs:

rclone lsl /home/nimby/.cache/rclone/bisync
    28803 2023-11-07 16:03:57.398040810 home_nimby_Documents_rclone..remote_rclone.path1.lst-dry
    28803 2023-11-07 16:22:33.649597900 home_nimby_Documents_rclone..remote_rclone.path1.lst-err
    30187 2023-11-07 16:04:00.790056729 home_nimby_Documents_rclone..remote_rclone.path2.lst-dry
    30187 2023-11-07 16:22:28.249101599 home_nimby_Documents_rclone..remote_rclone.path2.lst-err

For the second string of commands, here's the output: rclone bisync /home/nimby/Documents/rclone remote:rclone --remove-empty-dirs -vv - Pastebin.com

I've tried running Obsync again (the original script) and I'm still getting the same error:

obsync
2023/11/13 08:56:39 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/13 08:56:39 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/13 08:56:39 ERROR : Bisync aborted. Must run --resync to recover.
2023-11-13 00:56:45 Loading updated app package /home/nimby/snap/obsidian/x1/.config/obsidian/obsidian-1.4.16.asar
2023-11-13 00:56:45 Checking for update using Github
2023-11-13 00:56:46 Success.
2023-11-13 00:56:46 Latest version is 1.4.16
2023-11-13 00:56:46 App is up to date.
[7753:1113/085652.239962:ERROR:gl_surface_presentation_helper.cc(260)] GetVSyncParametersIfAvailable() failed for 1 times!
2023/11/13 08:57:03 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023/11/13 08:57:03 ERROR : Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run
2023/11/13 08:57:03 ERROR : Bisync aborted. Must run --resync to recover.

Thanks for all your help! Sorry for the late reply I wasn't online for the past few days.

nielash · November 13, 2023, 2:53am

Thank you, that was very helpful! I see the problem now. There is a duplicate directory Kim/.trash in Google Drive, and it is being inconsistently ignored by bisync.

The log shows it gets ignored during --resync:

2023/11/13 08:53:15 NOTICE: Kim/.trash: Duplicate directory found in destination - ignoring

but not during --check-sync, which is why the non-resync fails:

2023/11/13 08:53:43 ERROR : -          Path2 file not found in Path1       - Kim/.trash/[].md
2023/11/13 08:53:43 ERROR : -          Path2 file not found in Path1       - Kim/.trash/Untitled.canvas
2023/11/13 08:53:43 DEBUG : Lock file removed: /home/nimby/.cache/rclone/bisync/home_nimby_Documents_rclone..remote_rclone.lck
2023/11/13 08:53:43 ERROR : Bisync critical error: path1 and path2 are out of sync, run --resync to recover

(Path1 has only one copy of this folder, and --check-sync is noticing.)

I opened a PR last week that would fix the underlying bug, so a proper solution should hopefully be coming before too long! In the mean time, I can suggest a few options to work around this:

Remove that duplicate folder, either by deleting/renaming it or filtering it out of your bisync run. If you choose to delete/rename, there is a special rclone dedupe command for this which can be quite helpful!
If you don't want to delete/exclude it, you could alternatively use --check-sync=false to disable the check function that is complaining about it.
I'll also add... if you're interested in beta-testing the new version I'm working on where this is fixed (along with many other things), there is a preview available here. A few disclaimers: this is just an early preview, not yet reviewed or tested by anyone but me. It may still have bugs, and will probably change somewhat before it is released! That said, feel free to try it out if you wish (feedback is welcome!)

Hope that helps!

kimchijuice · November 13, 2023, 11:10pm

OMG! It works. I've deleted that folder and everything is working now. Thank you so much!

I'll have a look at this version and will use it for this script. I'll let you know if run into problems or if I have any feedback.

Thanks again for your help!

system · November 16, 2023, 11:11pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.