Bisync of folders with long paths unables to create a lock file. Unfortunately, I haven't found any information about the manual naming of it. Is it possible? Or maybe it is better to name the lock file with hashes instead of full paths?
Run the command 'rclone version' and share the full output of the command.
rclone v1.66.0
os/version: linuxmint 20.3 (64 bit)
os/kernel: 5.4.0-173-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.1
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Dropbox
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone bisync dropbox:"/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff" "/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]" --create-empty-src-dirs --ignore-listing-checksum -vvv
2024/03/12 14:52:47 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/home/user/opt/rclone" "bisync" "dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff" "/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]" "--create-empty-src-dirs" "--ignore-listing-checksum" "-vvv"]
2024/03/12 14:52:47 DEBUG : Creating backend with remote "dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff"
2024/03/12 14:52:47 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/03/12 14:52:49 DEBUG : Dropbox root '': Using root namespace "*********"
2024/03/12 14:52:50 DEBUG : fs cache: renaming cache item "dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff" to be canonical "dropbox:New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff"
2024/03/12 14:52:50 DEBUG : Creating backend with remote "/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]"
2024/03/12 14:52:50 NOTICE: bisync is IN BETA. Don't use in production!
2024/03/12 14:52:50 INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"HashType1": 0,
"HashType2": 0,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"SlowHashDetected": true,
"DownloadHash": false
}
2024/03/12 14:52:50 ERROR : /home/user/.cache/rclone/bisync/dropbox_New_SomeCompany_Team_2024_2024_S_2024_S_Something_Primary_School_Individuals_Groups_Staff_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff..home_user_temp_SomeCompany_dropbox_editors_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff_[63].lck: error reading lock file: open /home/user/.cache/rclone/bisync/dropbox_New_SomeCompany_Team_2024_2024_S_2024_S_Something_Primary_School_Individuals_Groups_Staff_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff..home_user_temp_SomeCompany_dropbox_editors_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff_[63].lck: file name too long
The lock file filename is derived from the bisync Session Name, which is also used for the listing files. It includes both full paths to guarantee uniqueness (which would not be guaranteed with truncated versions.) There's not currently a way to customize it, although I like the idea of being able to specify a custom session name, and I think that could fit nicely into the upcoming Sessions project.
In the meantime, some possible solutions are:
Use alias remotes to refer to these long paths by shorter names. For example:
[staff]
type = alias
remote = /home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]
Bisync from higher up in the tree, and use a --filters-file as needed to include/exclude certain directories. (This is what I do.)
This is an interesting idea, although the downside is that it would make these filenames a lot less readable to humans. I think I prefer the user-provided session name option, but perhaps the hash option could be added with a flag if there is significant user interest!
Thanks for your reply!
Unfortunately, aliases convert to the full path while bisync:
[remote]
type = alias
remote = dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff
[local]
type = alias
remote = /home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]
And I'm trying to use it:
rclone bisync "remote:" "local:" --create-empty-src-dirs --ignore-listing-checksum -vvv
2024/03/13 11:59:08 DEBUG : rclone: Version "v1.66.0" starting with parameters ["/home/user/opt/rclone" "bisync" "remote:" "local:" "--create-empty-src-dirs" "--ignore-listing-checksum" "-vvv"]
2024/03/13 11:59:08 DEBUG : Creating backend with remote "remote:"
2024/03/13 11:59:08 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2024/03/13 11:59:08 DEBUG : Creating backend with remote "dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff"
2024/03/13 11:59:08 DEBUG : dropbox: Loaded invalid token from config file - ignoring
2024/03/13 11:59:08 DEBUG : Saving config "token" in section "dropbox" of the config file
2024/03/13 11:59:08 DEBUG : dropbox: Saved new token in config file
2024/03/13 11:59:10 DEBUG : Dropbox root '': Using root namespace "**********"
2024/03/13 11:59:11 DEBUG : fs cache: renaming cache item "dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff" to be canonical "dropbox:New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff"
2024/03/13 11:59:11 DEBUG : fs cache: renaming cache item "remote:" to be canonical "dropbox:New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff"
2024/03/13 11:59:11 DEBUG : Creating backend with remote "local:"
2024/03/13 11:59:11 DEBUG : Creating backend with remote "/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]"
2024/03/13 11:59:11 DEBUG : fs cache: renaming cache item "local:" to be canonical "/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]"
2024/03/13 11:59:11 NOTICE: bisync is IN BETA. Don't use in production!
2024/03/13 11:59:11 INFO : Bisyncing with Comparison Settings:
{
"Modtime": true,
"Size": true,
"Checksum": false,
"HashType1": 0,
"HashType2": 0,
"NoSlowHash": false,
"SlowHashSyncOnly": false,
"SlowHashDetected": true,
"DownloadHash": false
}
2024/03/13 11:59:11 ERROR : /home/user/.cache/rclone/bisync/dropbox_New_SomeCompany_Team_2024_2024_S_2024_S_Something_Primary_School_Individuals_Groups_Staff_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff..home_user_temp_SomeCompany_dropbox_editors_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff_[63].lck: error reading lock file: open /home/user/.cache/rclone/bisync/dropbox_New_SomeCompany_Team_2024_2024_S_2024_S_Something_Primary_School_Individuals_Groups_Staff_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff..home_user_temp_SomeCompany_dropbox_editors_Edited_Groups_2024_S_Something_Primary_School_Individuals_Groups_Staff_[63].lck: file name too long
Yes, I tried to use only "local:" and only "remote:" alias, but the paths are still full.
The solution with the "--filters-file" key works well, thanks!
This is an interesting idea, although the downside is that it would make these filenames a lot less readable to humans
Maybe, it's better if filenames can be less readable for humans in some cases, but the bisync will work? I mean, if the result filename is very long rclone just shorters it with hashes. Just an idea, I'm not a professional programmist, sorry.
The solution with the "--filters-file" key works well, thanks!
I'm so sorry, but it doesn't work either, I checked the wrong folder pair. Maybe I do something wrong? I created a filter file:
+ /New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff/**
- /**
I tried to make a bisync with
rclone bisync dropbox:/ "/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]" --filters-file filter.txt
and it creates the full Dropbox's root structure in the destination folder, like /home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff
which is logical. So, I can't mark it as a solution. As I understand, there is no other way except
Hmm, that sounds like a bug! I will look into this further. In the meantime, a possible workaround is a combine remote, since it should internally have a different canonical name (which Bisync uses for the Session Name). Try something like:
[remote]
type = combine
upstreams = remote=dropbox:/New SomeCompany Team/2024/2024_S/2024_S_Something Primary School Individuals Groups Staff/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff
[local]
type = combine
upstreams = local=/home/user/temp/SomeCompany/dropbox_editors/Edited_Groups_2024_S_Something Primary School Individuals Groups Staff_[63]
rclone bisync remote:remote local:local
(You could also rename the upstreams something else.)
Just to follow up on this: I looked into it more and it looks like alias is functioning as designed, and I was just wrong to suggest it as a solution here
The technical explanation (if you're interested) is that alias is not truly a "wrapping" backend like combine, crypt, hasher, etc. -- you can think of it more as a "redirect" or shortcut to another remote. As a result, the Fs that bisync receives is the real dropbox remote, not an alias remote wrapping a dropbox remote. In contrast, if combine is used, bisync receives a combine remote wrapping a dropbox remote, which is why combine is really the better bet for this particular use case.
All of that said, I definitely agree that this kind of hacky workaround shouldn't be necessary to use bisync with long paths, and I will plan to address this as part of the "sessions" project, most likely by adding an option to specify a custom --session some_custom_name (for example.)
It's possible that --session hash could additionally be added as a shortcut to automatically hash the default name into a shorter md5. I would be interested in your feedback (same for any other bisync users who might be reading this!) about whether you would prefer this option to --session some_short_name, and if so why. (For the purposes of the question, assume that there is already some persistent config to store the long->short name mapping and other session settings, so doing it statelessly is not a requirement.)
First of all, thank you so much, the combine method works like a charm! I very much appreciate your help!
I would be interested in your feedback (same for any other bisync users who might be reading this!) about whether you would prefer this option to --session some_short_name , and if so why
Well, I think that both variants with the session name and session hash will be fine. Additionally, an error message about a very long path with a suggestion should be very helpful (like in cases where the '--resync' key is necessary).
I can just assume that some people can use rclone for their automation scripts (like me) or another software and need to work with session files. In this case, two new keys, for example, --session-hash and --session-name <name> will be perfect. Or just --session <name> that uses hashes if the <name> is 'hash' / '_hash' / '_hash_' / etc, let's say.
Can I mark your answer as a solution? I very rarely create bug reports and I'm not sure in this case, because the solution has been found, but the bug generally exists.
