Git repo on GDrive remote

Fatmice · May 6, 2019, 11:43pm

I have a remote that I use to keep backups of some git repo for compiling D toolchains. Normally I will do the git pull and compilation locally then rclone sync the results for backup. Recently I tried to do git pull directly on the mounted remote just as an experiment. It seems to work but fails at the step where it needs to update some refs. Should this even work with rclone’s vfs or would I need to use overlayfs/unionfs/mergefs or one of their kins?

Here’s my mount command

REMOTE=/home/ec2-user;RCLONE_CONFIG_PASS=$(ssh salts openssl enc -d -aes-256-cbc -in "$REMOTE/<cryptedsalt>" -kfile \<\(openssl rsautl -decrypt -inkey $REMOTE/<privatekey> -in "$REMOTE/<cryptedrcloneinfos>"\) | awk '($0 ~ /^Password/){print substr($0,match($0,":")+2)}') rclone mount crypted_GDrive_remote:/ ~/eDevelopments -c -vv --vfs-read-chunk-size 4k --vfs-read-chunk-size-limit 2M --vfs-cache-mode writes --write-back-cache --dir-cache-time 48h --poll-interval 5m --buffer-size 2M --fast-list --tpslimit 8 --tpslimit-burst 2 --links --ask-password=false

[ocelot@yellowtrain dmd-test]$ git pull --verbose
error: cannot update the ref 'refs/remotes/origin/master': unable to append to '.git/logs/refs/remotes/origin/master': Bad file descriptor
From https://github.com/dlang/dmd
 ! b8c8d956f..855dbe092  master                               -> origin/master  (unable to update local ref)
 = [up to date]          dcache                               -> origin/dcache
 = [up to date]          dmd-1.x                              -> origin/dmd-1.x
 = [up to date]          dmd-cxx                              -> origin/dmd-cxx
 = [up to date]          markdown                             -> origin/markdown
 = [up to date]          newCTFE                              -> origin/newCTFE
 = [up to date]          revert-9282-globtemplate             -> origin/revert-9282-globtemplate
 = [up to date]          revert-9484-move-diagnostic-reporter -> origin/revert-9484-move-diagnostic-reporter
error: cannot update the ref 'refs/remotes/origin/stable': unable to append to '.git/logs/refs/remotes/origin/stable': Bad file descriptor
 ! 881381ea9..becd870db  stable                               -> origin/stable  (unable to update local ref)

dmd-test/ is a directory on the remote that has a copy of a git repo

calisro · May 7, 2019, 12:25am

I don’t see why it wouldn’t with vfs. Vfs writes or full mode should work.

Fatmice · May 7, 2019, 2:29am

Nope, neither worked.

calisro · May 7, 2019, 2:45am

What was the error? Might be worth producing a debug output and potentially opening an issue. Vfs full should support most file system operations.

-vv --debug-fuse

You could also use the cache backend.

Fatmice · May 7, 2019, 3:06am

Well on the git end it is error: cannot update the ref 'refs/remotes/origin/master': unable to append to '.git/logs/refs/remotes/origin/master': Bad file descriptor

On the mount end, the log says either DEBUG : fuse: -> [ID=0xf36] Read error=bad file descriptor or DEBUG : fuse: -> [ID=0xf86] Lookup error=ENOENT

Here’s the full log

gist.github.com

https://gist.github.com/Fatmice/6680d0a403ccb8fc13e947ae37308885

gistfile1.txt

[ocelot@yellowtrain ~]$ REMOTE=/home/ec2-user;RCLONE_CONFIG_PASS=$(ssh salts openssl enc -d -aes-256-cbc -in "$REMOTE/.ssh/backup/pwstores/rcloneth/backup/Factorio -in "$REMOTE/.ssh/backup/pwstores/rclonetresplaspw.enc"\) | awk '($0 ~ /^Password/){print substr($0,match($0,":")+2)}') rclone mount tresplas_Collec---vfs-read-chunk-size 4k --vfs-read-chunk-size-vfs-read-chunk-size 4k --vfs-read-chunk-sid--vfs-read-chunk-size 4k --vfs-read-chunk-sie--vfs-read-chunk-size 4k --vfs-read-chunk-sib--vfs-read-chunk-size 4k --vfs-read-chunk-siu--vfs-read-chunk-size 4k --vfs-read-chunk-sig--vfs-read-chunk-size 4k --vfs-read-chunk-size-vfs-read-chunk-size 4k --vfs-read-chunk-sif--vfs-read-chunk-size 4k --vfs-read-chunk-siu--vfs-read-chunk-size 4k --vfs-read-chunk-sis--vfs-read-chunk-size 4k --vfs-read-chunk-sie--vfs-read-chunk-size 4k --vfs-read-chunk-si --vfs-read-chunk-size 4k --vfs-read-chunk-size-limit 2M --vfs-cache-mode writes --write-back-cache --vfs-cache-max-age 48h --dir-cache-time 48h --poll-interval 5m --buffer-size 8M --fast-list --tpsli
Host key fingerprint is SHA256:q2v+dQcoZ/c5c/J1hECV/DVuZi4bBkmSl6G8iL00iqs
+---[ECDSA 256]---+
|          ..+o.. |
|        .o.=  o..|
|         o+ o ..o|
|      o . .+ . *.|
|     . =S.+ + * .|
|    . o o= . * + |
|   . . .. . o @ +|

This file has been truncated. show original

calisro · May 7, 2019, 11:28am

I would open an issue. I don’t believe you should be getting that error. The cache backend might be more appropriate for your needs anyway though.

ncw · May 7, 2019, 1:12pm

This is the problem

2019/05/06 21:56:16 DEBUG : gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master: Open: flags=O_WRONLY|O_APPEND
2019/05/06 21:56:16 DEBUG : gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master: >Open: fd=gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master (rw), err=<nil>
2019/05/06 21:56:16 DEBUG : gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master: >Open: fh=&{gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master (rw)}, err=<nil>
2019/05/06 21:56:16 DEBUG : fuse: -> [ID=0xf34] Open 0x5 fl=0
2019/05/06 21:56:16 DEBUG : fuse: <- Read [ID=0xf36 Node=0x38 Uid=1000 Gid=1000 Pid=13861] 0x5 4096 @0x0 dir=false fl=0 lock=0 ffl=OpenWriteOnly+OpenAppend
2019/05/06 21:56:16 DEBUG : &{gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master (rw)}: Read: len=4096, offset=0
2019/05/06 21:56:16 DEBUG : &{gitDToolChain/dmd-test/.git/logs/refs/remotes/origin/master (rw)}: >Read: read=0, err=bad file descriptor
2019/05/06 21:56:16 DEBUG : fuse: -> [ID=0xf36] Read error=bad file descriptor

The file is opened write only and append, but then read from. Rclone returns EBADF for that which according to the man page for write(2) is the correct thing to return

   EBADF  fd is not a valid file descriptor or is not open for reading.

Why git does that, I don’t know…

It might be worth stracing git to see if that is what it thinks it is doing or whether something has got lost in translation in the kernel/FUSE layers

Fatmice · May 7, 2019, 3:38pm

Hm, yeah I don’t get it. Git is weird… This is a strace of a git clone instead of git pull as I’ve already updated the dmd repo

[ocelot@yellowtrain gitDToolChain]$ strace -yy -o ~/gitstrace git clone --verbose https://github.com/dlang/dmd.git ~/eDevelopments/gitDToolChain/dmd-test
Cloning into '/home/ocelot/eDevelopments/gitDToolChain/dmd-test'...
fatal: Unable to read current working directory: No such file or directory

There’s clearly a dmd-test directory on that remote. I created it before issuing the git clone command

gist.github.com

https://gist.github.com/Fatmice/c91b01926d3d89886f68fae5b5f5e9ca

gistfile1.txt

[ocelot@yellowtrain gitDToolChain]$ ls
dmd  dmd-test  druntime  install  phobos  tools  update.sh

[ocelot@yellowtrain ~]$ cat gitstrace 
execve("/usr/bin/git", ["git", "clone", "--verbose", "https://github.com/dlang/dmd.git", "/home/ocelot/eDevelopments/gitDT"...], 0x7ffc9a8e9b58 /* 56 vars */) = 0
brk(NULL)                               = 0x562ca54b6000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe4cc79140) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3</etc/ld.so.cache>
fstat(3</etc/ld.so.cache>, {st_mode=S_IFREG|0644, st_size=113317, ...}) = 0

This file has been truncated. show original

gist.github.com

https://gist.github.com/Fatmice/607d4dd5377367e22cfe11ebfa6b83b6

gistfile1.txt

ocelot@yellowtrain:~[ocelot@yellowtrain ~]$ REMOTE=/home/ec2-user;RCLONE_CONFIG_PASS=$(ssh salts openssl enc -d -aes-256-cbc -in "$REMOTE/.ssh/backup/pwstores/rclonetresplaspw" -kfile \<\h/backup/Factorio -in "$REMOTE/.ssh/backup/pwstores/rclonetresplaspw.enc"\) | awk '($0 ~ /^Password/){print substr($0,match($0,":")+2)}') rclone mount tresplas_Collections_Developments:/ --debug-fuse --vfs-read-chunk-size 4k --vfs-read-chunk-size-limit 2M --vfs-cache-mode writes --write-back-cache --vfs-cache-max-age 48h --dir-cache-time 48h --poll-interval 5m --buffer-silimit 8 --tpslimit-burst 2 --links --ask-password=false
Host key fingerprint is SHA256:q2v+dQcoZ/c5c/J1hECV/DVuZi4bBkmSl6G8iL00iqs
+---[ECDSA 256]---+
|          ..+o.. |
|        .o.=  o..|
|         o+ o ..o|
|      o . .+ . *.|
|     . =S.+ + * .|
|    . o o= . * + |
|   . . .. . o @ +|

This file has been truncated. show original

So it seems that git expects a .git directory but doesn’t make one if it doesn’t exits like it would normally if cloning locally…Hm non-existent directory being open readonly is really fishy…

ncw · May 8, 2019, 9:31am

I’m not sure what is going on there! I tried the clone command to a local disk which worked fine, and then to google drive which took ages but appeared to be working…

I think there are corner cases in rclone mount which still need to be wrinkled out. For example this issue: https://github.com/ncw/rclone/issues/3006 and I guess git is hitting those, being quite an intensive file system user. If we can figure out exactly what is happening then I’m sure I can fix it!

In fact git operations are probably a good test - maybe I could make a test suite for rclone mount using them…

I note also this project which can make a git annex using rclone: https://github.com/DanielDent/git-annex-remote-rclone

system · August 6, 2019, 9:43am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.