Google Drive: directories with shortcuts cannot be removed via a mounted remote

Suspected Bug
21

disregard the link above. I can't recreate it with drive-use-trash=false but I can recreate it without it.

It looks to me like when rclone is looking to see if that directory is empty, that it is looking up the id of the shortcut target instead of the shortcut itself. Its finding that 'file' still exists and fails to remove it.

@dell-rob:~$ cd test
@dell-rob:~/test$ rm -rd d2
rm: cannot remove 'd2': Input/output error
@dell-rob:~/test$ rm d1/file
@dell-rob:~/test$ rm -rf d2
@dell-rob:~/test$

IF i delete the original file then I cant delete the directory that contained teh shortcut.

22

I think I see the problem... When we resolve the shortcut we lose the fact that it is trashed

I think this should fix it - can you give it a go?

https://beta.rclone.org/branch/v1.52.1-057-g2832d298-fix-drive-shortcut-beta/ (uploaded in 15-30 mins)

23

Wasn't OP use case deleting the shortcut though and not the original?

Or do I have something backwards?

24

No you had it right. I was saying that the code looked to have been checking the ID of the target though rather than the shortcut. By deleting the source of the shortcut in d1, then only was i able to delete the directory that contained the shortcut d2.

25

But why does the problem only correct itself when I delete the shortcut source... unmounting/remounting doesnt even help.

26

When you delete the source the shortcut becomes unresolvable and rclone just sees it as a trashed file.

I'm pretty sure the above will fix it... (when it finished building!)

1 Like
27

fixed it for me. @1xe ?

1 Like
28

Fixed it for me as well! Excellent work guys. :slight_smile:

1 Like
29

I've merged the fix to the latest beta now.

Thanks for testing!

I'll put it in a 1.52.2 at some point.

30

Glad to help!

I'll be using the new shortcuts feature myself. I wish it was more convenient to use with encryption, but I guess I can put together a shell script to help with that until native support improves, which I'm sure it will. :wink:

Anyway, as an aside, I've discovered that shortcuts cannot be created in an Application Data folder:

2020/06/12 14:42:18 Failed to backend: command "shortcut" failed: shortcut creation failed: googleapi: Error 403: Method not supported for files within the Application Data folder., notSupportedForAppDataFolderFiles

Trying to move data out of the Application Data folder fails with:

2020/06/12 15:23:14 ERROR : [...]: Couldn't move: googleapi: Error 403: Method not supported for files within the Application Data folder., notSupportedForAppDataFolderFiles

Fortunately, copy/sync works (and server-side), at least for test data sizes, so eventually I'll probably want to run something like

rclone sync drive-with-appfolder: drive-without-appfolder: --create-empty-src-dirs --progress

to get my around 15TB of data out of the Application Data folder, to use the shortcuts feature. Any gotchas I should be aware of?

31

A shell script should be easy to build with the backend command in the crypt backend to translate the names.

I didn't know about the limitations of the appfolder.

You'll probably want --server-side-across-configs but I see no reason why it shouldnt work if the big G allows it.

32

I'll look into that!

Great! I'll fire up tmux and hope for the best. :slight_smile:

33 
#!/bin/bash

if [ "$#" -ne 3 ]; then
    echo Syntax:
    echo " $0 remote src dest"
    echo "Example:"
    echo " $0 robgs-cryptp: \"/Media/Videos/Movies/Onward.(2020).2160p.webdl.h265\"
\"/Media/Videos/KidsMovies/Onward.(2020).2160p.webdl.h265\""
    exit 1
fi

REMOTE=${1%":"}
SRC=${2#"/"}
SRC=${SRC%"/"}
DES=${3#"/"}
DES=${DES%"/"}
CRY=`rclone config show $REMOTE | grep remote | awk -F" = " '{print $2}'`
BASEREMOTE=`echo "$CRY"| awk -F: '{print $1}'`
BASEPATH=`echo "$CRY"| awk -F: '{print $2}'`
ENCS=`rclone backend encode $REMOTE: "$SRC"`
ENCT=`rclone backend encode $REMOTE: "$DES"`
CMD="rclone backend shortcut $BASEREMOTE: $BASEPATH/$ENCS $BASEPATH/$ENCT"
echo $CMD
sleep 2
$CMD
1 Like
34

Thanks! I'll probably end up making my own, but yours is a start. :slight_smile:

35

Nice script!

I had an idea for a wrapper command for the crypt backend

rclone backend cryptrun crypt: command param1 param2

So what that would do is run this command for you on the backend crypt is wrapping

rclone backend command wrappedbackend: encrypted(param1) encrypted(param2)

That would certainly work for the drive shortcuts. I'm not sure it is general purpose enough or worth doing - what do you think?

rclone backend cryptrun crypt: shortcut path/to/a path/to/b
36

Maybe you can make something more general-purpose with "aliases" and "templates", e.g.,

# add alias "shortcut"
rclone alias --add shortcut "backend shortcut %(config --get remote %1) %(backend encode %1 %2) %(backend encode %1 %3)"
# remove alias "shortcut"
rclone alias --remove shortcut
# use alias "shortcut"
rclone alias shortcut crypt: file shortcut

?

Anyway, here's another shell script:

#!/bin/bash

# assume crypt remote CRYPT_REMOTE mounted at MOUNTPOINT with drive remote DRIVE_REMOTE
DRIVE_REMOTE="drive:rclone"
CRYPT_REMOTE="crypt:"
MOUNTPOINT="/mnt/remote"

print_usage() {
  cat <<EOF
Usage: shortcut SOURCE DESTINATION
  or:  shortcut SOURCE... DIRECTORY
Shortcut SOURCE to DESTINATION, or multiple SOURCE(s) to DIRECTORY.

See the top of the script for configuration parameters.
EOF
  exit 0
}

die() {
  echo "$@" >&2
  exit 1
}

convert_path() {
  crypt_path="$1"

  drive_path=($(IFS=/ eval rclone backend encode '"$CRYPT_REMOTE"' '$crypt_path'))

  IFS=/ eval echo '"${drive_path[*]}"'
}

[ $# -eq 0 ] && print_usage
[ $# -lt 2 ] && die "error: two or more arguments required"

for i in "${@:1:$#-1}"; do
  tmp="$(readlink -m "$i")"
  if [ ! -e "$tmp" ] || [ "$tmp" = "${tmp#$MOUNTPOINT}" ]; then
    die "error: $i does not exist or is not within \$MOUNTPOINT"
  fi
done

tmp="$(readlink -m "${!#}")"
if [ "$tmp" = "${tmp#$MOUNTPOINT}" ]; then
  die "error: ${!#} is not within \$MOUNTPOINT"
fi

if [ $# -eq 2 ]; then
  if [ -f "$2" ]; then
    die "error: $2 exists already"
  fi
  tmp="${2%/}/$(basename "$1")"
  if [ -f "$1" ] && [ -d "$2" ] && [ -f "$tmp" ]; then
    die "error: $tmp exists already"
  fi
  tmp="$(dirname "$2")"
  if [ ! -d "$tmp" ]; then
    die "error: $tmp does not exist or is not a directory"
  fi
else
  if [ ! -d "${!#}" ]; then
    die "error: ${!#} does not exist or is not a directory"
  fi
fi

# normalize, just in case
DRIVE_REMOTE="${DRIVE_REMOTE%/}"
CRYPT_REMOTE="${CRYPT_REMOTE%/}"
MOUNTPOINT="${MOUNTPOINT%/}"

rename_mode="false"
if [ $# -eq 2 ] && [ ! -e "$2" ]; then
  # special case: shortcut-rename file/directory
  rename_mode="true"
fi

dst_top_path="$(readlink -f "${!#}")"
dst_path="$dst_top_path"

for i in "${@:1:$#-1}"; do
  src_top_path="$(readlink -e "$i")"
  [ "$rename_mode" = "false" ] && dst_path="$dst_top_path/$(basename "$src_top_path")"

  while read -r path; do
    src_path="$(readlink -e "$path")"
    tmp="${src_path#$src_top_path}"
    shared_path="${tmp#/}"

    if [ -d "$src_path" ]; then
      #echo mkdir "${dst_path}${shared_path:+/$shared_path}"
      mkdir "${dst_path}${shared_path:+/$shared_path}"
      #echo touch -r "${src_path}" "${dst_path}${shared_path:+/$shared_path}"
      touch -r "${src_path}" "${dst_path}${shared_path:+/$shared_path}"
    elif [ -f "$src_path" ]; then
      src_crypt_path="${src_path#$MOUNTPOINT/}"
      src_drive_path="$(convert_path "$src_crypt_path")"
  
      dst_crypt_path="${dst_path#$MOUNTPOINT/}${shared_path:+/$shared_path}"
      dst_drive_path="$(convert_path "$dst_crypt_path")"
  
      #echo rclone backend shortcut "$DRIVE_REMOTE" "[$src_crypt_path]" "[$dst_crypt_path]"
      #echo rclone backend shortcut "$DRIVE_REMOTE" "$src_drive_path" "$dst_drive_path"
      rclone backend shortcut "$DRIVE_REMOTE" "$src_drive_path" "$dst_drive_path" >/dev/null
    fi
  done < <(find "$i")
done

I modelled it on "cp -a", except that it shortcuts files instead of copying them and only preserves modification times (for directories). It never creates shortcuts to directories. You can use it like this, assuming /mnt/remote is your crypt: mountpoint.

The start state:

# cd /mnt/remote
# find
.
./archive
./archive/linux-distros
./archive/linux-distros/debian
./archive/linux-distros/debian/buster.iso
./archive/linux-distros/devuan
./archive/linux-distros/devuan/ascii.iso
./archive/linux-distros/devuan/beowulf.iso

Shortcut archive/linux-distros, keeping linux-distros as the destination name:

# shortcut archive/linux-distros/ .
# find
.
./archive
./archive/linux-distros
./archive/linux-distros/debian
./archive/linux-distros/debian/buster.iso
./archive/linux-distros/devuan
./archive/linux-distros/devuan/ascii.iso
./archive/linux-distros/devuan/beowulf.iso
./linux-distros
./linux-distros/debian
./linux-distros/debian/buster.iso
./linux-distros/devuan
./linux-distros/devuan/ascii.iso
./linux-distros/devuan/beowulf.iso

Shortcut archive/linux-distros/devuan, using my-linux-distro as the destination name:

# shortcut archive/linux-distros/devuan/ my-linux-distro
# find
.
./archive
./archive/linux-distros
./archive/linux-distros/debian
./archive/linux-distros/debian/buster.iso
./archive/linux-distros/devuan
./archive/linux-distros/devuan/ascii.iso
./archive/linux-distros/devuan/beowulf.iso
./linux-distros
./linux-distros/debian
./linux-distros/debian/buster.iso
./linux-distros/devuan
./linux-distros/devuan/ascii.iso
./linux-distros/devuan/beowulf.iso
./my-linux-distro
./my-linux-distro/ascii.iso
./my-linux-distro/beowulf.iso

Shortcut archive/linux-distros/devuan/beowulf.iso, keeping beowulf.iso as the destination name:

# shortcut archive/linux-distros/devuan/beowulf.iso .
# find
.
./archive
./archive/linux-distros
./archive/linux-distros/debian
./archive/linux-distros/debian/buster.iso
./archive/linux-distros/devuan
./archive/linux-distros/devuan/ascii.iso
./archive/linux-distros/devuan/beowulf.iso
./beowulf.iso
./linux-distros
./linux-distros/debian
./linux-distros/debian/buster.iso
./linux-distros/devuan
./linux-distros/devuan/ascii.iso
./linux-distros/devuan/beowulf.iso
./my-linux-distro
./my-linux-distro/ascii.iso
./my-linux-distro/beowulf.iso

Shortcut archive/linux-distros/devuan/beowulf.iso, using linux.iso as the destination name:

# shortcut archive/linux-distros/devuan/beowulf.iso linux.iso
# find
.
./archive
./archive/linux-distros
./archive/linux-distros/debian
./archive/linux-distros/debian/buster.iso
./archive/linux-distros/devuan
./archive/linux-distros/devuan/ascii.iso
./archive/linux-distros/devuan/beowulf.iso
./beowulf.iso
./linux-distros
./linux-distros/debian
./linux-distros/debian/buster.iso
./linux-distros/devuan
./linux-distros/devuan/ascii.iso
./linux-distros/devuan/beowulf.iso
./linux.iso
./my-linux-distro
./my-linux-distro/ascii.iso
./my-linux-distro/beowulf.iso

You can shortcut any number of files and directories at the same time, e.g., instead of the initial

# shortcut archive/linux-distros/ .

you could have run

# mkdir linux-distros
# shortcut archive/linux-distros/debian archive/linux-distros/devuan linux-distros

The script is not very fast, but it does work. I considered using the async rc interface, but then bash probably wouldn't be the right tool for the job.

Now I need some way to determine whether a file is a shortcut or not. Without it, shortcuts are dangerous because you cannot tell whether you're editing a copy of a file or the one original through a shortcut.

Perhaps what's needed is a drive-specific version of ls* that returns this piece of information?

37

Damn. My sync just hit Error 403: User rate limit exceeded., userRateLimitExceeded. I naively thought that wouldn't be a problem when doing a server-side copy/sync, but apparently I was wrong. I guess I'll be syncing for the next 3 weeks. :man_facepalming:

Does rclone sync know to pause now and try again in 24 hours? Or do I need to stop the process and retry manually later?

38

You'd need to restart it or you could just limit it to a bwlimit that will keep it under 750G. Like --bwlimit=8M.

39

Hmm, my sync process actually seems to be making progress from time to time. It's very slow, though. Are the Google limits reset gradually or at once?

Anyway, I like the idea of "slow and steady" progress better, so I'll restart with a bwlimit set. Thanks!

40

The are reset at once in my experience.