Google Drive rclone: failed to open source object: can’t read dangling shortcut

pinkit · February 18, 2022, 9:07am

This is similar to a previous discussion of this error when rclone encounters a (broken) shortcut object that it can't follow to a destination. However, in my case the shortcuts are not actually dangling / broken, I just don't have access to the destination file. Here's a picture so you can see what I mean:

I'm using rclone to backup a shared drive used by lots of people. Several people have made use of the shortcut feature to link off to private / sensitive files that are not in the shared drive. It's a legitimate use of shortcuts. Due to the sensitive nature of the files and the fact that they're not in the shared drive, they shouldn't be copied.

The problem is that, while rclone is able to copy all the other files, it won't do the delete / cleanup step at the end, which means the backup is full of redundant data. Due to the size of the shared drive, it's not practical to wipe it and start over each time - we're talking days to do a full backup vs. minutes to do an incremental one.

Some options I've thought about:

Disabling shortcuts. I can't find a way to do this in Google Workspace Admin - after doing a few Google searches I found a couple of threads that say it's not possible.
Manually removing shortcuts, potentially temporarily for the duration of the backup. This will get harder as the number of people and shortcuts increases over time.
Posting here. I'm wondering if there's a way to catch this case of the error and either ignore it or downgrade it to a warning a la "Duplicate object found in source".

Does anyone have any other suggestions?

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
- os/version: darwin 11.6.2 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.2
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive (shared drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

./rclone sync Admin: /Volumes/Admin\ Drive -P

The rclone config contents with secrets removed.

[Admin]
type = drive
scope = drive.readonly
token = {"access_token":"***","token_type":"Bearer","refresh_token":"***","expiry":"2022-02-18T09:44:24.182131Z"}
team_drive = ***
root_folder_id =

A log from the command with the `-vv` flag

I've redacted logs for other files.

2022/02/18 08:44:58 DEBUG : rclone: Version "v1.57.0" starting with parameters ["./rclone" "sync" "Admin:" "/Volumes/Admin Drive" "-vv" "--log-file" "log.txt"]
2022/02/18 08:44:58 DEBUG : Creating backend with remote "Admin:"
2022/02/18 08:44:58 DEBUG : Using config file from "/Users/ben/.config/rclone/rclone.conf"
2022/02/18 08:44:58 DEBUG : Creating backend with remote "/Volumes/Admin Drive"
2022/02/18 08:44:59 DEBUG : README.docx: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/02/18 08:44:59 DEBUG : README.docx: Unchanged skipping
...
2022/02/18 08:45:00 NOTICE: Dangling shortcut "Events Archive" detected
2022/02/18 08:45:00 ERROR : Roles and teams/Events Team/Events Archive: Failed to copy: failed to open source object: can't read dangling shortcut
...
2022/02/18 08:46:54 NOTICE: Dangling shortcut "Events Archive" detected
2022/02/18 08:46:54 ERROR : Roles and teams/Events Team/Events Archive: Failed to copy: failed to open source object: can't read dangling shortcut
...
2022/02/18 08:48:57 NOTICE: Dangling shortcut "Events Archive" detected
2022/02/18 08:48:57 ERROR : Roles and teams/Events Team/Events Archive: Failed to copy: failed to open source object: can't read dangling shortcut
...
2022/02/18 08:51:07 DEBUG : Local file system at /Volumes/Admin Drive: Waiting for checks to finish
2022/02/18 08:51:07 DEBUG : Local file system at /Volumes/Admin Drive: Waiting for transfers to finish
2022/02/18 08:51:07 ERROR : Local file system at /Volumes/Admin Drive: not deleting files as there were IO errors
2022/02/18 08:51:07 ERROR : Local file system at /Volumes/Admin Drive: not deleting directories as there were IO errors
2022/02/18 08:51:07 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: can't read dangling shortcut
2022/02/18 08:51:07 INFO  :
Transferred:   	   30.118 MiB / 30.118 MiB, 100%, 327.253 KiB/s, ETA 0s
Errors:                 1 (retrying may help)
Checks:             32055 / 32055, 100%
Transferred:           12 / 12, 100%
Elapsed time:       6m8.9s

2022/02/18 08:51:07 DEBUG : 20 go routines active
2022/02/18 08:51:07 Failed to sync: failed to open source object: can't read dangling shortcut

ncw · February 18, 2022, 12:17pm

There is an issue about this here drive: Skip or suppress errors for dangling shortcuts · Issue #5949 · rclone/rclone · GitHub

Can you try the beta linked on the page with the --drive-skip-dangling-shortcuts flag and see if it works for you.

pinkit · February 18, 2022, 1:33pm

Oh nice! I wish I'd thought to look there first.

I've tried out the beta and it works - other files are now copied / removed as expected. Just a couple of comments about the log message:

"NOTICE: Dangling shortcut "Events Archive" detected" - would it be possible to print the full path? That would help track them down esp. if the name is reused.
The action taken is ambiguous from the log. Would it be possible to add "- ignoring" on the end, like the existing logs for duplicate objects?

asdffdsa · February 18, 2022, 1:54pm

hi, this has been requested numerous times and the answer is always the same.
rclone never prints the full path.

could write a simple script to parse the rclone output/logfile,
printout the source root path + relative path.

pinkit · February 18, 2022, 6:07pm

I think it depends what you mean by "full path". Currently the log message just shows the file name; if there are other files with the same name then it will be harder to track down. Still possible, just harder, so it's a nice to have.

So this would be just to print the relative path, rather than "source root path + relative path", if I've understood you correctly.

asdffdsa · February 18, 2022, 6:16pm

yes, it would be nice to have.

well, that should not be an issue, as rclone includes the path of the file
INFO : keepass/database/01.kdbx: Copied (new)

so, in my case, the full path would be
b:\rclone\vss\data_files_wasabi_en07-data-crypt_20220218.102428\data\
+
keepass/database/01.kdbx

pinkit · February 18, 2022, 6:25pm

Sorry, I'm confused now. Let me try and explain again: the beta version of rclone that fixes the issue prints out a log line for the dangling shortcut. The log line only has the name of the file in it, so it's hard to know at a glance where the file is located.

Example:

2022-02-18 13:25:45 NOTICE: Dangling shortcut "Events Archive" detected

Ideally the log line would be:

2022-02-18 13:25:45 NOTICE: Dangling shortcut "Roles and teams/Events Team/Events Archive" detected

This is a new problem in the beta version because it'll no longer show the ERROR logs, which do have the relative path in them e.g.

2022/02/18 08:45:00 ERROR : Roles and teams/Events Team/Events Archive: Failed to copy: failed to open source object: can't read dangling shortcut

I suppose I could run it without the flag to see where the shortcuts are, assuming it's still possible to toggle the behaviour after the beta.

Does that make sense?

asdffdsa · February 18, 2022, 6:44pm

with that detailed example, yes, makes sense.

the new NOTICE entry should include the relative path, same at the ERROR entry did.

ncw · February 20, 2022, 8:19am

Can you make a note on the issue about that and I'll stick the full path in if possible, or at least the path to the directory.

system · March 22, 2022, 8:20am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.