Rclone dedupe permission error

Suspected Bug
1

What is the problem you are having with rclone?

rclone is not able to solve duplicates in google drive due to permission error.

Run the command 'rclone version' and share the full output of the command.

rclone version

rclone v1.58.0

  • os/version: ubuntu 21.10 (64 bit)
  • os/kernel: 5.13.0-40-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone dedupe --dedupe-mode largest gdrive-media: \
  --config /root/.config/rclone/rclone9.conf \
  -vv \
  --fast-list \
  --progress \
  --human-readable

The rclone config contents with secrets removed.

# cat .config/rclone/rclone9.conf
[gdrive-media]
type = drive
client_id = XXX.apps.googleusercontent.com
client_secret = GOCSXXX
scope = drive
root_folder_id = XXXXXXXX
service_account_file = /root/rclone_service_account #has domain wide delegation
impersonate = [email address of account which has editor rights on the folder. account is part of same domain as service account]
chunk_size = 256M


[gdrive-media-crypt]
type = crypt
remote = gdrive-media:/media
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

A log from the command with the -vv flag

2022-04-28 16:46:19 INFO  : media/ef0jejslv4t93krcb6nlf4q4l0/90321aifb8nhm7g75agc6halt4/jdn7a8sr2uht5qgn6e6hq6qf98: Merging contents of duplicate directories
2022-04-28 16:46:19 INFO  : media/ef0jejslv4t93krcb6nlf4q4l0/90321aifb8nhm7g75agc6halt4/jdn7a8sr2uht5qgn6e6hq6qf98: removing empty directory
2022-04-28 16:46:20 ERROR : merge duplicate dirs: MergeDirs move failed to rmdir "media/ef0jejslv4t93krcb6nlf4q4l0/90321aifb8nhm7g75agc6halt4/jdn7a8sr2uht5qgn6e6hq6qf98": googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions

rclone move is working with same config.

2

There's not much rclone can do to solve a permission error.

You'd have to identify what permissions you are missing on the service account you have setup.

Sounds like you need owner and not editor I'd imagine.

3

But a service acccount cannot be the owner of a google drive folder. I think rclone is not using imperonate when trying to delete the folder. How can we debug this in detail if rcclone is using ipersonate when trying to delete the folder?

4

I don't use/have access to the GSuite items anymore.

The word might be contributor as I'm not sure what it is as I can't see it.

You need a higher permission or don't use impersonate to dedupe.

5

What is the problem with this dubicate files? Are they just wasting space or can they make any problems?

I will try again with token auth instead of service account.

6

It depends.

If you have a mount, most operating systems don't support duplicate files.

Most cloud providers don't either. Google happens to along with a few others.

7

i tried again without service account / impersonate and instead with token auth. Dedupe is working now. How can i help to investigate the bug regarding to impersonate dedupe?

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.