Dedupe Terminating on insufficientFilePermisson error

Suspected Bug
1

What is the problem you are having with rclone?

Rclone dedupe termintates completely if it encounters an insufficientFilePermisson error,
shouldn't it just skip the files it cannot edit/delete and continue operation on the rest of the files?

What is your rclone version (output from rclone version)

rclone v1.48.0-211-g82c6c77e-beta
- os/arch: linux/amd64
- go version: go1.12.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Centos 7, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone dedupe first gdrive: -vv

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Failed to dedupe: merge duplicate dirs: MergeDirs move failed to rmdir "testdir": googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions

Thanks for the help!

2

IO Error: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions

I am getting a similar error

3

That only looks like a clip of the log, do you have the full log?

4

The rest of the log is just successful moves and deletes.....
Rclone terminates right after this error

5

Can you share the log or no?

6

Here you go,

DUMMY_URL:~/rclone$ rclone dedupe largest of3: -vv --tpslimit 10 --checkers 10 --fast-list
2019/08/21 21:59:23 DEBUG : rclone: Version "v1.48.0-211-g82c6c77e-beta" starting with parameters ["rclone" "dedupe" "largest" "of3:" "-vv" "--tpslimit" "10" "--checkers" "10" "--fast-list"]
2019/08/21 21:59:23 DEBUG : Using config file from "/home/rs/.local/bin/rclone.conf"
2019/08/21 21:59:23 INFO  : Starting HTTP transaction limiter: max 10 transactions/s with burst 1
2019/08/21 21:59:23 INFO  : Google drive root '': Looking for duplicates using largest mode.
2019/08/21 21:59:23 DEBUG : of3: Loaded invalid token from config file - ignoring
2019/08/21 21:59:28 DEBUG : of3: Saved new token in config file
2019/08/21 22:01:16 INFO  : DUMMY_FOLDER: Merging contents of duplicate directories
2019/08/21 22:01:17 INFO  : DUMMY_FOLDER: merging "DUMMY_FILE_1"
2019/08/21 22:01:18 INFO  : DUMMY_FOLDER: removing empty directory
2019/08/21 22:01:20 INFO  : DUMMY_FOLDER: Merging contents of duplicate directories
2019/08/21 22:01:20 INFO  : DUMMY_FOLDER_1: merging "DUMMY_FILE_10"
2019/08/21 22:01:21 INFO  : DUMMY_FOLDER_1: merging "DUMMY_FILE_3"
2019/08/21 22:01:23 INFO  : DUMMY_FOLDER_1: merging "DUMMY_FILE_6"
2019/08/21 22:01:27 INFO  : DUMMY_FOLDER_1: merging "DUMMY_FILE_9"
2019/08/21 22:01:28 INFO  : DUMMY_FOLDER_1: merging "DUMMY_FILE_3"
2019/08/21 22:01:29 INFO  : DUMMY_FOLDER_1: removing empty directory
2019/08/21 22:01:31 INFO  : DUMMY_FOLDER_2: merging "DUMMY_FILE_1"
2019/08/21 22:01:32 INFO  : DUMMY_FOLDER_2: merging "DUMMY_FILE_2"
2019/08/21 22:01:33 INFO  : DUMMY_FOLDER_2: merging "DUMMY_FILE_5"
2019/08/21 22:01:34 INFO  : DUMMY_FOLDER_2: merging "DUMMY_FILE_8"
2019/08/21 22:01:35 INFO  : DUMMY_FOLDER_3: merging "DUMMY_FILE_2"
2019/08/21 22:01:36 INFO  : DUMMY_FOLDER_3: removing empty directory
2019/08/21 22:01:39 INFO  : DUMMY_FOLDER_3: Merging contents of duplicate directories
2019/08/21 22:01:40 INFO  : DUMMY_FOLDER_3: removing empty directory
2019/08/21 22:01:41 Failed to dedupe: merge duplicate dirs: MergeDirs move failed to rmdir "DUMMY_FOLDER_3: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions
DUMMY_URL:~/rclone$
7

Are you able to delete the "DUMMY_FOLDER" via a normal delete command?

If no, can you run the command with --dump headers so we can see the response?

8 
DUMMY_URL:~/rclone$ rclone rmdir "of2:DUMMY_FOLDER" -vv --dump headers --retries 1
2019/08/21 22:20:31 DEBUG : rclone: Version "v1.48.0-211-g82c6c77e-beta" starting with parameters ["rclone" "rmdir" "of2:DUMMY_FOLDER" "-vv" "--dump" "headers" "--retries" "1"]
2019/08/21 22:20:31 DEBUG : Using config file from "/home/rs/.local/bin/rclone.conf"
2019/08/21 22:20:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:31 DEBUG : HTTP REQUEST (req 0xc000239a00)
2019/08/21 22:20:31 DEBUG : GET /drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271B_YXJW9YgRYeMx0qGm277bognhwcMLqc%27+in+parents%29+and+%28name%3D%27A%27%29+and+mimeType%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:31 DEBUG : HTTP RESPONSE (req 0xc000239a00)
2019/08/21 22:20:31 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:02 GMT
Expires: Wed, 21 Aug 2019 16:46:02 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:31 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:31 DEBUG : HTTP REQUEST (req 0xc0001a7600)
2019/08/21 22:20:31 DEBUG : GET /drive/v3/about?alt=json&fields=exportFormats%2CimportFormats&prettyPrint=false HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:31 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:32 DEBUG : HTTP RESPONSE (req 0xc0001a7600)
2019/08/21 22:20:32 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:02 GMT
Expires: Wed, 21 Aug 2019 16:46:02 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:32 DEBUG : HTTP REQUEST (req 0xc000239b00)
2019/08/21 22:20:32 DEBUG : GET /drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271qP_F_IVWOerY-Bea6cQzdB29eLFIOv6-%27+in+parents%29+and+%28name%3D%27Hi10%27%29+and+mimeType%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:32 DEBUG : HTTP RESPONSE (req 0xc000239b00)
2019/08/21 22:20:32 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:02 GMT
Expires: Wed, 21 Aug 2019 16:46:02 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:32 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:32 DEBUG : HTTP REQUEST (req 0xc0001a7f00)
2019/08/21 22:20:32 DEBUG : GET /drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271TCa8_2rM2-illSTZwnAFEPY09fu0DJ4g%27+in+parents%29+and+%28name%3D%27Shows%27%29+and+mimeType%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:32 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:33 DEBUG : HTTP RESPONSE (req 0xc0001a7f00)
2019/08/21 22:20:33 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:03 GMT
Expires: Wed, 21 Aug 2019 16:46:03 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:33 DEBUG : HTTP REQUEST (req 0xc000656200)
2019/08/21 22:20:33 DEBUG : GET /drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271GA6gYDxuAJjJwzevCD913LtXFslmcial%27+in+parents%29+and+%28name%3D%27Senyuu.+-+Collection+%5BBD+1080p%5D%27%29+and+mimeType%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:33 DEBUG : HTTP RESPONSE (req 0xc000656200)
2019/08/21 22:20:33 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:03 GMT
Expires: Wed, 21 Aug 2019 16:46:03 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:33 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:33 DEBUG : HTTP REQUEST (req 0xc000133500)
2019/08/21 22:20:33 DEBUG : GET /drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=trashed%3Dfalse+and+%28%271hVOyw7mjwmV6MuceCgvLb_DMmPaafkPz%27+in+parents%29+and+%28name%3D%27Senyuu.+2+%5BBD+1080p%5D%27%29+and+mimeType%3D%27application%2Fvnd.google-apps.folder%27&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:35 DEBUG : HTTP RESPONSE (req 0xc000133500)
2019/08/21 22:20:35 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:05 GMT
Expires: Wed, 21 Aug 2019 16:46:05 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:35 DEBUG : Google drive root 'DUMMY_FOLDER': Removing directory
2019/08/21 22:20:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:35 DEBUG : HTTP REQUEST (req 0xc000656500)
2019/08/21 22:20:35 DEBUG : GET /drive/v3/files?alt=json&fields=files%28id%2Cname%2Csize%2Cmd5Checksum%2Ctrashed%2CmodifiedTime%2CcreatedTime%2CmimeType%2Cparents%2CwebViewLink%29%2CnextPageToken&includeItemsFromAllDrives=true&pageSize=1000&prettyPrint=false&q=%28%271e9A36YECkj5wOSNxBSh5rf428sv8oAiW%27+in+parents%29&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Authorization: XXXX
Accept-Encoding: gzip

2019/08/21 22:20:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:35 DEBUG : HTTP RESPONSE (req 0xc000656500)
2019/08/21 22:20:35 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:05 GMT
Expires: Wed, 21 Aug 2019 16:46:05 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:35 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:35 DEBUG : HTTP REQUEST (req 0xc0002a4400)
2019/08/21 22:20:35 DEBUG : PATCH /drive/v3/files/1e9A36YECkj5wOSNxBSh5rf428sv8oAiW?alt=json&fields=&prettyPrint=false&supportsAllDrives=true HTTP/1.1
Host: www.googleapis.com
User-Agent: rclone/v1.48.0-211-g82c6c77e-beta
Content-Length: 17
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

2019/08/21 22:20:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2019/08/21 22:20:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:36 DEBUG : HTTP RESPONSE (req 0xc0002a4400)
2019/08/21 22:20:36 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Alt-Svc: quic=":443"; ma=2592000; v="46,43,39"
Cache-Control: private, max-age=0
Content-Type: application/json; charset=UTF-8
Date: Wed, 21 Aug 2019 16:46:06 GMT
Expires: Wed, 21 Aug 2019 16:46:06 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2019/08/21 22:20:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2019/08/21 22:20:36 ERROR : Attempt 1/1 failed with 2 errors and: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions
2019/08/21 22:20:36 Failed to rmdir with 2 errors: last error was: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions
DUMMY_URL:~/rclone$
9

rclone dedupe will terminate if any of the commands it does fail... This isn't ideal!

It could be made to continue some of the errors quite easily I think.

Can you please make a new issue on github about this?

10
11

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.