Rclone sync and copy failed error 403

lebossejames · May 30, 2020, 9:10am

Hello,

What is the problem you are having with rclone?

I cannot copy or sync files. I can list files on the remote only. the account service used as service_account_file works fine.

What is your rclone version (output from `rclone version`)

v1.51_x64_linux

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 19.04

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

`` `
Tools/rclone_v1.51.0_linux_x64/rclone sync Logs/Networks/wifi_informations_DEV.txt remoteAndroid_Drive:/DEV

The rclone config contents with secrets removed.

    [remoteAndroid_Drive]
     type = drive 
    service_account_credentials = {... contents of the service account file ...}
    scope = drive
    root_folder_id = ID

#### A log from the command with the `-vv` flag  

2020/05/30 11:04:23 DEBUG : rclone: Version "v1.51.0" starting with parameters ["Tools/rclone_v1.51.0_linux_x64/rclone" "-vv" "copy" "Logs/Networks/wifi_informations_DEV.txt" "remoteAndroid_Drive:/DEV/wifi.txt"]
2020/05/30 11:04:23 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/30 11:04:24 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/30 11:04:25 ERROR : wifi_informations_DEV.txt: Failed to copy: failed to make directory: googleapi: Error 403: Forbidden, forbidden
2020/05/30 11:04:25 ERROR : Attempt 1/3 failed with 1 errors and: failed to make directory: googleapi: Error 403: Forbidden, forbidden
2020/05/30 11:04:25 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/30 11:04:26 ERROR : wifi_informations_DEV.txt: Failed to copy: failed to make directory: googleapi: Error 403: Forbidden, forbidden
2020/05/30 11:04:26 ERROR : Attempt 2/3 failed with 1 errors and: failed to make directory: googleapi: Error 403: Forbidden, forbidden
2020/05/30 11:04:26 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/30 11:04:27 ERROR : wifi_informations_DEV.txt: Failed to copy: failed to make directory: googleapi: Error 403: Forbidden, forbidden
2020/05/30 11:04:27 ERROR : Attempt 3/3 failed with 1 errors and: failed to make directory: googleapi: Error 403: Forbidden, forbidden
2020/05/30 11:04:27 Failed to copy: failed to make directory: googleapi: Error 403: Forbidden, forbidden

Animosity022 · May 30, 2020, 11:02am

Can you update to 1.52 and retest? Does seem like something permission related with the service account.

lebossejames · May 30, 2020, 2:46pm

Same problem with v1.52:

2020/05/30 16:43:17 DEBUG : rclone: Version "v1.52.0" starting with parameters ["Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "sync" "Logs/Networks/wifi_informations_DEV.txt" "remoteAndroid_Drive:/DEV"]
2020/05/30 16:43:17 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/30 16:43:17 DEBUG : fs cache: renaming cache item "Logs/Networks/wifi_informations_DEV.txt" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks"
2020/05/30 16:43:19 DEBUG : fs cache: renaming cache item "remoteAndroid_Drive:/DEV" to be canonical "remoteAndroid_Drive:DEV"
2020/05/30 16:43:19 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/30 16:43:20 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/30 16:43:20 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/30 16:43:21 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/30 16:43:22 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/30 16:43:22 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/30 16:43:22 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/30 16:43:23 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/30 16:43:23 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/30 16:43:23 INFO :
Transferred: 768 / 768 Bytes, 100%, 247 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 3.0s
2020/05/30 16:43:23 DEBUG : 6 go routines active
2020/05/30 16:43:23 Failed to sync: googleapi: Error 403: Forbidden, forbidden

Animosity022 · May 30, 2020, 3:00pm

The 404s are permission errors. It doesn't seem your service account has permissions to write files.

lebossejames · May 30, 2020, 3:07pm

403 you want mean? Where can i see the permission of service account because as i say before, if i use service_account_file, i can write.

ncw · May 30, 2020, 5:29pm

lebossejames:

2020/05/30 11:04:23 DEBUG : rclone: Version "v1.51.0" starting with parameters ["Tools/rclone_v1.51.0_linux_x64/rclone" "-vv" "copy" "Logs/Networks/wifi_informations_DEV.txt" "remoteAndroid_Drive:/DEV/wifi.txt"]

Note that if you are copying a file to a file you want copyto instead of copy otherwise rclone will create a directory called wifi.txt with your file in it!

I think that is probably the problem - rclone is failing to create the directory by the look of the log.

lebossejames · May 31, 2020, 7:53am

I try also:

Tools/rclone_v1.52.0_linux_x64/rclone sync Logs/Networks/wifi_informations_DEV.txt remoteAndroid_Drive:/DEV/

Same problem.

ncw · May 31, 2020, 10:16am

Try

Tools/rclone_v1.51.0_linux_x64/rclone -vv copyto Logs/Networks/wifi_informations_DEV.txt remoteAndroid_Drive:/DEV/wifi.txt

lebossejames · May 31, 2020, 2:02pm

I get this:

2020/05/31 16:00:50 DEBUG : rclone: Version "v1.52.0" starting with parameters ["Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "copyto" "Logs/Networks/wifi_informations_DEV.txt" "remoteAndroid_Drive:/DEV/wifi.txt"]
2020/05/31 16:00:50 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/31 16:00:50 DEBUG : fs cache: renaming cache item "Logs/Networks/wifi_informations_DEV.txt" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks"
2020/05/31 16:00:51 DEBUG : fs cache: renaming cache item "remoteAndroid_Drive:/DEV/" to be canonical "remoteAndroid_Drive:DEV"
2020/05/31 16:00:51 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/31 16:00:52 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/31 16:00:52 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/31 16:00:52 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/31 16:00:53 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/31 16:00:53 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/31 16:00:53 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/31 16:00:54 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/31 16:00:54 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/31 16:00:54 INFO :
Transferred: 768 / 768 Bytes, 100%, 324 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 2.3s
2020/05/31 16:00:54 DEBUG : 6 go routines active
2020/05/31 16:00:54 Failed to copyto: googleapi: Error 403: Forbidden, forbidden

Animosity022 · May 31, 2020, 2:14pm

What does:

rclone ls Logs/Networks/wifi_informations_DEV.txt

show?

lebossejames · May 31, 2020, 2:47pm

Tools/rclone_v1.52.0_linux_x64/rclone ls Logs/Networks/wifi_informations_DEV.txt
256 wifi_informations_DEV.txt

Animosity022 · May 31, 2020, 2:57pm

Can you go into that directory and just test a copy to the root and see?

So like:

rclone copy wifi_informations_DEV.txt remoteAndroid_Drive:

and if that does work, try a test with copyto

felix@gemini:~$ rclone copy hosts GD: -vv
2020/05/31 10:56:40 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "copy" "hosts" "GD:" "-vv"]
2020/05/31 10:56:40 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/05/31 10:56:40 DEBUG : fs cache: renaming cache item "hosts" to be canonical "/home/felix"
2020/05/31 10:56:40 DEBUG : hosts: Need to transfer - File not found at Destination
2020/05/31 10:56:41 DEBUG : hosts: MD5 = 157c632175c4251012e651322736f0ae OK
2020/05/31 10:56:41 INFO  : hosts: Copied (new)
2020/05/31 10:56:41 INFO  :
Transferred:   	       413 / 413 Bytes, 100%, 447 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.9s

2020/05/31 10:56:41 DEBUG : 5 go routines active

and

felix@gemini:~$ rclone copyto hosts GD:testing
felix@gemini:~$ rclone lsf GD:
crypt/
hosts
jellyfish-400-mbps-4k-uhd-hevc-10bit.mkv
testing

lebossejames · May 31, 2020, 5:47pm

2020/05/31 19:41:02 DEBUG : rclone: Version "v1.52.0" starting with parameters ["../../Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "copy" "wifi_informations_DEV.txt" "remoteAndroid_Drive:"]
2020/05/31 19:41:02 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/31 19:41:02 DEBUG : fs cache: renaming cache item "wifi_informations_DEV.txt" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks"
2020/05/31 19:41:02 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/31 19:41:03 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/31 19:41:03 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/31 19:41:03 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/31 19:41:04 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/31 19:41:04 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/31 19:41:04 DEBUG : wifi_informations_DEV.txt: Need to transfer - File not found at Destination
2020/05/31 19:41:05 ERROR : wifi_informations_DEV.txt: Failed to copy: googleapi: Error 403: Forbidden, forbidden
2020/05/31 19:41:05 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: Forbidden, forbidden
2020/05/31 19:41:05 INFO :
Transferred: 768 / 768 Bytes, 100%, 311 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 2.4s
2020/05/31 19:41:05 DEBUG : 6 go routines active
2020/05/31 19:41:05 Failed to copy: googleapi: Error 403: Forbidden, forbidden

second command:

2020/05/31 19:41:10 DEBUG : rclone: Version "v1.52.0" starting with parameters ["../../Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "copy" "hosts" "remoteAndroid_Drive:"]
2020/05/31 19:41:10 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/31 19:41:10 DEBUG : fs cache: renaming cache item "hosts" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks/hosts"
2020/05/31 19:41:11 ERROR : : error reading source directory: directory not found
2020/05/31 19:41:11 DEBUG : Google drive root '': Waiting for checks to finish
2020/05/31 19:41:11 DEBUG : Google drive root '': Waiting for transfers to finish
2020/05/31 19:41:11 INFO : There was nothing to transfer
2020/05/31 19:41:11 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2020/05/31 19:41:11 ERROR : : error reading source directory: directory not found
2020/05/31 19:41:11 DEBUG : Google drive root '': Waiting for checks to finish
2020/05/31 19:41:11 DEBUG : Google drive root '': Waiting for transfers to finish
2020/05/31 19:41:11 INFO : There was nothing to transfer
2020/05/31 19:41:11 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2020/05/31 19:41:11 ERROR : : error reading source directory: directory not found
2020/05/31 19:41:11 DEBUG : Google drive root '': Waiting for checks to finish
2020/05/31 19:41:11 DEBUG : Google drive root '': Waiting for transfers to finish
2020/05/31 19:41:11 INFO : There was nothing to transfer
2020/05/31 19:41:11 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2020/05/31 19:41:11 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.0s
2020/05/31 19:41:11 DEBUG : 6 go routines active
2020/05/31 19:41:11 Failed to copy: directory not found

2020/05/31 19:43:26 DEBUG : rclone: Version "v1.52.0" starting with parameters ["../../Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "copyto" "wifi_informations_DEV.txt" "remoteAndroid_Drive:"]
2020/05/31 19:43:26 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/31 19:43:26 DEBUG : fs cache: renaming cache item "wifi_informations_DEV.txt" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks"
2020/05/31 19:43:26 "remoteAndroid_Drive:" is a directory

2020/05/31 19:45:30 DEBUG : rclone: Version "v1.52.0" starting with parameters ["../../Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "copyto" "hosts" "remoteAndroid_Drive:"]
2020/05/31 19:45:30 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/31 19:45:30 DEBUG : fs cache: renaming cache item "hosts" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks/hosts"
2020/05/31 19:45:31 ERROR : : error reading source directory: directory not found
2020/05/31 19:45:31 DEBUG : Google drive root '': Waiting for checks to finish
2020/05/31 19:45:31 DEBUG : Google drive root '': Waiting for transfers to finish
2020/05/31 19:45:31 INFO : There was nothing to transfer
2020/05/31 19:45:31 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2020/05/31 19:45:31 ERROR : : error reading source directory: directory not found
2020/05/31 19:45:31 DEBUG : Google drive root '': Waiting for checks to finish
2020/05/31 19:45:31 DEBUG : Google drive root '': Waiting for transfers to finish
2020/05/31 19:45:31 INFO : There was nothing to transfer
2020/05/31 19:45:31 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2020/05/31 19:45:31 ERROR : : error reading source directory: directory not found
2020/05/31 19:45:31 DEBUG : Google drive root '': Waiting for checks to finish
2020/05/31 19:45:31 DEBUG : Google drive root '': Waiting for transfers to finish
2020/05/31 19:45:31 INFO : There was nothing to transfer
2020/05/31 19:45:31 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2020/05/31 19:45:31 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.0s
2020/05/31 19:45:31 DEBUG : 6 go routines active
2020/05/31 19:45:31 Failed to copyto: directory not found

2020/05/31 19:46:20 DEBUG : rclone: Version "v1.52.0" starting with parameters ["../../Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "copyto" "hosts" "remoteAndroid_Drive:testing"]
2020/05/31 19:46:20 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
2020/05/31 19:46:20 DEBUG : fs cache: renaming cache item "hosts" to be canonical "/home/james/Desktop/LookJames/DEV/v1.00/Logs/Networks/hosts"
2020/05/31 19:46:21 ERROR : : error reading source directory: directory not found
2020/05/31 19:46:21 DEBUG : Google drive root 'testing': Waiting for checks to finish
2020/05/31 19:46:21 DEBUG : Google drive root 'testing': Waiting for transfers to finish
2020/05/31 19:46:21 INFO : There was nothing to transfer
2020/05/31 19:46:21 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2020/05/31 19:46:21 ERROR : : error reading source directory: directory not found
2020/05/31 19:46:21 DEBUG : Google drive root 'testing': Waiting for checks to finish
2020/05/31 19:46:21 DEBUG : Google drive root 'testing': Waiting for transfers to finish
2020/05/31 19:46:21 INFO : There was nothing to transfer
2020/05/31 19:46:21 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2020/05/31 19:46:21 ERROR : : error reading source directory: directory not found
2020/05/31 19:46:21 DEBUG : Google drive root 'testing': Waiting for checks to finish
2020/05/31 19:46:21 DEBUG : Google drive root 'testing': Waiting for transfers to finish
2020/05/31 19:46:21 INFO : There was nothing to transfer
2020/05/31 19:46:21 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2020/05/31 19:46:21 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.0s
2020/05/31 19:46:21 DEBUG : 6 go routines active
2020/05/31 19:46:21 Failed to copyto: directory not found

2020/05/31 19:47:26 DEBUG : rclone: Version "v1.52.0" starting with parameters ["../../Tools/rclone_v1.52.0_linux_x64/rclone" "-vv" "lsf" "remoteAndroid_Drive:"]
2020/05/31 19:47:26 DEBUG : Using config file from "/home/james/.config/rclone/rclone.conf"
DEV/
PROD/
remote_modele.json
2020/05/31 19:47:27 DEBUG : 6 go routines active

Animosity022 · May 31, 2020, 6:16pm

And you in the GSuite admin, you can validate the account is setup and permissioned?

You see something like:

That 403 is really about permissions.

lebossejames · June 1, 2020, 8:30am

I don't have GSuite account.

Animosity022 · June 1, 2020, 10:21am

I'm not sure how you permission a regular user with a service account.

Can you just use your own user? What's the reason you are using a service account?

lebossejames · June 1, 2020, 5:03pm

Service account because i don't want change my credentials every time.

Animosity022 · June 1, 2020, 5:04pm

What do you mean by change your credentials? I use a regular account and never touch it since I've set it up.

lebossejames · June 1, 2020, 7:03pm

So your regular account with Email and password readable?

Animosity022 · June 1, 2020, 8:15pm

No, in your config file is an access token that's specific after you login.

You are prompted to enter your user name and password and you get the normal oauth application page that would be the same when you connect any app.

If you follow the prompts in the rclone config process, it would walk you through it.

I'm still not sure how you authenticate a service account to a local google drive as that's a GSuite thing. I'm not super well versed in that.