Copy with --drive-server-side-across-configs problem

kdantas · May 9, 2022, 4:41am

What is the problem you are having with rclone?

I have 02 shared drives on the same gmail account and I want to copy the content (14TB) from SD1 to SD2 using --drive-server-side-across-configs flag. If I do --> rclone copy SD2: SD1: --drive-server-side-across-configs, I can copy without any problems. But if I do rclone copy SD1: SD2: --drive-server-side-across-configs gives the following error --> Failed to copy: googleapi: Error 404: File not found: 1LDP1ZwKqfwcVJt4An1gKMy2ztZQdUCrj., notFound for all files in SD1. I've already shared the two SDs, from one to the other. I also created the Client ID and Secret Key for each SD. I really don't know where the error is. In resume: rclone copy SD2: SD1: --drive-server-side-across-configs (ok), but rclone copy SD1: SD2: --drive-server-side-across-configs (error)

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

os/version: ubuntu 16.04
os/kernel: 4.9.160-armada375 (armv7l)
os/type: linux
os/arch: arm
go/version: go1.17.9
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Shared Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy brhd:Kleber enterprise:backup --fast-list --progress --max-backlog=999999 --drive-acknowledge-abuse --drive-server-side-across-configs -vv --log-file=/home/kdantas/rcloncopy.log

The rclone config contents with secrets removed.

[enterprise]
type = drive
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"ya29.A0ARrdaM_xxxxxxxxxxxxxxxmhYHcOAkErihdAHieAia-FnGt94ra16gaspAmirM0_VtHMxEWR5_LKoRLMJ7wbh_G647tcKohPEsPWzDCXTzJaGfm8cG9rdxxxxxxxxxxxxxxxTxr$
team_drive = 0AEmxc_xxxxxxxxxxxxxxx
root_folder_id =

[brhd]
type = drive
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
scope = drive
token = {"access_token":"ya29.A0ARrdaM_xxxxxxxxxxxxxxx_1YLZLAon0k_txAtnrZ1_b9N8CEmbPufxBId2qFWupm0-8bEla6kNvoSHgqEFHzGnCwpqBzKbpb7gkKqifGtsTKsZoNsj97h3zzaWT4dxxxxxxxxxxxxxxx
team_drive = 0AAsDD_xxxxxxxxxxxxxxx
root_folder_id =

A log from the command with the `-vv` flag

2022/05/09 06:28:56 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "copy" "brhd:Kleber" "enterprise:backup" "--fast-list" "--progress" "--max-backlog=999999" "--drive-acknowledge-abuse" "--drive-server-side-across-configs" "-vv" "--log-file=/home/kdantas/rcloncopy.log"]
2022/05/09 06:28:56 DEBUG : Creating backend with remote "brhd:Kleber"
2022/05/09 06:28:56 DEBUG : Using config file from "/home/kdantas/.config/rclone/rclone.conf"
2022/05/09 06:28:56 DEBUG : brhd: detected overridden config - adding "{xx9kk}" suffix to name
2022/05/09 06:28:56 DEBUG : fs cache: renaming cache item "brhd:Kleber" to be canonical "brhd{xx9kk}:Kleber"
2022/05/09 06:28:56 DEBUG : Creating backend with remote "enterprise:backup"
2022/05/09 06:28:56 DEBUG : enterprise: detected overridden config - adding "{xx9kk}" suffix to name
2022/05/09 06:28:57 DEBUG : fs cache: renaming cache item "enterprise:backup" to be canonical "enterprise{xx9kk}:backup"
2022/05/09 06:28:57 DEBUG : Google drive root 'backup': Disabling ListR to work around bug in drive as multi listing (4) returned no entries
2022/05/09 06:28:57 DEBUG : Google drive root 'backup': Recycled 4 entries
2022/05/09 06:28:58 DEBUG : Google drive root 'backup': Re-enabling ListR as previous detection was in error
2022/05/09 06:29:35 DEBUG : (1995) - O Combate_Lágrimas do Guerreiro.iso: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/05/09 06:29:35 DEBUG : (1995) - O Combate_Lágrimas do Guerreiro.iso: Unchanged skipping
2022/05/09 06:29:35 DEBUG : -C.A.A.DVD5.2009.INABALAVEL.ISO: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/05/09 06:29:35 DEBUG : -C.A.A.DVD5.2009.INABALAVEL.ISO: Unchanged skipping
2022/05/09 06:29:35 DEBUG : -C.A.A.DVD5.2009.THE_SOLOIST.ISO: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/05/09 06:29:35 DEBUG : -C.A.A.DVD5.2009.THE_SOLOIST.ISO: Unchanged skipping
2022/05/09 06:29:35 DEBUG : 11.9.ISO: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/05/09 06:29:35 DEBUG : 11.9.ISO: Unchanged skipping
2022/05/09 06:29:35 DEBUG : 13 - O jogador.iso: Size and modification time the same (differ by 0s, within tolerance 1ms)
2022/05/09 06:29:35 DEBUG : 13 - O jogador.iso: Unchanged skipping
.
.
.
2022/05/09 06:29:36 ERROR : Antes So do que Mal Casado.ISO: Failed to copy: googleapi: Error 404: File not found: 1FfAgFn2h88I_W1TdI48F8Hml835XP44h., notFound
2022/05/09 06:29:36 ERROR : Andrea Bocelli - Vivere - Live in Tuscany.iso: Failed to copy: googleapi: Error 404: File not found: 18IllizTnarYYbCz8W2OC0-2tV1aYTYQ6., notFound
2022/05/09 06:29:36 ERROR : Annabelle.3.2019.DVD-R.THYRSO10.iso: Failed to copy: googleapi: Error 404: File not found: 1qcrGbBOB17snIvecXPKL5VuGyXmsMIFv., notFound
2022/05/09 06:29:36 ERROR : Armadilha.1999.DVD-R_MiNeIrO.iso: Failed to copy: googleapi: Error 404: File not found: 1cc_qIyFB4nRkTJ5sq3-VvkEx6zPFua8b., notFound
2022/05/09 06:29:36 DEBUG : Google drive root 'backup': Waiting for checks to finish
2022/05/09 06:29:36 DEBUG : Google drive root 'backup': Waiting for transfers to finish
2022/05/09 06:29:36 ERROR : Arquivo.X.DVD-R.MiNeIrO.iso: Failed to copy: googleapi: Error 404: File not found: 18ZarKyZqTVDyfVReIFl0Tthe7aBO80pd., notFound
2022/05/09 06:29:36 ERROR : As 24 Horas de Le Mans 1971 DVD-R.iso: Failed to copy: googleapi: Error 404: File not found: 1WqpYfKHMMZdf8HmvQszMcOLksvbz7stV., notFound
2022/05/09 06:29:36 ERROR : As.Donas.da.Noite.DVD-R.MiNeIrO.iso: Failed to copy: googleapi: Error 404: File not found: 1jSH-cZAwzUsgOuD4gOo7cAtj2RFicQgs., notFound
2022/05/09 06:29:37 ERROR : As.Ruinas.2008.DVD-R_MiNeIrO.iso: Failed to copy: googleapi: Error 404: File not found: 1WSCNUJaP_9DjggOJCm60pXJRnLlCL5u4., notFound
2022/05/09 06:29:37 ERROR : Ashley.Garcia.Genius.in.Love.2020.The.True.Meaning.of.Christmas.Togetherness.Funtime.1080p.NF.WEB-DL.DDP5.1.H.264-NTb.mkv: Failed to copy: googleapi: Error 404: File not found: 1KO-UdAWnMkySAAR53zlU63wRNP4BQcjf., notFound
2022/05/09 06:29:37 ERROR : Atlantis.O.Reino.Perdido.2001.DVDR-Professor_X.iso: Failed to copy: googleapi: Error 404: File not found: 1HmZznfXdAcHBJAorHReoFb3Uq_HVk04Q., notFound
2022/05/09 06:29:37 ERROR : AustinPowersOAgenteBondCama.ISO: Failed to copy: googleapi: Error 404: File not found: 1x5Ss3U5py4NquKg8bRQaBJkfk5Zb7KZl., notFound
2022/05/09 06:29:37 ERROR : Aves.de.Rapina.Arlequina.e.sua.Emancipacao.Fantabulosa.2020.DVD-R.THYRSO10.iso: Failed to copy: googleapi: Error 404: File not found: 1MmAc2rdQFhf3eJ9N-uKotnVnb7gaJVdM., notFound
2022/05/09 06:29:37 ERROR : BILL_TED.iso: Failed to copy: googleapi: Error 404: File not found: 1sLEfSBrnd19LDE0RGJ-T_Cj4MoS5Ujdj., notFound
2022/05/09 06:29:38 ERROR : BRUBAKER_1980.iso: Failed to copy: googleapi: Error 404: File not found: 1Ibu8Yp6Pw2TG6ogPotRfyJl2geugbEAf., notFound
2022/05/09 06:29:38 ERROR : BRAVE.rar: Failed to copy: googleapi: Error 404: File not found: 1-mDed0HfvyTl2c6D7-8NRvOcMhj52dL0., notFound
2022/05/09 06:29:38 ERROR : Bad Boys 2 DVD-R TOMATE.ISO: Failed to copy: googleapi: Error 404: File not found: 1VNEB3_xTkJuXUkmG2olZW2-WGL4H-VY4., notFound
2022/05/09 06:29:38 ERROR : Aynen Aynen S04E01 1080p WEBRip AAC H.264-BdC.mkv: Failed to copy: googleapi: Error 404: File not found: 1PbfqQL22LYGYJ0yEECpmAu7LA9pW5qP5., notFound
2022/05/09 06:29:38 ERROR : Bad.Boys.Para.Sempre.2020.DVD-R.THYRSO10.iso: Failed to copy: googleapi: Error 404: File not found: 1soK1vqLk7Lw-MJNkGwvDLwotXMiMJy7l., notFound
2022/05/09 06:29:38 ERROR : Balle.Perdue.2020.1080p.NF.WEB-DL.DDP5.1.HEVC.DUAL-PD.mkv: Failed to copy: googleapi: Error 404: File not found: 1covAJMn4xYS-ZNSt5NFdyrFhh4c3Kqe3., notFound
2022/05/09 06:29:38 ERROR : Banana.Joe.DVD-R_MiNeIrO.iso: Failed to copy: googleapi: Error 404: File not found: 1KJBYhd0uHbJ0erbmxcqKIpDPM_kcAj9I., notFound
2022/05/09 06:29:38 ERROR : Barbie.Super.Princesa.2015.DVD-R.Thyrso10.iso: Failed to copy: googleapi: Error 404: File not found: 13nazShC_JShn2J15xAP19ZfgSvz6DL1p., notFound
2022/05/09 06:29:39 ERROR : Barbie.e.os.Golfinhos.Mágicos.2018.DVD-R.THYRSO10.iso: Failed to copy: googleapi: Error 404: File not found: 1DzucH-n3byD23Zx51CwlzsuT0JuHj9nQ., notFound
.
.
.

Animosity022 · May 9, 2022, 11:20am

You need permissions to do that.

You can either fix permissions or other folks have added a shortcut as well.

Google drive (rclone.org)

kdantas · May 9, 2022, 4:35pm

Hi Animosity022, first thanks for the reply. But I confess that now I'm a little confused. What kind of permission are you referring to? Has there been any kind of change in the use of the --drive-server-side-across-configs flag? And why if I copy from SD2 to SD1 this error doesn't happen? I remember that it was enough to do the sharing between the drives for the flag to work. And is there a way to use Shortcuts to solve this error? If so, how should I proceed?

Animosity022 · May 9, 2022, 4:41pm

The destination I believe needs content owner or write on the source to make the copy (I get the directions mixed sometimes, plenty of posts on the forum on it if I have it backwards).

None.

One probably has permissions but the other way does not.

Yes, that's one way around it as I shared the shortcut link above.

Up to you as Shortcuts or permissions are fine. You just have to pick.

asdffdsa · May 9, 2022, 4:55pm

https://forum.rclone.org/t/unable-to-sync-across-drive-server-side-configs/26836/2

kdantas · May 9, 2022, 7:40pm

Hi @Animosity022, I am the owner of both Shared Drivers. And as I said, the Shared Drivers are in the same email account. So there is no way to have a permission problem. So it's very strange to be showing this error.

Failed to copy: googleapi: Error 404: File not found: 1LDP1ZwKqfwcVJt4An1gKMy2ztZQdUCrj., notFound

And speaking of shortcuts, I'm still confused on how to use it. So could you help me by editing my copy command to make it work with shortcuts?

rclone copy brhd:Kleber enterprise:backup --fast-list --progress --max-backlog=999999 --drive-acknowledge-abuse --drive-server-side-across-configs -vv

My best regards, kdantas.

Animosity022 · May 9, 2022, 7:45pm

The 404 is a permissions error as it can't do something which generates the error. If you pick one file and don't use server side copy, that would validate it.

Use

--disable move,copy

On one file. If it copies, there would be a permissions error. If not, we can dig deeper.

In the shared with me area, I believe you do:

I'm sure there is a CLI way but I have no access for Google Drive these days to really play around.

kdantas · May 9, 2022, 8:23pm

Using this flag, copying works at normal throughput as if not using the --drive-server-side-across-configs flag. So it makes no sense. That is, without the --drive-server-side-across-configs flag copying occurs normally.

Are shortcuts created on the same command line used with rclone copy SD1:SD2? Or are they separate commands from each other? As you can see, this issue of shortcuts is a bit confused in my head.

kdantas · May 9, 2022, 9:53pm

Shortcut 1
Shortcut 2

I have created the shortcuts of the folders that I want to copy directly from the sharing drives. But when running the copy command with the --drive-server-side-across-configs flag, the same error persists. Or should I create the shortcuts using the command: rclone backend?

Animosity022 · May 10, 2022, 12:45am

Sorry as I don't have Google Drive myself and can't test any of that stuff unfortunately.

If you give some trial and error and see what works.

Both shared accounts have content owner or something along those lines? I don't recall what the permission is. You can test with one file and I think @ncw can add some detail to see what is going on perhaps.

ncw · May 10, 2022, 6:32pm

This normally means that the SD2: user does not have read permissions on SD1:. When I say user, it is the user you used to do the initial oauth to set the remote up.

kdantas · May 12, 2022, 1:54am

Hi Nick, thanks for the reply. How can I check/set this permission in the Google Cloud Console? Or would it be somewhere else? In my mind, if I shared the folder(s) from SD1 to SD2 and from SD2 to SD1, using the email address, I wouldn't have problems like this.

ncw · May 12, 2022, 8:26am

I think that will work - sharing the folder with the email address.

I'm sorry I don't know anything about the permissions in the google cloud console though.

kdantas · May 15, 2022, 4:38pm

Thanks for you reply. But I am still getting the same error. I think google must have changed some permission. I will keep doing some tests. Best regards, kdantas.

system · May 18, 2022, 4:39pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.