Errors when syncing Google Drive -> Onedrive with --backup-dir

Help and Support
#1

What is the problem you are having with rclone?

I'm trying to sync a google drive backend to a folder on onedrive with --backup-dir option. New files are copied ("synced") without problems, but when Rclone tries to move the files to be deleted to the folder indicated by the --backup-dir option, it gets errors, and the files remain on the destination (onedrive).

What is your rclone version (output from rclone version)

Rclone version v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 Enterprise 64 bits

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Onedrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

RCLONE sync google-drive-backend: onedrive-backend:gdrive-folder --fast-list --drive-acknowledge-abuse --backup-dir  onedrive-backend:--rclone--versions/gdrive-folder/

The rclone config contents with secrets removed.

[google-drive-backend]
type = drive
client_id = [redacted]
client_secret = [redacted]
scope = drive.readonly
token = {"access_token":"[redacted]"}

[onedrive-backend]
type = onedrive
client_id = [redacted]
client_secret = [redacted]
token = {"access_token":"[redacted]"}
drive_id = [redacted]
drive_type = personal

A log from the command with the -vv flag

...
148016	ERROR : One drive root '[redacted]': not deleting files as there were IO errors
148017	ERROR : One drive root '[redacted]': not deleting directories as there were IO errors
...     
296044	ERROR : One drive root '[redacted]': not deleting files as there were IO errors
296045	ERROR : One drive root '[redacted]': not deleting directories as there were IO errors
...     
444075	ERROR : One drive root '[redacted]': not deleting files as there were IO errors
444076	ERROR : One drive root '[redacted]': not deleting directories as there were IO errors
...
#2

That is what is supposed to happen if there are IO errors.

You'd have to share the full debug log so you can see what's going on.

#3

But new files are being copied without problems...

It's a 500k lines file...

#4

How would you suggest I help you out if I can't a log file of the problem?

#5

Well... I thought I had provided the necessary information, but that doesn't seem to be the case...

I suggest you start by evaluating all 443986 lines of this file (wait for it to load, give the site nginx some time...):

https://zerobin.net/?c8dde24d86b3b863#hwkuXc3SZx0Ci4r+WiCd59a3KPpsVHcw8G08sxvrtNU=

password: error

Seems to me to be important information judging by your answer.

Thank you for your support!

#6

You'd want to clean up these errors:

2021/07/07 23:01:05 ERROR : Attempt 2/3 failed with 20 errors and: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:[redacted]::
2021/07/07 23:05:14 ERROR : _multimidia/BH e arredores/Ouro Preto - Luiz Fontana/ouropreto.mg.gov.br/luizfontana/index1c3c.json: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:[redacted]::
2021/07/07 23:05:14 ERROR : _multimidia/BH e arredores/Ouro Preto - Luiz Fontana/ouropreto.mg.gov.br/luizfontana/index3361.json: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:[redacted]::
2021/07/07 23:05:15 ERROR : _multimidia/BH e arredores/Ouro Preto - Luiz Fontana/ouropreto.mg.gov.br/luizfontana/index3165.json: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:[redacted]::
2021/07/07 23:05:15 ERROR : _multimidia/BH e arredores/Ouro Preto - Luiz Fontana/ouropreto.mg.gov.br/luizfontana/index1d10.json: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:[redacted]::
2021

I'd imagine you need:

https://rclone.org/drive/#drive-acknowledge-abuse

#7

he already has '--drive-acknowledge-abuse' in his command. He'll need '--ignore-errors' but that can be dangerous too if he's not careful. Or exclude those files.

FYI: Fails when file is blocked by Google Drive as malware, despite downloading it · Issue #2338 · rclone/rclone · GitHub

#8

The second item is duplicates:

2021/07/07 22:18:51 NOTICE: ______Temp - organizar os arquivos duplicados/Foto de Luiz: Duplicate object found in source - ignoring:[redacted]::
2021/07/07 22:18:51 NOTICE: ______Temp - organizar os arquivos duplicados/Foto de Luiz: Duplicate object found in source - ignoring:[redacted]::
2021/07/07 22:18:51 NOTICE: ______Temp - organizar os arquivos duplicados/Foto de Luiz: Duplicate object found in source - ignoring:[redacted]::
2021/07/07 22:18:51 NOTICE: ______Temp - organizar os arquivos duplicados/Foto de Luiz: Duplicate object found in source - ignoring:[redacted]::
2021/07/07 22:18:51 NOTICE: ______Temp - organizar os arquivos duplicados/Foto de Luiz: Duplicate object found in source - ignoring:[redacted]::

You want to dedupe the source.

#9

Is the flag not working then? That issue relates back to a team drive and this looks to be a regular drive.

#10

He can check with dump -vv --dump headers that it's being passed. It may be a share though without owner permissions. The message talks about "Only the owner can download abusive files" implying he isn't.

This one is definately the issue though:

2021/07/07 23:23:27 Failed to sync with 20 errors: last error was: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:::

#11

Hmm, none of the flags or the rclone.conf indicate a shared drive.

If it isn't a share, we'd need to copy with --dump headers on one bad file.

2021/07/07 23:05:14 ERROR : _multimidia/BH e arredores/Ouro Preto - Luiz Fontana/ouropreto.mg.gov.br/luizfontana/index1c3c.json: Failed to copy: failed to open source object: open file failed: googleapi: Error 403: Only the owner can download abusive files., cannotDownloadAbusiveFile:[redacted]::

As an example.

#12

I wonder if this causes that.

#13

The source could still have shared files in it though. If someone shares a file with you, and you select 'add to drive' it will add it as a non-owner of that file.

#14

Oh right, I forgot about that as I never use the GUI and I'm just solely rclone via encryption.

#15

Exactly, I already use the option. About these files I had already identified the cause, which is related to the owner of the files (Google login).

Yes, I had already identified this one too, it's related to the fact that google accepts files with the same name, it will be resolved.

It's a regular drive.

This entry is related to the source (Google Drive), and the problem here is related to the destination (Onedrive), where the deleted folders are not being moved to the folder indicated in --backup-dir.

#16

If the source has issues, it's not going to delete on the destination due to IO errors.

You have to clean up the source.

#17

When you fix these files from having issues, the backup will work and the deletes. The error is pretty clear. Are those files shared files? Are you actually the owner? If not, the google policy seems to be that you can't download them even though you have specified acknowledge-abuse. Exclude those, and things should work.

#18

In the root of Google drive (source) I have 33 folders. In the root of Onedrive (destination) I have 40. The errors mentioned above (owner and dedupe) refer only to two of these folders.

Some of these folders were renamed in the source, sync copied them to the destination using the new name but did not move the folder with the old name according to the --backup-dir option.

Others were deleted in the source, but also not moved according to the --backup-dir option.

Are you saying that the above errors are preventing the other folders from syncing correctly?

#19

Yes, that's what both of us are saying.

#20

OK. Definitive solution: I deleted the two folders.

I'm running sync again, later I'll report the result here.