Rclone GCS timemachine issue

I am using rclone to attempt to sync the contents of my macos timemachine backup located on a debian NAS. Everything appears to be working and I was able to successfully sync the backup the first time (close to 500gb).

Now on trying to sync the content, I get an error during the sync:

2019-10-07 17:30:51 ERROR : : Couldn't delete: googleapi: Error 404: No such object: 2019-workstation-backup/Mark, notFound

The debugging error log doesn't show any additional detail.

Thanks.

Can you paste the command you are using and the log with -vv of the failure?

You are syncing debian NAS to google drive? is that right?

I'm just in the middle of syncing the changes from last night's backup, here is the command:

rclone sync -P --fast-list -vv /mnt/storage/timemachine/L5172164491.sparsebundle/ GCS:2019-workstation-backup/Mark

the /mnt/storage/timemachine/ location is a share on my debian NAS that is used by timemachine on my macos to perform nightly backups. The sparsebundle format contains the individual bands that the backup gets split into - probably 10s of thousands of files. It appears that rclone understands the structure, its just this one error.

Thanks!

Thanks for explaining your setup.

That error appears to be saying it is trying to delete the file "2019-workstation-backup/Mark" on "googleapi:" but it wasn't found.

Which is weird since that path isn't a file, or maybe it is?

Do the other files copy OK? Or does the sparsebundle just copy over as one file?

Which version of rclone are you using? Can you try the latest release (v1.49.5) if you aren't already?

What does rclone lsl GCS:2019-workstation-backup/Mark show?

From my attempts to help earlier in Slack, it seems like 2019-workstation-backup/Mark is the ddestination directory, and most files sync properly - but then this error also occurs and we were not able to nail down (even with debug) what exact object this is referring to. Or at least it is not obvious - why would it be trying to delete the destination directory if that's what it means - and even so why would it not be finding it?

We tried dedupe on a hunch since that can sometimes cause some weird behaviour, but no hits on that scan. Hopefully you can provide some better insights NCW :slight_smile:

It is referring to the root I think, - the filename should be within the two colons here ERROR : : Couldn't delete: and an empty file is the root directory.

Can you post the log file generated with these extra flags -vv --dump bodies --retries 1 --log-file timemachine.log - that will at least give a clue as to what rclone is doing! If you feel it has sensitive stuff in you can email it to me nick@craig-wood.com (put a link to this post in the email please).

It is also worth giving it a go with the latest beta

That would be a crazy log, as the -vv dumps a ton of info and it takes approximately 2 hours to complete.

I updated to rclone v1.49.5-189-g0cac9d9b-beta and I get the same error

root@CorePlex:/tmp# rclone --version
rclone v1.49.5-189-g0cac9d9b-beta
- os/arch: linux/amd64
- go version: go1.13.1
root@CorePlex:/tmp# rclone sync -P --fast-list /mnt/storage/timemachine/L5172164491.sparsebundle/ GCS:2019-workstation-backup/Mark
2019-10-08 22:22:59 ERROR : : Couldn't delete: googleapi: Error 404: No such object: 2019-workstation-backup/Mark, notFound
2019-10-08 22:22:59 ERROR : GCS bucket 2019-workstation-backup path Mark: not deleting directories as there were IO errors
2019-10-08 22:22:59 ERROR : Attempt 1/3 failed with 3 errors and: failed to delete 1 files
2019-10-08 22:23:09 ERROR : : Couldn't delete: googleapi: Error 404: No such object: 2019-workstation-backup/Mark, notFound
2019-10-08 22:23:10 ERROR : GCS bucket 2019-workstation-backup path Mark: not deleting directories as there were IO errors
2019-10-08 22:23:10 ERROR : Attempt 2/3 failed with 3 errors and: failed to delete 1 files
2019-10-08 22:23:20 ERROR : : Couldn't delete: googleapi: Error 404: No such object: 2019-workstation-backup/Mark, notFound
2019-10-08 22:23:20 ERROR : GCS bucket 2019-workstation-backup path Mark: not deleting directories as there were IO errors
2019-10-08 22:23:20 ERROR : Attempt 3/3 failed with 3 errors and: failed to delete 1 files
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 3 (retrying may help)
Checks:            155742 / 155742, 100%
Transferred:            0 / 0, -
Elapsed time:       600ms
2019/10/08 22:23:20 Failed to sync with 3 errors: last error was: failed to delete 1 files

:slight_smile: Could you maybe create a sparsebundle with 1 file in and see if it has the same problem then send the log for that if it does?

Thanks for testing!