Unreliable one-way sync Google Photos to my local drive

Help and Support
1

What is the problem you are having with rclone?

I am unable to reliably one-way sync Google Photos to my local drive. Many of the files come in, but can't trust that all do.

What is your rclone version (output from rclone version)

rclone v1.56.1

  • os/version: linuxmint 20.2 (64 bit)
  • os/kernel: 5.11.0-36-lowlatency (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.8
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone --config /home/kenny/.config/rclone/rclone.conf sync -l -v -P GPhotos:/media/by-year /media/kenny/Timeshift/GPhotos --log-file /home/kenny/logs/rclone-gphotos.log


#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->
[OneDrive]
type = onedrive
token =
drive_id = 
drive_type = personal

[S3]
type = s3
provider = AWS
env_auth = false
access_key_id = 
secret_access_key = 
region = us-east-1

[Google-MyDrive]
type = drive
scope = drive
token =

[GPhotos]
type = google photos
token = 

[Google-SharedDrive]
type = drive
scope = drive
token = 
team_drive = 
root_folder_id =

A log from the command with the -vv flag

https://gist.githubusercontent.com/kendoori/e9187985530779cdbf53690c7d92396d/raw/ef6df20d74a186677490d53ad830a502c2c09bdf/gistfile1.txt

2

hello,

this seems to be some permissions error.
and that log look very strange to me, do not think i ever saw a rclone log like that before.

if you repeat the sync, do the same files errors out each time?
if true, try to copy a single file and post the full debug log?
to create a smaller debug log, add --retries=1 --low-level-retries=1

not sure it matters but i do not see client id in your google remotes

[gphotos]
type = google photos
client_id = 
client_secret = 
token =
3

You might be running into google's rate limiter. You can try adding --tpslimit 2 to slow everything down.

4

I bet you are right. I will try that.

5

hi,
i looked at the log, did not see any pacer messages, just 403.

i know there gphotos does suffer from pacer issues

was there something in that log that indicates a pacer issue?

6

I am trying this setting, which I saw elsewhere here:
--tpslimit 5 --bwlimit 8650k --fast-list

I am running the job right now, and it seems that consistently it hangs on some very large video files (~20MB).

7

how long has rclone been hanged on that 20MB file?

can you post a debug log?

8

@asdffdsa It can be a few hours... Can you tell me what I need to add to this command set to properly give you the debug info you are looking for?

rclone --config /home/kenny/.config/rclone/rclone.conf sync -l -v -P --tpslimit 5 --bwlimit 8650k --fast-list GPhotos:/media/by-year /media/kenny/Timeshift/GPhotos --log-file /home/kenny/logs/rclone-gphotos.log
9

if the current completes without error, i guess no need for this.

for debug output

  • remove -v
  • add --log-level=DEBUG

@ncw suggested --tpslimit 2

i would use client id
https://rclone.org/drive/#making-your-own-client-id

10

Is the Drive client-id technique relevant for Photos?

11

not 100% sure and not an expert with gphotos.
during the process to create the gphoto remote, rclone does ask for it.

i did mention that in my first post and @ncw did not comment/suggest that it was needed.

let's see what, if any, errors you get with the current running rclone command...

12

20MB doesn't sound like a very big file. Did you mean 20GB?

Are you getting errors in the log with -vv?

--tpslimit is usually the answer to google photos problems.

I had a conversation with a google engineer who suggested the best rate might be --tpslimit 0.5 !!

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.