Rclone and 5TB in Google Photos

[jvehent (Julien Vehent)]

Hi everyone,

I've been using Rclone for a few years to sync a large photo collection to Google Photos. It works, for the most part, but struggles a bit and I'm looking for ways to make subsequent runs more efficient.

The command I typically use is this one:

PS C:\rclone> .\rclone.exe sync -P --tpslimit 1 --transfers 1 
--retries-sleep 1m H:\Photos\ remote_google_photos:album/ 
--include "**.{JPG,jpg,PNG,png,TIFF,tiff,NEF,nef,ARW,arw}"

This seem to alleviate quota errors and, for the most part, runs without issue even for several days. Here's the current run I launched yesterday:

Transferred:       59.897Gi / 92.423 GiByte, 65%, 41.981 KiByte/s, ETA 1w2d9h39m33s
Errors:               125 (retrying may help)
Checks:            344425 / 344425, 100%
Transferred:         9711 / 19713, 49%
Elapsed time:   9h31m12.6s

This run is estimated to take over a week to complete. My issue is that 99% of the files in this collection have already been uploaded to Google Photos, and there may only be a few outliers that need to be readded, so an entire week of upload seems excessive.

I've been looking for a way to keep state between rclone and Google Photos. It doesn't look like Google Photos has an API to return the hash of a photo, which would have been my preferred approach to check that the upstream and local files are identical without reuploading them. Failing that, I was hoping Rclone could store a local DB with a list of photos already uploaded, and check that they have not changed locally before uploading them again. Is there a way to achieve this?

Thank you for your assistance!
Julien

[kapitainsky]

@jvehent

Transferred:       59.897Gi / 92.423 GiByte, 65%, 41.981 KiByte/s, ETA 1w2d9h39m33s
Errors:               125 (retrying may help)
Checks:            344425 / 344425, 100%
Transferred:         9711 / 19713, 49%
Elapsed time:   9h31m12.6s

I think all answer is here.

Already uploaded photos have been already checked - Checks: 344425 / 344425, 100% - there is no issue with it

What takes time is uploading new photos - Transferred: 9711 / 19713, 49%

It is slow as your upload speed is very slow: 41.981 KiByte/s,

@jvehent

My issue is that 99% of the files in this collection have already been uploaded to Google Photos, and there may only be a few outliers that need to be readded, so an entire week of upload seems excessive.

According to stats above - you have 344425 files in the cloud - and you are uploading 19713 new ones - it is like 6% not 1%.

No local DB can help with it. What is your normal internet upload speed? You can use https://www.speedtest.net/ to test.

You can try to increase --tpslimit 1 --transfers 1, run your sync with '-vv' and post it here. I would try with --tpslimit 9 --transfers 4 and removed --retries-sleep 1m. Also as it is gdrive add --fast-list - doc

PS C:\rclone> .\rclone.exe sync -P --tpslimit 9 --transfers 4 
--fast-list H:\Photos\ remote_google_photos:album/ 
--include "**.{JPG,jpg,PNG,png,TIFF,tiff,NEF,nef,ARW,arw}" -vv --log-file disk:\path\to\rclone.log

As with this amount of files log file will be huge I suggest you create test folder on gdrive and try to sync small amounts of files there. It is impossible to say without log why it is so slow e.g. maybe you should add --drive-skip-checksum-gphotos?

Are you using your personal client ID? You should.

[jvehent (Julien Vehent)]

Thanks @kapitainsky .

There are no bandwidth issue from this connection. This is a speedtest ran from the computer that also runs rclone.

I will retry with --tpslimit 9 --transfers 4 --fast-list -vv --log-file rclone.log and post the results here, but in the past increasing --tpslimit and --transfers yielded many issues with API quotas. We'll see if that has changed.

[kapitainsky]

Yes your connection definitely is not a problem.

The most important thing is to create your own App ID - when using default rclone one you are sharing it with thousands of other users... you never know what speed you can expect.

From my experience gdrive without personal App ID is barely usable. Definitely for big syncs like yours.

[kapitainsky]

Here some recent post dealing with setting gdrive App ID - might be useful:

GDrive expiring after a week?

rclone version rclone v1.62.2 - os/version: raspbian 11.7 - os/kernel: 6.1.21-v7+ (armv7l) - os/type: linux - os/arch: arm (ARMv7 compatible) - go/version: go1.20.2 - go/linking: static - go/tags: none I have activated GDrive share with client_id and Client_secret. rclone is running as a service. After a week of operations, it stopped working due to: 2023/06/14 12:40:50 INFO : Google drive root '': Failed to get StartPageToken: Get "https://www.googleapis.com/drive/v3/changes/startPageToken…

[jvehent (Julien Vehent)]

@kapitainsky

Are you using your personal client ID ? You should.

I am indeed hitting the API limits.

2023/06/26 07:55:46 DEBUG : pacer: low level retry 3/10 (error Quota exceeded for quota 'concurrent write request' of service 'photoslibrary.googleapis.com' for consumer 'project:202264815644'. (429 RESOURCE_EXHAUSTED))
2023/06/26 07:55:46 DEBUG : pacer: Rate limited, increasing sleep to 1.449243532s

Do the config options for google drive also apply to google photos? I'm not sure they're effective after using my own credentials:

[remote_google_photos]
type = google photos
token = {"access_token":"y....
client_id = "6983...."
client_secret = "GOCSPX-......"

[jvehent (Julien Vehent)]

My config is incorrect, rclone is clearly still using the default credentials.

for consumer 'project_number:202264815644'. (429 RESOURCE_EXHAUSTED))

// Constants
const (
	rcloneClientID              = "202264815644-rt1o1c9evjaotbpbab10m83i8cnjk077.apps.googleusercontent.com"

[kapitainsky]

@jvehent

project:202264815644

means you are using default rclone client_id not yours.

It can happen when you added your own client_id after initial remote creation - token is bound to client_id during authentication,

Delete token from remote config and run:

rclone config reconnect remote_google_photos:

It will generate new token and bind it to your client ID. Authentication screen should have also this scary look:

unless you configured it for testing - but then it expires in 1 week

[kapitainsky]

Then I think that:

rclone.exe sync -P --tpslimit 9 --tpslimit-burst 0 H:\Photos\ remote_google_photos:album/ --include "**.{JPG,jpg,PNG,png,TIFF,tiff,NEF,nef,ARW,arw}" --drive-skip-checksum-gphotos

should work.

As you have many small files it might be needed to limit transfers by adding e.g. --transfers 3 - but you will only see when you try.

Either way the biggest issue is abysmal speed you are having at the moment. If this is fixed it should work smooth - of course it is 5TB/350k files so it wont finish in 1h:) but should not be 1 week neither.

[jvehent (Julien Vehent)]

I think you were correct about the access token. The reconnect command failed with an invalid app error, but I redid the config and got it to work on my own app id now.

However, instead of fixing the issue, it seem to have make the quota errors even worse

2023/06/26 08:16:05 DEBUG : pacer: Rate limited, increasing sleep to 8.745304981s
2023/06/26 08:16:06 DEBUG : pacer: Reducing sleep to 0s
2023/06/26 08:16:07 DEBUG : pacer: low level retry 3/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:07 DEBUG : pacer: Rate limited, increasing sleep to 1.838325651s
2023/06/26 08:16:07 DEBUG : pacer: low level retry 2/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:07 DEBUG : pacer: Rate limited, increasing sleep to 2.21699496s
2023/06/26 08:16:07 DEBUG : pacer: low level retry 8/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:07 DEBUG : pacer: Rate limited, increasing sleep to 4.331509907s
2023/06/26 08:16:07 DEBUG : pacer: low level retry 1/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:07 DEBUG : pacer: Rate limited, increasing sleep to 8.368607468s
2023/06/26 08:16:07 DEBUG : pacer: low level retry 4/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:07 DEBUG : pacer: Rate limited, increasing sleep to 16.502478026s
2023/06/26 08:16:09 DEBUG : pacer: low level retry 3/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:09 DEBUG : pacer: Rate limited, increasing sleep to 16.985996617s
2023/06/26 08:16:25 DEBUG : pacer: low level retry 9/10 (error Quota exceeded for quota metric 'Write requests' and limit 'Write requests per minute per user' of service 'photoslibrary.googleapis.com' for consumer 'project_number:698393376087'. (429 RESOURCE_EXHAUSTED))
2023/06/26 08:16:25 DEBUG : pacer: Rate limited, increasing sleep to 16.05522497s

[kapitainsky]

Add --fast-list

rclone.exe sync -P --tpslimit 9 --tpslimit-burst 0 H:\Photos\ remote_google_photos:album/ --include "**.{JPG,jpg,PNG,png,TIFF,tiff,NEF,nef,ARW,arw}" --drive-skip-checksum-gphotos --fast-list

[jvehent (Julien Vehent)]

I have. This is was is running now. It stalls completely.

PS C:\rclone> .\rclone.exe sync -P --tpslimit 9 --transfers 4 
--tpslimit-burst 0 --drive-skip-checksum-gphotos --fast-list 
-vv --log-file rclone.log --retries-sleep 1m 
H:\AllMyPix\ remote_photos_vehent:album/ 
--include "**.{JPG,jpg,PNG,png,TIFF,tiff,NEF,nef,ARW,arw}"

Transferred:        2.811Mi / 17.557 GiByte, 0%, 4.570 KiByte/s, ETA 6w4d14h39m42s
Transferred:           14 / 10026, 0%
Elapsed time:      1m18.1s
Transferring:
 * 000a_Tours-2002-2004/2….JPG:  0% /129.750Ki, 0/s, -
 * 000a_Tours-2002-2004/2….JPG:100% /160.043Ki, 1.049Ki/s, 0s
 * 000a_Tours-2002-2004/2….JPG:100% /89.804Ki, 684/s, 0s
 * 000a_Tours-2002-2004/2….JPG:100% /102.796Ki, 945/s, 0s

[kapitainsky]

Also you could use your original throttling settings:) my settings work for me - but who knows maybe throttling depends on locality and google servers load? no idea

rclone.exe sync -P --tpslimit 1 --tpslimit-burst 0 --transfers 1 H:\Photos\ remote_google_photos:album/ --include "**.{JPG,jpg,PNG,png,TIFF,tiff,NEF,nef,ARW,arw}" --drive-skip-checksum-gphotos --fast-list

This is always run and try approach.

For sure --fast-list helps with number of transactions and --drive-skip-checksum-gphotos prevents using broken photos hashes

[jvehent (Julien Vehent)]

I'm experimenting with --tpslimit and --transfers. It looks like I can safely increase --tpslimit up to 10 but I cannot bump --transfers past 1. As soon as I try to have even 2 transfers in parallel, I hit the quota.

2023/06/26 08:42:18 DEBUG : pacer: low level retry 1/10 (error Quota exceeded for quota 'concurrent write request' of service 'photoslibrary.googleapis.com' for consumer 'project:698393376087'. (429 RESOURCE_EXHAUSTED))

[kapitainsky]

True you are uploading.. I mostly download - explains difference.

What about speed? Any better?

[jvehent (Julien Vehent)]

Yes, it's much faster than the original 40 KiByte/s. Averaging 1.9 MiByte/s now!

Transferred:      706.333Mi / 55.595 GiByte, 1%, 1.963 MiByte/s, ETA 7h57m17s
Errors:                 4 (retrying may help)
Transferred:          156 / 10165, 2%
Elapsed time:      5m14.6s

Errors are always way down in the dashboard.

Thanks for the help! Hopefully that helps others too.

Side question: has anyone tried to register rclone with the partner program?

[kapitainsky]

Google is terrible with throttling. rclone sync is clearly not usage pattern they encourage:)

I think here the key was to make sure that your own AppID is in use and --drive-skip-checksum-gphotos to avoid re-uploading.

Let us know if this sync finishes trouble free - always useful to see real life results especially on big dataset like yours.