Copying Google Drive to Google Photos

I'm trying to copy tens of thousands of assets from Google Drive to Google Photos. I've tried a whole bunch of options, including transfers from 1 to 20 and tpslimit as low as 1, but no matter what I try I keep getting lots of
error Quota exceeded for quota 'concurrent write request' of service 'photoslibrary.googleapis.com'. (429 RESOURCE_EXHAUSTED)

I am watching how often I get Committed sync batch length __ starting with: messages in the log, as that seems to indicate "actual" successful uploads, and it does a few dozen in a given hour, then nothing at all for 2-4 hours.

Hour	          batches  	assets  
2024/12/05 14	     138	 786
2024/12/05 15	     86	     493
2024/12/05 16	     0	     0
2024/12/05 17	     0	     0
2024/12/05 18	     13	     79
2024/12/05 19	     8	     13
2024/12/05 20	     2	     3
2024/12/05 21	     15	     96
2024/12/05 22	     0	     0
2024/12/05 23	     15	     83
2024/12/06 00	     0	     0
2024/12/06 01	     12	     60
2024/12/06 02	     13	     63
2024/12/06 03	     0	     0
2024/12/06 04	     5	     18
2024/12/06 05	     156     903
2024/12/06 06	     0	     0
2024/12/06 07	     0	     0
2024/12/06 08	     0	     0
2024/12/06 09	     2	     28
2024/12/06 10	     0	     0
2024/12/06 11	     1	     1

Some hours have hundreds of successful uploads, other have zero.

Very frustrating.

My current command is

rclone copy -Pvv \
  --transfers 16 \
  --tpslimit 1 \
  --drive-chunk-size 128M \
  --exclude "*.{xml,bin,json,thm,DS_Store}*" --ignore-case \
  --log-file=rclone-photos-"$(date +%Y%m%d_%H%M%S)".log \
gdrive:___ gphotos:album/rclone"

I tried the option gphotos-batch-mode async but that crashed as I reported here: `--gphotos-batch-mode async` error · Issue #8233 · rclone/rclone · GitHub

Can anything be done to make this more efficient?

$ rclone version
rclone v1.68.2
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-1015-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.23.3
- go/linking: static
- go/tags: none

hi,

are you using client id+secret, for both remotes?

might try to shrink --drive-chunk-size

Yes I'm using unique client IDs for both.

I've tried several chunk sizes, doesn't make much difference. Definitely no diff with regards to the 429 errors.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.