Google Photos API limiting help

Help and Support
1

What is the problem you are having with rclone?

I'm having trouble copying a large number of photos from a local machine to google photos without going over the 10k daily API limit. The base directory has many other files, and I'd like to separate my videos from my photos so I'm using a find command based on filetype and then exec'ing rclone copy so they all end up directly in the desired album. I got about 1000 photos uploaded the first day, but hit the 10k limit on the second day just checking existing files.

Is there a more elegant solution than using the find command while still separating photos from videos? My goal is to get all photos in one album and all videos in another.

Edit: Forgot to mention, I am using my own client id / secret.

image
image1639×461 32.6 KB

Run the command 'rclone version' and share the full output of the command.

# rclone --version
rclone v1.59.2
- os/version: slackware 14.2+ (64 bit)
- os/kernel: 5.10.28-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

find /directory/with/many/types/of/files/ -type f \( -iname \*.jpg -o -iname \*.jpeg -o -iname \*.png -o -iname \*.arw \) -exec rclone copy {} photos:album/My\ Album \;

The rclone config contents with secrets removed.

[photos]
type = google photos
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com
client_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2022-10-21T17:49:29.138368102-07:00"}

A log from the command with the -vv flag

2022/10/22 01:33:52 DEBUG : 3 go routines active
2022/10/22 01:33:52 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/IMG_3494.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:33:52 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/IMG_3494.jpg"
2022/10/22 01:33:52 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:33:52 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/IMG_3494.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:33:52 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:33:52 DEBUG : Google Photos path "album/My Album": NewObject: remote="IMG_3494.jpg"
2022/10/22 01:33:52 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:33:55 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:33:55 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:33:55 DEBUG : IMG_3494.jpg: Size: 
2022/10/22 01:33:55 DEBUG : IMG_3494.jpg: >Size: 
2022/10/22 01:33:55 DEBUG : IMG_3494.jpg: Sizes identical
2022/10/22 01:33:55 DEBUG : IMG_3494.jpg: Unchanged skipping
2022/10/22 01:33:55 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.4s

2022/10/22 01:33:55 DEBUG : 3 go routines active
2022/10/22 01:33:55 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/IMG_4704.JPG" "photos:album/My Album" "-vv"]
2022/10/22 01:33:55 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/IMG_4704.JPG"
2022/10/22 01:33:55 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:33:55 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/IMG_4704.JPG", "/directory/with/many/types/of/files/foo"
2022/10/22 01:33:55 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:33:55 DEBUG : Google Photos path "album/My Album": NewObject: remote="IMG_4704.JPG"
2022/10/22 01:33:55 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:33:58 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:33:58 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:33:58 DEBUG : IMG_4704.JPG: Size: 
2022/10/22 01:33:58 DEBUG : IMG_4704.JPG: >Size: 
2022/10/22 01:33:58 DEBUG : IMG_4704.JPG: Sizes identical
2022/10/22 01:33:58 DEBUG : IMG_4704.JPG: Unchanged skipping
2022/10/22 01:33:58 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.2s

2022/10/22 01:33:58 DEBUG : 3 go routines active
2022/10/22 01:33:59 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-1.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:33:59 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-1.jpg"
2022/10/22 01:33:59 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:33:59 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-1.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:33:59 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:33:59 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-1.jpg"
2022/10/22 01:33:59 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:02 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:02 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:02 DEBUG : bar-1.jpg: Size: 
2022/10/22 01:34:02 DEBUG : bar-1.jpg: >Size: 
2022/10/22 01:34:02 DEBUG : bar-1.jpg: Sizes identical
2022/10/22 01:34:02 DEBUG : bar-1.jpg: Unchanged skipping
2022/10/22 01:34:02 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.2s

2022/10/22 01:34:02 DEBUG : 3 go routines active
2022/10/22 01:34:02 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-2.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:02 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-2.jpg"
2022/10/22 01:34:02 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:02 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-2.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:02 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:02 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-2.jpg"
2022/10/22 01:34:02 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:05 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:05 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:05 DEBUG : bar-2.jpg: Size: 
2022/10/22 01:34:05 DEBUG : bar-2.jpg: >Size: 
2022/10/22 01:34:05 DEBUG : bar-2.jpg: Sizes identical
2022/10/22 01:34:05 DEBUG : bar-2.jpg: Unchanged skipping
2022/10/22 01:34:05 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.3s

2022/10/22 01:34:05 DEBUG : 3 go routines active
2022/10/22 01:34:05 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-3.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:05 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-3.jpg"
2022/10/22 01:34:05 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:05 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-3.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:05 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:05 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-3.jpg"
2022/10/22 01:34:05 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:09 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:09 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:09 DEBUG : bar-3.jpg: Size: 
2022/10/22 01:34:09 DEBUG : bar-3.jpg: >Size: 
2022/10/22 01:34:09 DEBUG : bar-3.jpg: Sizes identical
2022/10/22 01:34:09 DEBUG : bar-3.jpg: Unchanged skipping
2022/10/22 01:34:09 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.4s

2022/10/22 01:34:09 DEBUG : 3 go routines active
2022/10/22 01:34:09 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-4.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:09 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-4.jpg"
2022/10/22 01:34:09 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:09 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-4.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:09 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:09 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-4.jpg"
2022/10/22 01:34:09 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:12 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:12 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:12 DEBUG : bar-4.jpg: Size: 
2022/10/22 01:34:12 DEBUG : bar-4.jpg: >Size: 
2022/10/22 01:34:12 DEBUG : bar-4.jpg: Sizes identical
2022/10/22 01:34:12 DEBUG : bar-4.jpg: Unchanged skipping
2022/10/22 01:34:12 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.5s

2022/10/22 01:34:12 DEBUG : 3 go routines active
2022/10/22 01:34:12 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-5.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:12 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-5.jpg"
2022/10/22 01:34:12 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:12 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-5.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:12 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:12 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-5.jpg"
2022/10/22 01:34:12 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:15 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:15 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:15 DEBUG : bar-5.jpg: Need to transfer - File not found at Destination
2022/10/22 01:34:15 DEBUG : Google Photos path "album/My Album": Put: src=bar-5.jpg
2022/10/22 01:34:15 DEBUG : bar-5.jpg: Update: src=bar-5.jpg
2022/10/22 01:34:17 DEBUG : bar-5.jpg: >Update: err=<nil>
2022/10/22 01:34:17 DEBUG : Google Photos path "album/My Album": >Put: 
2022/10/22 01:34:17 DEBUG : bar-5.jpg: Size: 
2022/10/22 01:34:17 DEBUG : bar-5.jpg: >Size: 
2022/10/22 01:34:17 INFO  : bar-5.jpg: Copied (new)
2022/10/22 01:34:17 INFO  : 
Transferred:   	    2.544 MiB / 2.544 MiB, 100%, 2.543 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         5.1s

2022/10/22 01:34:17 DEBUG : 4 go routines active
2022/10/22 01:34:17 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-6.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:17 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-6.jpg"
2022/10/22 01:34:17 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:17 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-6.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:17 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:17 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-6.jpg"
2022/10/22 01:34:17 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:21 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:21 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:21 DEBUG : bar-6.jpg: Size: 
2022/10/22 01:34:21 DEBUG : bar-6.jpg: >Size: 
2022/10/22 01:34:21 DEBUG : bar-6.jpg: Sizes identical
2022/10/22 01:34:21 DEBUG : bar-6.jpg: Unchanged skipping
2022/10/22 01:34:21 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.3s

2022/10/22 01:34:21 DEBUG : 3 go routines active
2022/10/22 01:34:21 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-7.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:21 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-7.jpg"
2022/10/22 01:34:21 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:21 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-7.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:21 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:21 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-7.jpg"
2022/10/22 01:34:21 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:24 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:24 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:24 DEBUG : bar-7.jpg: Size: 
2022/10/22 01:34:24 DEBUG : bar-7.jpg: >Size: 
2022/10/22 01:34:24 DEBUG : bar-7.jpg: Sizes identical
2022/10/22 01:34:24 DEBUG : bar-7.jpg: Unchanged skipping
2022/10/22 01:34:24 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.4s

2022/10/22 01:34:24 DEBUG : 3 go routines active
2022/10/22 01:34:24 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-8.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:24 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-8.jpg"
2022/10/22 01:34:24 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:24 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-8.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:24 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:24 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-8.jpg"
2022/10/22 01:34:24 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:27 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:27 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:27 DEBUG : bar-8.jpg: Size: 
2022/10/22 01:34:27 DEBUG : bar-8.jpg: >Size: 
2022/10/22 01:34:27 DEBUG : bar-8.jpg: Sizes identical
2022/10/22 01:34:27 DEBUG : bar-8.jpg: Unchanged skipping
2022/10/22 01:34:27 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.2s

2022/10/22 01:34:27 DEBUG : 3 go routines active
2022/10/22 01:34:27 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-9.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:27 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-9.jpg"
2022/10/22 01:34:27 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:27 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-9.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:27 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:27 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-9.jpg"
2022/10/22 01:34:27 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:31 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:31 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:31 DEBUG : bar-9.jpg: Size: 
2022/10/22 01:34:31 DEBUG : bar-9.jpg: >Size: 
2022/10/22 01:34:31 DEBUG : bar-9.jpg: Sizes identical
2022/10/22 01:34:31 DEBUG : bar-9.jpg: Unchanged skipping
2022/10/22 01:34:31 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.3s

2022/10/22 01:34:31 DEBUG : 3 go routines active
2022/10/22 01:34:31 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-10.png" "photos:album/My Album" "-vv"]
2022/10/22 01:34:31 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-10.png"
2022/10/22 01:34:31 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:31 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-10.png", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:31 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:31 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-10.png"
2022/10/22 01:34:31 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:34 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:34 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:34 DEBUG : bar-10.png: Size: 
2022/10/22 01:34:34 DEBUG : bar-10.png: >Size: 
2022/10/22 01:34:34 DEBUG : bar-10.png: Sizes identical
2022/10/22 01:34:34 DEBUG : bar-10.png: Unchanged skipping
2022/10/22 01:34:34 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.4s

2022/10/22 01:34:34 DEBUG : 3 go routines active
2022/10/22 01:34:34 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "/directory/with/many/types/of/files/foo/bar-11.jpg" "photos:album/My Album" "-vv"]
2022/10/22 01:34:34 DEBUG : Creating backend with remote "/directory/with/many/types/of/files/foo/bar-11.jpg"
2022/10/22 01:34:34 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2022/10/22 01:34:34 DEBUG : fs cache: adding new entry for parent of "/directory/with/many/types/of/files/foo/bar-11.jpg", "/directory/with/many/types/of/files/foo"
2022/10/22 01:34:34 DEBUG : Creating backend with remote "photos:album/My Album"
2022/10/22 01:34:34 DEBUG : Google Photos path "album/My Album": NewObject: remote="bar-11.jpg"
2022/10/22 01:34:34 DEBUG : Google Photos path "album/My Album": List: dir=""
2022/10/22 01:34:37 DEBUG : Google Photos path "album/My Album": >List: err=<nil>
2022/10/22 01:34:37 DEBUG : Google Photos path "album/My Album": >NewObject: 
2022/10/22 01:34:38 DEBUG : bar-11.jpg: Size: 
2022/10/22 01:34:38 DEBUG : bar-11.jpg: >Size: 
2022/10/22 01:34:38 DEBUG : bar-11.jpg: Sizes identical
2022/10/22 01:34:38 DEBUG : bar-11.jpg: Unchanged skipping
2022/10/22 01:34:38 INFO  : 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         3.3s
2

Hi D34DC3N73R,

Perhaps you are looking for --include ?

It can be used like this:

rclone copy --include="*.{jpg,jpeg,png,arw}" /directory/with/many/types/of/files/ photos:album/My/Album

and easily tested like this:

rclone lsl  --include="*.{jpg,jpeg,png,arw}" /directory/with/many/types/of/files/
3

Thanks, I was actually just looking at that and wondering if it would reduce API usage at all or if I'd still have trouble.

I'm going to try this when my temp ban / rate limit is lifted.

rclone copy --include="*.{jpg,jpeg,png,arw}" --ignore-case /directory/with/many/types/of/files/ photos:album/My\ Album
4

Yes, it would reduce API usage considerable in most situations.

If you call rclone copy once per file, then rclone will (minimum) need one API call per file to check if the file already exists and needs to be copied.

If you call rclone with a filter, then it only needs one API call per folder to check all the files in the folder. Or maybe something like 1 API call per 1000 files in the folder. So considerable less API calls. There will also be a much lower overhead to authenticate etc.

If you are really concerned then --fast-list will enable you to list multiple folders in one API call, but it may sometimes make your copy slower (despite the name), so read the linked documentation before deciding/trying to use it.

1 Like
5

Why did you add --ignore-case?

I guess you will be much better off with:

 --include="*.{jpg,JPG,jpeg,JPEG,png,PNG,arw,ARW}"
6

It was my impression that --ignore-case would do the same thing as jpg,JPG,jpeg,JPEG etc.

Is there some overhead to using --ignore-case vs listing out extensions? Regardless, thank you very much for the suggestions and explanations.

1 Like
7

Not quite, --ignore-case would search for jpg, jpG, jPg, jPG, Jpg, JpG, JPg, JPG, jpeg, jpeG, ...

I don't know how efficient the rclone/Go pattern matching is, it may not make a difference, I just play it safe.

You are welcome.

1 Like
8

The problem with this method is that photos are not copied directly into the desired album. Files in subdirectories found within the base directory (/directory/with/many/types/of/files/) are uploaded to separate albums. For instance let's take /directory/with/many/types/of/files/foo/bar.jpg when it's copied a new album is created My Album/foo, which is actually an entirely separate album from My Album. How can I get all photos uploaded directly to the intended album My Album?

9

I guess you will have to copy folder by folder like this:

rclone copy --max-depth=1 /directory/with/many/types/of/files/foo photos:album/myAlbum

Disclaimer: I have never tried Google Photos upload, so this is guesswork to be verified by you.

10

Unfortunately, that's not really feasible. There are many levels of nested subdirectories that would make this approach quite painful.

11

It's not exactly efficient but this seems to be more efficient than using find -exec

find /directory/with/many/types/of/files/* -type d | sort | awk '$0 !~ last "/" {print last} {last=$0} END {print last}' |
while read -r n; do
    rclone -v copy --include="*.{jpg,JPG,jpeg,JPEG,png,PNG,arw,ARW}" "$n" photos:album/My\ Album
done

This assumes that files are only in leaf directories (directories that contain no subdirectories), which is true in my case. This appears to be another case for --flatten.

12

OK, how about this?

# Make a temporary flattened folder using hardlinks (selecting the newest on filename clashes)
mkdir /some/temp/folder/with/hardlinks
find /directory/with/many/types/of/files/ -type f -exec cp -lu '{}' /some/temp/folder/with/hardlinks/ ';'
# Copy the flattened folder to your photo album (restartable)
rclone copy -v /some/temp/folder/with/hardlinks/ photos:album/My\ Album

and then add the file type filtering in either the find or rclone command depending on your overall workflow

13

@D34DC3N73R Kindly pinging in case you didn't notice my above post due to the forum issue.

1 Like
14

In setting this up, I've realized there are many duplicate file names that are actually different files. For example
/directory/with/many/types/of/files/event-r1/photos/100-400mm/DSC00822.ARW
/directory/with/many/types/of/files/event-r2/photos/200-600mm/DSC00822.ARW

Google Photos does seem to be able to handle duplicate file names in the same album, but I don't think I'll be able to use the hard link solution.

15

Good point.

If you have a lot of photos and may have to restart several times, then you can save some time and API calls by keeping track on the folders successfully uploaded by doing something like this (pseudo code):

find /directory/with/many/types/of/files/* -type d | sort | awk '$0 !~ last "/" {print last} {last=$0} END {print last}' |
while read -r n; do
    if (grep -Fxq "$n" completed_folders.txt) == 1 then 
         if (rclone -v copy --include="*.{jpg,JPG,jpeg,JPEG,png,PNG,arw,ARW}" "$n" photos:album/My\ Album) 
             echo $n >> completed_folders.txt
done
16

Thanks, that's a good idea.
I'm using this for now

find /directory/with/many/types/of/files/* -type d | sort | awk '$0 !~ last "/" {print last} {last=$0} END {print last}' |
while read -r n; do
    if (grep -Fxq "$n" photo_folders.txt); then
        echo "$n has been uploaded"; else
        rclone -v copy --include="*.{jpg,JPG,jpeg,JPEG,png,PNG,arw,ARW}" "$n" photos:album/My\ Album
        echo $n >> photo_folders.txt;
    fi
done

I'm far from a bash wizard, so if you have any suggestions, I'm all ears.

17

Me too, I know the possibilities but use it too seldom to remember the syntax, that is why I was a bit lazy.

I think you also need an if around the rclone calls to only update photo_folders.txt when rclone exits with 0 (successully)

and may benefit from adding --log-file=photo_log.txt --progress

18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.