Google Photos not listing photos by date correctly

Suspected Bug
1

What is the problem you are having with rclone?

I noticed that my sync from Google Photos wasn't listing all of my images (I'm new to rcloud). When I called
ls GPRemote:media/by-day/2022/2022-12-30

It only listed 2 photos and not the 3 from that day I could see in Google Photos. I went to the photos API and saw this information:

{
  "mediaItems": [
    {
      "id": "APl2FcQ8BjgM0s72-G1tZkIJrV9s3ivzdCuKXbYm2zR34HQ1nNL3GB9gZ_bpXYPDw-oWwkPyL-JXsq37Ow_jqauOHVD4I0tdzw",
      "productUrl": "https://photos.google.com/lr/photo/APl2FcQ8BjgM0s72-G1tZkIJrV9s3ivzdCuKXbYm2zR34HQ1nNL3GB9gZ_bpXYPDw-oWwkPyL-JXsq37Ow_jqauOHVD4I0tdzw",
      "baseUrl": "https://lh3.googleusercontent.com/lr/.....",
      "mimeType": "image/jpeg",
      "mediaMetadata": {
        "creationTime": "2022-12-30T15:03:38Z",
        "width": "2000",
        "height": "1125",
        "photo": {}
      },
      "filename": "IMG-20221230-WA0003.jpg"
    },
    {
      "id": "APl2FcTmi2pL9rNq0_6qaGaj8gTU0eyJRq0Gi1yV7IJh4dLaZwd0weNK5IDXfqgCmT5Vv8OSU4a7fWnQieUV0aKYARk0M2USbA",
      "productUrl": "https://photos.google.com/lr/photo/APl2FcTmi2pL9rNq0_6qaGaj8gTU0eyJRq0Gi1yV7IJh4dLaZwd0weNK5IDXfqgCmT5Vv8OSU4a7fWnQieUV0aKYARk0M2USbA",
      "baseUrl": "https://lh3.googleusercontent.com/lr/....",
      "mimeType": "image/jpeg",
      "mediaMetadata": {
        "creationTime": "2022-12-30T12:16:42Z",
        "width": "3072",
        "height": "4080",
        "photo": {
          "cameraMake": "Google",
          "cameraModel": "Pixel 7",
          "focalLength": 6.81,
          "apertureFNumber": 1.85,
          "isoEquivalent": 995,
          "exposureTime": "0.006289999s"
        }
      },
      "filename": "PXL_20221230_121642511.jpg"
    },
    {
      "id": "APl2FcQa-SPzlUmf4S-E6JRuY6ihdNtWCGvi2K3PRMFzmNAvemZYfo1nQst2UNgqarkFud6GVcRamZlCVlezJIw-jgpS697g9g",
      "productUrl": "https://photos.google.com/lr/photo/APl2FcQa-SPzlUmf4S-E6JRuY6ihdNtWCGvi2K3PRMFzmNAvemZYfo1nQst2UNgqarkFud6GVcRamZlCVlezJIw-jgpS697g9g",
      "baseUrl": "https://lh3.googleusercontent.com/lr/...",
      "mimeType": "image/jpeg",
      "mediaMetadata": {
        "creationTime": "2022-12-30T12:16:37Z",
        "width": "3072",
        "height": "4080",
        "photo": {
          "cameraMake": "Google",
          "cameraModel": "Pixel 7",
          "focalLength": 6.81,
          "apertureFNumber": 1.85,
          "isoEquivalent": 206,
          "exposureTime": "0.026598999s"
        }
      },
      "filename": "PXL_20221230_121637167.jpg"
    },
    {.....etc}
}

The file not showing up wasn't taken by my phone, but was a whatsapp image that is set to sync to google photos. In google photos web ui, the photo has the correct date. When I download it, I can see that the exif "datetaken" is blank (compared to the photos taken by my phone). If I go into the google photos web, and edit the date taken (changed the minute by +1), the photo then appears in the rcloud ls.

So it seems (wild guess here) rcloud is reading the exif data, and not using the creationTime in the google photos API as a fall back if exif data is empty. Which means that not all photos will be listed correctly by rcloud . I know an obvious fix is "sort your meta data out", but I think this feels like a bug, or maybe a feature request? (listing as a bug, as it certainly confused me)

Run the command 'rclone version' and share the full output of the command.

PS C:\Users\jdunmore\Downloads\rclone-v1.60.1-windows-amd64\rclone-v1.60.1-windows-amd64> .\rclone.exe version
rclone v1.62.0-beta.6672.98fa93f6d

  • os/version: Microsoft Windows 11 Enterprise 21H2 (64 bit)
  • os/kernel: 10.0.22000.1335 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: cmount
    PS C:\Users\jdunmore\Downloads\rclone-v1.60.1-windows-amd64\rclone-v1.60.1-windows-amd64>

Which cloud storage system are you using? (eg Google Drive)

Google Photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

ls GPRemote:media/by-day/2022/2022-12-30

The rclone config contents with secrets removed.

[GPRemote]
type = google photos
read_only = true
client_id = 
client_secret = 
token = {}

A log from the command with the -vv flag

2022/12/31 12:24:39 DEBUG : rclone: Version "v1.62.0-beta.6672.98fa93f6d" starting with parameters ["C:\\Users\\jdunmore\\Downloads\\rclone-v1.60.1-windows-amd64\\rclone-v1.60.1-windows-amd64\\rclone.exe" "-vv" "ls" "GPRemote:media/by-day/2022/2022-12-30"]
2022/12/31 12:24:39 DEBUG : Creating backend with remote "GPRemote:media/by-day/2022/2022-12-30"
2022/12/31 12:24:39 DEBUG : Using config file from "C:\\Users\\jdunmore\\AppData\\Roaming\\rclone\\rclone.conf"
2022/12/31 12:24:39 DEBUG : Google Photos path "media/by-day/2022/2022-12-30": List: dir=""
2022/12/31 12:24:40 DEBUG : GPRemote: Loaded invalid token from config file - ignoring
2022/12/31 12:24:40 DEBUG : Saving config "token" in section "GPRemote" of the config file
2022/12/31 12:24:40 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2022/12/31 12:24:40 DEBUG : GPRemote: Saved new token in config file
2022/12/31 12:24:40 DEBUG : Google Photos path "media/by-day/2022/2022-12-30": >List: err=<nil>
2022/12/31 12:24:40 DEBUG : IMG-20221230-WA0003.jpg: Size:
2022/12/31 12:24:40 DEBUG : IMG-20221230-WA0003.jpg: >Size:
2022/12/31 12:24:40 DEBUG : PXL_20221230_121642511.jpg: Size:
2022/12/31 12:24:40 DEBUG : PXL_20221230_121642511.jpg: >Size:
2022/12/31 12:24:40 DEBUG : PXL_20221230_121637167.jpg: Size:
2022/12/31 12:24:40 DEBUG : PXL_20221230_121637167.jpg: >Size:
2022/12/31 12:24:40 DEBUG : IMG-20221230-WA0003.jpg: Size:
2022/12/31 12:24:40 DEBUG : IMG-20221230-WA0003.jpg: >Size:
       -1 IMG-20221230-WA0003.jpg
2022/12/31 12:24:40 DEBUG : PXL_20221230_121637167.jpg: Size:
2022/12/31 12:24:40 DEBUG : PXL_20221230_121637167.jpg: >Size:
       -1 PXL_20221230_121637167.jpg
2022/12/31 12:24:40 DEBUG : PXL_20221230_121642511.jpg: Size:
2022/12/31 12:24:40 DEBUG : PXL_20221230_121642511.jpg: >Size:
       -1 PXL_20221230_121642511.jpg
2022/12/31 12:24:40 DEBUG : 4 go routines active
PS C:\Users\jdunmore\Downloads\rclone-v1.60.1-windows-amd64\rclone-v1.60.1-windows-amd64>
2

It's listed on the documentation so not really a wild guess :slight_smile:

3

Thanks - not quite sure how I didn't spot that.
So from reading that, this is a bug as it should use "or the upload date if that is not known." which isn't happening here.

4

The Google Photos API is pretty garbage imo.

You'd want to check the one photo that is problematic and see if the API is seeing the EXIF data. If so, it'll use that.

5

From that JSON dump in my post, it looks like the API has the createdTime metadata?

6

I believe you want to use some EXIF tool and check that on the photo in question before it was uploaded and see what it says. I'm not sure if it stores that EXIF timestamp somewhere else or how it works.

7

Right - just discovered, it's completely the google API at fault here. Media items list shows the items, but search by date, excludes them.

I'll raise a bug with google.

Thanks for your time looking into this.

8

Google have shown no interest in fixing any of the bugs in the Google Photos API. They clearly don't use it for the Google Photos web app so just don't care :frowning:

1 Like
9

Frustrating! I raised this bug Google Issue Tracker
But I can see they don't respond - and can see from stack that others have this issue too.

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.