Google photos directory characters and naming issues

Dev Discussions
1

What is the problem you are having with rclone?

rclone files uploaded have the directory names in them with odd slashes. When downloaded, they are jumbled. Is this expected behavir because i'm uploading photos? Iwould have thought it would have downloaded with the correctly. This could be google doing its 'magic' and breaking me though.

What is your rclone version (output from rclone version)

rclone version
rclone v1.51.0-311-ge91b5095-beta

  • os/arch: linux/amd64
  • go version: go1.14.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 64

Which cloud storage system are you using? (eg Google Drive)

drive and photos

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Multiple.  See log

The rclone config contents with secrets removed.

[robgs]
type = drive
...
[robgs-photos]
type = google photos
read_size = true

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

root@s163042:/data/bin# rclone lsf robgs:gp/rob/DCIM/Camera/IMG_20200511_103341.jpg
IMG_20200511_103341.jpg
root@s163042:/data/bin# rclone copy robgs:gp/rob/DCIM/Camera/IMG_20200511_103341.jpg robgs:gp-test/2020/04/
root@s163042:/data/bin# rclone lsl robgs:gp-test 
  3419147 2020-05-11 10:33:44.315000000 2020/04/IMG_20200511_103341.jpg
root@s163042:/data/bin# rclone -vv copy robgs:gp-test robgs-photos:upload
2020/05/14 11:12:20 DEBUG : rclone: Version "v1.51.0-311-ge91b5095-beta" starting with parameters ["rclone" "-vv" "copy" "robgs:gp-test" "robgs-photos:upload"]
2020/05/14 11:12:20 DEBUG : Using config file from "/root/.rclone.conf"
2020/05/14 11:12:20 DEBUG : Google Photos path "upload": List: dir=""
2020/05/14 11:12:20 DEBUG : Google Photos path "upload": >List: err=directory not found
2020/05/14 11:12:21 DEBUG : Google Photos path "upload": Waiting for checks to finish
2020/05/14 11:12:21 DEBUG : Google Photos path "upload": Waiting for transfers to finish
2020/05/14 11:12:22 DEBUG : Google Photos path "upload": Put: src=2020/04/IMG_20200511_103341.jpg
2020/05/14 11:12:22 DEBUG : 2020/04/IMG_20200511_103341.jpg: Update: src=2020/04/IMG_20200511_103341.jpg
2020/05/14 11:12:25 DEBUG : 2020/04/IMG_20200511_103341.jpg: >Update: err=<nil>
2020/05/14 11:12:25 DEBUG : Google Photos path "upload": >Put: 
2020/05/14 11:12:25 DEBUG : 2020/04/IMG_20200511_103341.jpg: Size: 
2020/05/14 11:12:25 DEBUG : Google Photos path "upload": List: dir="2020/04"
2020/05/14 11:12:25 DEBUG : Google Photos path "upload": >List: err=directory not found
2020/05/14 11:12:25 DEBUG : 2020/04/IMG_20200511_103341.jpg: Size: Failed to read metadata: object not found
2020/05/14 11:12:25 DEBUG : 2020/04/IMG_20200511_103341.jpg: >Size: 
2020/05/14 11:12:25 INFO  : 2020/04/IMG_20200511_103341.jpg: Copied (new)
2020/05/14 11:12:25 INFO  : 
Transferred:   	    3.261M / 3.261 MBytes, 100%, 987.589 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         3.3s

2020/05/14 11:12:25 DEBUG : 5 go routines active
root@s163042:/data/bin# rclone lsf robgs-photos:media/by-month/2020/2020-05/ -v
2020／04／IMG_20200511_103341.jpg
root@s163042:/data/bin# rclone copy "robgs-photos:media/by-month/2020/2020-05/2020／04／IMG_20200511_103341.jpg" .
root@s163042:/data/bin# ls -ltra *103341*
-rw-r--r-- 1 root root 1872166 May 11 10:33 '2020��'$'\217''04��'$'\217''IMG_20200511_103341.jpg'
2

I see this is being done on purpose.

        for i := range items {
                item := &result.MediaItems[i]
                remote := item.Filename
                remote = strings.Replace(remote, "/", "／", -1)
                err = fn(remote, item, false)
                if err != nil {
                        return err
                }
        }

What would be the impact to using another character like a dash or space or underscore?

3

Google allows / in file names (both in Photos and in Drive). In both cases we translate / to a unicode equivalent ／ which is a unicode oblique slash.

The impact of changing it would be that it would cause all the files with that character in to be re-downloaded if you were doing continuous syncing so I'm not keen on changing it.

If you set your console to be UTF-8 it will display properly. dpkg-reconfigure locales in debian/ubuntu is enough normally.

I see you are trying to sync directories to the upload path in google photos. Google photos is not good at directories... The uploader appears to be taking the whole path as the file name which maybe isn't too sensible! Maybe a better fix for you would be for the uploader to just use the leaf name as the name of the file?

4

I didn't realize this was also translated in drive.

Yes but I sure hope no one is using this to backup and download photos considering they are pretty low res! But yes it would break them. I was trying to fix this more from a listing and verification perspective that I've uploaded all the photos I have.

Perhaps. I considered that as well as having it actually use the directory path specified on the list/download (translate the slash to a hierarchy) with a flag like --unflatten. When I uploaded,I actually did it like this seeing that it kept the path.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.