--size-only not working properly with Google Photos

KrLx_roller · June 11, 2020, 7:24pm

What is the problem you are having with rclone?

Rclone is not overwriting files on remote when their size is different with the flag --size-only. Because the origin is Google Photos, I tried adding the flag --gphotos-read-size and the process takes ages longer because of additional transactions, but it has the exact same outcome.

It says it would transfer 6 files only when I know there are more! I first uploaded lower resolution version of the files as I uploaded them to Google Photos and triggered sync before they got processed. Now the videos are fully processed and I want to trigger sync again but rclone won't transfer more than the 6 new files I uploaded. It's not taking the older uploaded videos with higher resolution now.

What is your rclone version (output from `rclone version`)

rclone v1.51.0

os/arch: linux/amd64
go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian 10 x64

Which cloud storage system are you using? (eg Google Drive)

Transferring from Google Photos to OneDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

First I tried with:

rclone copy "gphotos:album/Family" "onedrive1:PMS/Family" -v --size-only

Then with:

rclone copy "gphotos:album/Family" "onedrive1:PMS/Family" -v --size-only --gphotos-read-size

The rclone config contents with secrets removed.

[gphotos]
type = google photos
token = {"access_token":"TOKEN HERE"}

[gdrive1]
type = drive
scope = drive
token = {"access_token":"TOKEN HERE"}
root_folder_id = ROOT FOLDER ID HERE

[onedrive1]
type = onedrive
token = {"access_token":"TOKEN HERE"}
drive_id = DEVICE ID HERE
drive_type = business
client_id = CLIENT ID HERE
client_secret = CLIENT SECRET HERE

A log from the command with the `-vv` flag

2020/06/11 12:16:56 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "copy" "gphotos:album/Family" "onedrive1:PMS/Family" "-vv" "--size-only" "--dry-run" "--log-file=/home/plex/log.log"]
2020/06/11 12:16:56 DEBUG : Using config file from "/home/plex/.config/rclone/rclone.conf"
2020/06/11 12:16:59 DEBUG : Google Photos path "album/Family": List: dir=""
2020/06/11 12:17:10 DEBUG : Google Photos path "album/Family": >List: err=<nil>
[...] MORE STUFF HERE
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: Size: 
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: Size: 
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: Size: 
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: Sizes identical
2020/06/11 12:17:10 DEBUG : grandma with the little boy.mp4: Unchanged skipping
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: Size: 
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: Size: 
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: Size: 
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: Sizes identical
2020/06/11 12:17:10 DEBUG : baby grabbing carla finger.mp4: Unchanged skipping
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: Sizes identical
2020/06/11 12:17:10 DEBUG : doggo running wild.mp4: Unchanged skipping
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: Size: 
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: Size: 
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: Size: 
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: >Size: 
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: Sizes identical
2020/06/11 12:17:10 DEBUG : julia cycling with max.mp4: Unchanged skipping
2020/06/11 12:17:10 INFO  : One drive root 'PMS/Family': Waiting for transfers to finish
2020/06/11 12:17:10 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              1225 / 1225, 100%
Transferred:            6 / 6, 100%
Elapsed time:         0.0s

2020/06/11 12:17:10 DEBUG : 7 go routines active
2020/06/11 12:17:10 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "copy" "gphotos:album/Family" "onedrive1:PMS/Family" "-vv" "--size-only" "--dry-run" "--log-file=/home/plex/log.log"]

ncw · June 11, 2020, 7:53pm

I think you will definitely need --gphotos-read-size to make this work.

Can you send a debug log with this?

KrLx_roller · June 11, 2020, 8:39pm

I already tried with --gphotos-read-size because I expected rclone to need the size to know if it should override or not -kinda obvious- but it took very long to finish as this album has over 1200 pictures and videos so I only tried once, with unsuccessful results, as already stated. I am going to try again with -vv and post the result here, but it's going to take quite long provided Google does not temporarily ban me for too many transactions

KrLx_roller · June 11, 2020, 8:51pm

So far, for the sake of speeding things up, as Google Photos is already slowing things down, rclone keeps posting this for every single video there is in the album:

2020/06/11 13:39:52 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "copy" "gphotos:album/Family" "onedrive1:PMS/Family" "-vv" "--size-only" "--gphotos-read-size" "--dry-run" "--log-file=/home/plex/log2.log"]
2020/06/11 13:39:52 DEBUG : Using config file from "/home/plex/.config/rclone/rclone.conf"
2020/06/11 13:39:52 DEBUG : onedrive1: Loaded invalid token from config file - ignoring
2020/06/11 13:39:53 DEBUG : Keeping previous permissions for config file: -rw-r-----
2020/06/11 13:39:53 DEBUG : onedrive1: Saved new token in config file
2020/06/11 13:39:56 DEBUG : Google Photos path "album/Family": List: dir=""
2020/06/11 13:39:56 DEBUG : gphotos: Loaded invalid token from config file - ignoring
2020/06/11 13:39:56 DEBUG : Keeping previous permissions for config file: -rw-r-----
2020/06/11 13:39:56 DEBUG : gphotos: Saved new token in config file
2020/06/11 13:40:06 DEBUG : Google Photos path "album/Family": >List: err=<nil>
[...]
2020/06/11 13:43:12 DEBUG : judith playing the cats.mp4: Size:
2020/06/11 13:43:13 DEBUG : judith playing the cats.mp4: >Size:
2020/06/11 13:43:13 DEBUG : dad with the little one!.mp4: Size:
2020/06/11 13:43:14 DEBUG : dad with the little one!.mp4: >Size:
2020/06/11 13:43:14 DEBUG : marathon (1).mp4: Size:
2020/06/11 13:43:15 DEBUG : marathon (1).mp4: >Size:
[...]

It feels like it's incapable of retrieving the size of the files... That'd explain why using rclone to mount the Google Photos album to use directly with Plex would not work... but that's a slightly different topic!

I'll post the end of the log once it's finished!

KrLx_roller · June 12, 2020, 6:59am

@ncw I had to go to sleep, I couldn't stay awake anymore

Here's the end of the log, unfortunately Google started restricting my access:

[...]
2020/06/11 14:42:49 DEBUG : ice cream time.mp4: Reading size failed:  (403 403 Forbidden)
2020/06/11 14:42:49 DEBUG : ice cream time.mp4: >Size: 
2020/06/11 14:42:49 DEBUG : ice cream time.mp4: Size: 
2020/06/11 14:42:50 DEBUG : pacer: Reducing sleep to 0s
2020/06/11 14:42:50 DEBUG : ice cream time.mp4: Reading size failed:  (403 403 Forbidden)
2020/06/11 14:42:50 DEBUG : ice cream time.mp4: >Size: 
2020/06/11 14:42:50 DEBUG : ice cream time.mp4: Sizes identical
2020/06/11 14:42:50 DEBUG : ice cream time.mp4: Unchanged skipping
2020/06/11 14:42:50 INFO  : One drive root 'PMS/Family': Waiting for transfers to finish
2020/06/11 14:42:50 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              1225 / 1225, 100%
Transferred:            6 / 6, 100%
Elapsed time:        50.9s

2020/06/11 14:42:50 DEBUG : 6 go routines active
2020/06/11 14:42:50 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "copy" "gphotos:album/Family" "onedrive1:PMS/Family" "-vv" "--size-only" "--gphotos-read-size" "--dry-run" "--log-file=/home/plex/log2.log"]

Let me know if you'd like me to test anything else!

KrLx_roller · June 12, 2020, 2:34pm

I just updated rclone to:

rclone v1.52.1
- os/arch: linux/amd64
- go version: go1.14.4

And same issue!

ncw · June 12, 2020, 3:10pm

So mostly the files checked as identical, however it did transfer 6! Can you find those 6 in the log and past all the log messages for one of them?

Rclone will just default to an unknown size if it can't read the size which is probably the reason for what you've seen if you are getting 403 messages.

I don't think reading the size is going to work well with this.

Google Photos is very limited alas, all we know is the name of the file we don't know the size unless we run a HEAD request against it which is probably quite intensive.

Could you get a list of the files which need to be uploaded? Maybe with the modification time (you can use rclone to help with this with the --min-age and --max-age flags (which can also take dates))

If you had a list then you could use rclone to delete them off Google Photos (with --files-from) and then the sync would work.

calisro · June 12, 2020, 3:26pm

I verified that it's actually returning the size here. Interestingly enough without the --gphotos-read-size flag set.

dell-rob:~/goproj/src/github.com/rclone/rclone/cmd/copy$ ls -l  /home/xx/test
2020/06/12 11:20:12 DEBUG : /: Attr: 
2020/06/12 11:20:12 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxr-x, err=<nil>
2020/06/12 11:20:12 DEBUG : /: ReadDirAll: 
2020/06/12 11:20:12 DEBUG : Google Photos path "media/by-day/2020/2020-05-19": List: dir=""
2020/06/12 11:20:13 DEBUG : Google Photos path "media/by-day/2020/2020-05-19": >List: err=<nil>
2020/06/12 11:20:13 DEBUG : 2020／05／IMG_20200519_101958.jpg: Size: 
2020/06/12 11:20:13 DEBUG : 2020／05／IMG_20200519_101958.jpg: >Size: 
2020/06/12 11:20:13 DEBUG : 2020／05／IMG_20200519_085358.jpg: Size: 
2020/06/12 11:20:13 DEBUG : 2020／05／IMG_20200519_085358.jpg: >Size: 
2020/06/12 11:20:13 DEBUG : 2020／05／IMG_20200519_085353.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : /: >ReadDirAll: item=3, err=<nil>
2020/06/12 11:20:15 DEBUG : /: Lookup: name="2020／05／IMG_20200519_085353.jpg"
2020/06/12 11:20:15 DEBUG : /: >Lookup: node=2020／05／IMG_20200519_085353.jpg, err=<nil>
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: Attr: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: ModTime: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: >ModTime: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085353.jpg: >Attr: a=valid=1s ino=0 size=3374483 mode=-rw-rw-r--, err=<nil>
2020/06/12 11:20:15 DEBUG : /: Lookup: name="2020／05／IMG_20200519_085358.jpg"
2020/06/12 11:20:15 DEBUG : /: >Lookup: node=2020／05／IMG_20200519_085358.jpg, err=<nil>
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: Attr: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: ModTime: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: >ModTime: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_085358.jpg: >Attr: a=valid=1s ino=0 size=3169218 mode=-rw-rw-r--, err=<nil>
2020/06/12 11:20:15 DEBUG : /: Lookup: name="2020／05／IMG_20200519_101958.jpg"
2020/06/12 11:20:15 DEBUG : /: >Lookup: node=2020／05／IMG_20200519_101958.jpg, err=<nil>
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: Attr: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: ModTime: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: >ModTime: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: >Size: 
2020/06/12 11:20:15 DEBUG : 2020／05／IMG_20200519_101958.jpg: >Attr: a=valid=1s ino=0 size=2587828 mode=-rw-rw-r--, err=<nil>
total 8918
-rw-rw-r-- 1 xx xx 3374483 May 19 08:53 2020／05／IMG_20200519_085353.jpg
-rw-rw-r-- 1 xx xx 3169218 May 19 08:53 2020／05／IMG_20200519_085358.jpg
-rw-rw-r-- 1 xx xx 2587828 May 19 10:19 2020／05／IMG_20200519_101958.jpg

ncw · June 12, 2020, 4:13pm

Did you just upload those images? Rclone might be doing some behind the scenes cheating

calisro · June 12, 2020, 4:26pm

No, those have been there for a longtime. rclone lsf, lsjson all report the sizes. The mount lists them. and if I dump headers/bodies its doing a HEAD and the response includes the sizes.

2020/06/12 12:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/12 12:24:39 DEBUG : 2020／01／IMG_20200101_152447.jpg: >Size: 
2020/06/12 12:24:39 DEBUG : 20200101_152441.jpg: Size: 
2020/06/12 12:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/12 12:24:39 DEBUG : HTTP REQUEST (req 0xc000956100)
2020/06/12 12:24:39 DEBUG : HEAD /lr/AFBm1_ZR8Z3PvSh7KSzTwRYg_I_PWyMVPssxb7h3q_QkucH5TZ1e5s_BQF6UZ9Ww9BV2-g2-QHmocfX-d9PuMfpOCjbWnyTEgH0jkwwNWeKRMa4ay7IuXWInIT2zr3XAVzFCpiimjvJXjxZM6loO0QggRbHNHeowkeaKm9DLucw7ovE2NYB5yeFF0cYvQyrYU6uX9oJ1YcIVvrhMVRHr2l2d-VOeCHfCIK4-gRzXeefUGsF-eXjxeuEVJYQ1Q4PA9chKwn5Gzdp88LHk-h-aMWqQBMx-dDdbGOm9nuOaS8kyu1tenFwIsH-E6fECVZcwxu_bUYBZjAIotVoGKv495dH2lyq4iMlFF4duvr3V30WlN6clsh1u6a28zKiF5txD54xNl6Fmj_RyrVnSO0tCLWX77c-5AqIy5zSEP9dAVlw18RPKEmCSO1-XUTQEB39iX1LmlCnbhVUUHVFP0sazPbWwCz-2DBh6OMUvxP0NAFRM8cspAXI2OEDKgSKyHEOz-e08cpahHxt2GdDACux_zQdUzu05elDyTGOxBoQjfczE0S5c_01MliIvqqMaqdxHNqSgEKuVrtXvSXV3A5azkGEdlAoeEwOUpHBD1MJ_Sw7SQhxFk6v_suhyTPzTE6cA7Yhz6u5k85lRqI3r_A0jv1oqWx2nTtU3riN_7NE6Z1kuT6kWxnGiBy5UtKAP_S-SSf26zp0Cy4HlM7mHZcwHeaPJzSzfixE0bIkhL4a7ofgDxm_tqr_oy_QLizHedRaTPC_bG3hYAwyCiQHY9PH_qoI2s7C53qx1ftZjq3WrIFVwGVI9EoZUdfTrBb_YYkQViytn=d HTTP/1.1
Host: lh3.googleusercontent.com
User-Agent: rclone/v1.52.0-001-g1cceadaf-beta
Authorization: XXXX

2020/06/12 12:24:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/12 12:24:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/12 12:24:39 DEBUG : HTTP RESPONSE (req 0xc000956100)
2020/06/12 12:24:39 DEBUG : HTTP/2.0 200 OK
Content-Length: 1506687
Access-Control-Expose-Headers: Content-Length
Alt-Svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private, max-age=86400, no-transform
Content-Disposition: attachment;filename="20200101_152441.jpg"
Content-Type: image/jpeg
Date: Fri, 12 Jun 2020 16:24:39 GMT
Etag: "v9535a"
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Server: fife
Vary: Origin
X-Content-Type-Options: nosniff
X-Xss-Protection: 0

KrLx_roller · June 12, 2020, 4:39pm

It did not transfer those 6 files because they were recently uploaded videos and I knew Google Photos would be processing them. Now that I know, I don't want to run into the same mistake again and trigger copy when videos are being processed

But still, this is what rclone logged:

[...]
2020/06/11 14:40:40 DEBUG : baby shower!.mp4: Reading size failed:  (403 403 Forbidden)
2020/06/11 14:40:40 DEBUG : baby shower!.mp4: >Size: 
2020/06/11 14:40:40 NOTICE: baby shower!.mp4: Not copying as --dry-run
[...]

The 403 forbidden error starts happening after a few dozens files have already been processed, so I guess that's Google Photos "banning" me for a little while.

It is impossible for me to retrieve the list of files, unfortunately... The only way I can think of, would be running lsd like calisro did and pray I get the filesizes from Google Photos. Then I do the same for OneDrive and do the comparison myself with a diff tool or automated with a Python script, for example. I will try lsd now see if that returns file size.

@calisro Could you try the copy command and see if it returns the file size for you?

EDIT: lsd is not returning file sizes either...

calisro · June 12, 2020, 5:06pm

@dell-rob:~/test$ rclone copy robgs-photos:media/by-day/2020/2020-01-01/VID_20191231_235947.mp4  robgd:  -vv
2020/06/12 13:05:48 DEBUG : rclone: Version "v1.52.0-001-g1cceadaf-beta" starting with parameters ["rclone" "copy" "robgs-photos:media/by-day/2020/2020-01-01/VID_20191231_235947.mp4" "robgd:" "-vv"]
2020/06/12 13:05:48 DEBUG : Using config file from "/home/xx/.rclone.conf"
2020/06/12 13:05:48 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": NewObject: remote="VID_20191231_235947.mp4"
2020/06/12 13:05:48 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": List: dir=""
2020/06/12 13:05:49 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": >List: err=<nil>
2020/06/12 13:05:49 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": >NewObject: 
2020/06/12 13:05:49 DEBUG : fs cache: renaming cache item "robgs-photos:media/by-day/2020/2020-01-01/VID_20191231_235947.mp4" to be canonical "robgs-photos:media/by-day/2020/2020-01-01"
2020/06/12 13:05:49 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": NewObject: remote="VID_20191231_235947.mp4"
2020/06/12 13:05:49 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": List: dir=""
2020/06/12 13:05:49 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": >List: err=<nil>
2020/06/12 13:05:49 DEBUG : Google Photos path "media/by-day/2020/2020-01-01": >NewObject: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: Size: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: >Size: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: Size: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: >Size: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: ModTime: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: >ModTime: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: Sizes identical
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: Unchanged skipping
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: Size: 
2020/06/12 13:05:50 DEBUG : VID_20191231_235947.mp4: >Size: 
2020/06/12 13:05:50 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2020/06/12 13:05:50 DEBUG : 7 go routines active

@dell-rob:~/test$ rclone lsjson robgd:VID_20191231_235947.mp4
[
{"Path":"VID_20191231_235947.mp4","Name":"VID_20191231_235947.mp4","Size":11867096,"MimeType":"video/mp4","ModTime":"2020-01-01T04:59:47.000Z","IsDir":false,"ID":"1wHJRvKpX7OZ60IFkrZVCu4JQDw0V3A7r"}
]

@dell-rob:~/test$ rclone lsjson robgs-photos:media/by-day/2020/2020-01-01/VID_20191231_235947.mp4
[
{"Path":"VID_20191231_235947.mp4","Name":"VID_20191231_235947.mp4","Size":11867096,"MimeType":"video/mp4","ModTime":"2020-01-01T04:59:47Z","IsDir":false,"ID":"AD5QJULiKJqjigfS8PKd5vuZptW9gGaAW8LZ3w7WkG33Ov44547EeYDc6YBoHiORQiHUBP-Nc4a_mjg4hugDCeCELBbsmkQaSA"}
]

This size and dates are correct...

What does it return? show us with a -vv debug?

KrLx_roller · June 12, 2020, 5:07pm

Some extra information. I added the flag --dump-bodies while having the --gphotos-read-size as well and for Google Photos requests I cannot see any file size anywhere:

2020/06/12 09:48:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/12 09:48:49 DEBUG : Google Photos path "album/Family": >List: err=<nil>
2020/06/12 09:48:49 DEBUG : cats playing with TV.mp4: Size: 
2020/06/12 09:48:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/12 09:48:49 DEBUG : HTTP REQUEST (req 0xc000ce0800)
2020/06/12 09:48:49 DEBUG : HEAD /lr/AFBm1_YwC9hwW4ue7D8WiuZoL4CsG3Dtn6pbV559UA7G1J3vul59mGm_kOihe_OSft6oB39BAQNIC5B2XyPpx4xYjlQvDFk1Q2B8JPCs9a8jHfTAY9lpQ1kmK6RLfoZnJOyNVF5ZDjmtpH2aULPGWCcnZlLyVMWvXnaddvzj27X1g5Qmf09zmVUwbhk1_t5LXIH2IHctLvsVPN3nn09idaEu0fVMc4cAdWT-hIqS9b9_i9mAnW8U24-8E-ZZWLnjmUNyPvIT8qvVc3JB597qxcj0vsOxYefmyLctvgDMS5EstpMhyMkhvw7qNzB6y3-4lTxrZG_B1LwjQrbonvgdT-Cs_xL8VlLwVHFcFTJ_jqwIOFamJXL2Y5Ac0Ju2s3QERwOTtnNdmqLix_i2jYQ_0bq_RQEP4NEMdIeBeA-_ikwtnzDXybWgR6ZXEah7e9q-NoyRubQVNLLBRcww5IRQ0gRMBxHr2-7WYPB7qYKjttZ6nrpXcjBRvt6l5BPWyDIVsF-m_58zqUv-T98JnDhwEEBz2Bsig8CCg1V11ZfJatYIJC4TYTNl1AstJlXLyvaO5UteBp0fZli7nirjz35DnNKXe6Nq-9wH8SXgDylFdrjraDrzkvTI6mSQGl03KtvGaaYvv3PXhXPOyOduW5G8QIPf5_sOEkBhoS8fsvxfYTKFV9EgLjlO5w7hZDhpYXt4wYd6MV0pIfGh63oyzidg9oy38=dv HTTP/1.1
Host: lh3.googleusercontent.com
User-Agent: rclone/v1.52.1
Authorization: XXXX

2020/06/12 09:48:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/12 09:48:49 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/12 09:48:49 DEBUG : HTTP RESPONSE (req 0xc000ce0800)
2020/06/12 09:48:49 DEBUG : HTTP/2.0 302 Found
Content-Length: 1194
Alt-Svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Date: Fri, 12 Jun 2020 16:48:49 GMT
Location: https://video-downloads.googleusercontent.com/AGQNM9LXfwRLlhiNsMAo4HtA6CeinsuG1mI0IQuzY0CLlIOsI2lgaMBDUCfmKh9KPIo2P8-G92TT8oMpT5lMXyfnDZ5aGw7MLcQvoj9oGTyvaI35bjmk1qpDY4X8fuUaLTJR7YAxSMTaEzvMATJbcknJCrWxeWF4CeQXlTHjtlLFGKKhiysohbzfIcq93l46qc8IZp82FdaKGQerNOQ6f-4BqdW5IUSC8c8HiywNDlkCQUtHl72MxA5o-ti7H8Cs_Gm51_zO4Ib5dmnTma7JQv5DIUurpCT8a8i48fuIwP9Xfzzm0zSL1oXU8FgoMOzRVbXy367fVKABz_N4Fs0_NExlvuXmwkYBRU6gzTuHW45l3wN6EoeuBhfZuY63Vjh2cDIMWfxaq2YY1ccv7vFZBW-GgB1lzrNGFaESqqY3p02MrqmDhQTbGHMBv2OC_ehA02ceeU6vusWNG5w0RNpURi_G9fpvTeXla9bD0j04slbX-85v8-9Rn3RnWeiRTV_KELPAFsQ6_9DOMNv3ZCRT3iJcd4eOcdmge_T_AsPP7R_SUbVd3BpAt-d_rtyZ2g4E-LR8RSuIFCXlNhUHDyS1LCaEIK6Y3WCcSHBpvvrYD3CWNfua2Roi4zpXe
Server: fife
Vary: Origin
X-Content-Type-Options: nosniff
X-Xss-Protection: 0

There was a list of items as well that reported the following for this file:

{
  "mediaItems": [
    {
      "id": "AD6dNpQz7yTA8AUL7int4mFBGl-0lJrXoABJ385UVuLbUoHh2mfGf2z4s-RQ",
      "productUrl": "https://photos.google.com/lr/album/AD6dNp7Ifi_oYXPkpCkC69nKEIvLlvutyLhwkjRbMCcgiCI/photo/AD6dNpQz7yTA8AUL7iGvfy1N-RhrErn8WgvWwH5iEW80LbyXoABJ385UVuLbUoHh2mfcG8Gf2z4s-RQ",
      "baseUrl": "https://lh3.googleusercontent.com/lr/AFBm1_YwC9hwW4ue7D8WiuZoL4CsG3Dtn6pbV559UA7G1J3vul59mGm_kOihe_OSft6oB39BAQNIC5B2XyPpx4xYjlQvDFk1Q2B8JPCs9a8jHfTAY9lpQ1kmK6RLfoZnJOyNVF5ZDjmtpH2aULPGWCc3Mg6Ta2MSC6P3-D2bG9dUoaLtvOGnaddvzj27X1g5Qmf09zmVUwbhk1_t5LXIH2IHctLvsVPN3nn09idaEu0fVMc4cAdWT-hIqS9b9_i9mAnW8U24-8E-ZZWLnjmUNyPvIT8qvVc3JB597qxcj0vsOxYefmyLctvg7qNzB6y3-4lTxrZG_B1LwjQrbonvgdT-Cs_D0foqlZNgSarmCVbwcexL8VlLwVHFcFTJ_jqwIOFamJXL2Y5Ac0Ju2s3QERwOTtnNdmqLix_i2jYQ_0bq_RQEP4NEMdIeBeA-_ikwtnzDXybWgR6ZXEah7e9q-NoRcww5IRQ0gRMBxHr2-7WYPB80EVXc0IXBDfphUCmzNtW7qYKjttZ6nrpXcjBRvt6l5BPWyDIVsF-m_58zqUv-T98JnDhwEEBz2Bsig8CCg1V11ZfJatYIJC4TYTNl1AstJlXLyva7nirjz35DnNKXe6Nq-9wH8SXgDylFdrjraDrzkvTI6mSQGl03KtvGaaYvv3PXhXPOyOduW5G8QIPf5_sOEkBhoS8KLVSu401hMxOZU0hWDJg73CCQai7fsvxfYTKFV9EgLjlO5w7hZDhpYXt4w",
      "mimeType": "video/mp4",
      "mediaMetadata": {
        "creationTime": "2020-04-10T04:01:44Z",
        "width": "1920",
        "height": "1080",
        "video": {
          "fps": 29.970029970029969,
          "status": "READY"
        }
      },
      "filename": "cats playing with TV.mp4"
    },
[...]

While for OneDrive the file size seems to be there:

[...]
"id":"014LDL7R24Z74ZAIF2JAEK",
"lastModifiedDateTime":"2020-04-10T04:01:44Z","name":"cats playing with TV.mp4",
"webUrl":"https://jia666-my.sharepoint.com/personal/user/Documents/PMS/Family/cats playing with TV.mp4",
"cTag":"\"c:{A1E70-4179-BCFF0BA4808A},2\"",
"size":82187132,
"createdBy":{"application":{"id":"b156656-4092-8539-0eec376afd59","displayName":"rclone"},
[...]

calisro · June 12, 2020, 5:11pm

That is the repoted size. Seems small though.

instead of using album, try with the media directory....

KrLx_roller · June 12, 2020, 5:13pm

The correct size is what OneDrive is displaying: "size":82187132...

Also, if that were the size rclone thinks Google Photos is reporting, then I guess it would override the file while it does not...

calisro · June 12, 2020, 5:21pm

xx@dell-rob:~/test$ rclone lsjson robgd-photos:media/all --gphotos-read-size
[
{"Path":"VID_20191231_235947.mp4","Name":"VID_20191231_235947.mp4","Size":1228431,"MimeType":"video/mp4","ModTime":"2020-01-02T23:03:10Z","IsDir":false,"ID":"AKps9KE1EnUvpsJobhk7JymDaOHS0vOiMV6fOtnJgXPAl8TtPUp-TTHkWPFygx5jPfyyGTPp29PEkhXB7BWQwfoy3JhQQoUYTw"}
]
xx@dell-rob:~/test$ rclone lsjson robgd-photos:album/test/ --gphotos-read-size
[
{"Path":"2019","Name":"2019","Size":-1,"MimeType":"inode/directory","ModTime":"2020-06-12T13:20:50-04:00","IsDir":true},
{"Path":"VID_20191231_235947.mp4","Name":"VID_20191231_235947.mp4","Size":1228431,"MimeType":"video/mp4","ModTime":"2020-01-02T23:03:10Z","IsDir":false,"ID":"AKps9KE1EnUvpsJobhk7JymDaOHS0vOiMV6fOtnJgXPAl8TtPUp-TTHkWPFygx5jPfyyGTPp29PEkhXB7BWQwfoy3JhQQoUYTw"}
]
xx@dell-rob:~/test$ rclone lsjson robgd-photos:album/test/ 
[
{"Path":"2019","Name":"2019","Size":-1,"MimeType":"inode/directory","ModTime":"2020-06-12T13:20:56-04:00","IsDir":true},
{"Path":"VID_20191231_235947.mp4","Name":"VID_20191231_235947.mp4","Size":-1,"MimeType":"video/mp4","ModTime":"2020-01-02T23:03:10Z","IsDir":false,"ID":"AKps9KE1EnUvpsJobhk7JymDaOHS0vOiMV6fOtnJgXPAl8TtPUp-TTHkWPFygx5jPfyyGTPp29PEkhXB7BWQwfoy3JhQQoUYTw"}
]

I'm not sure... I'm not having the same issue. When you run those, you don't get sizes? can you run those exact ones for one of your vids? You can't use media/all obviously but you shouldbe able to narrow it down to a date.

KrLx_roller · June 12, 2020, 5:24pm

This is what I got when running ls or lsd in the Family album:

plex@debian:~$ rclone lsd "gphotos:album/Family" -vv --gphotos-read-size
2020/06/12 10:19:49 DEBUG : rclone: Version "v1.52.1" starting with parameters ["rclone" "lsd" "gphotos:album/Family" "-vv" "--gphotos-read-size"]
2020/06/12 10:19:49 DEBUG : Using config file from "/home/plex/.config/rclone/rclone.conf"
2020/06/12 10:19:49 DEBUG : Google Photos path "album/Family": List: dir=""
2020/06/12 10:20:03 DEBUG : Google Photos path "album/Family": >List: err=<nil>
2020/06/12 10:20:03 DEBUG : cats playing with the TV.mp4: Size:
2020/06/12 10:20:03 DEBUG : cats playing with the TV.mp4: >Size:
2020/06/12 10:20:03 DEBUG : mum and dad.mp4: Size:
2020/06/12 10:20:04 DEBUG : mum and dad.mp4: >Size:
2020/06/12 10:20:04 DEBUG : kids in the garden.mp4: Size:
2020/06/12 10:20:04 DEBUG : kids in the garden.mp4: >Size:
2020/06/12 10:20:04 DEBUG : kelly and james over.mp4: Size:
2020/06/12 10:20:05 DEBUG : kelly and james over.mp4: >Size:
[...]

I'm going to try the exact same command you are using, see if there's any noticeable difference...

calisro · June 12, 2020, 5:25pm

lsd is to list directories though. THis is a ls

xx@dell-rob:~/test$ rclone ls robgd-photos:album/test/ 
       -1 VID_20191231_235947.mp4
xx@dell-rob:~/test$ rclone ls robgd-photos:album/test/  --gphotos-read-size
  1228431 VID_20191231_235947.mp4

KrLx_roller · June 12, 2020, 5:25pm

I tried both ls and lsd to make sure and both report the exact same log you can see above your last post

calisro · June 12, 2020, 5:26pm

lsd wont report anything. It just lists directories. Can you share the actual output of ls or lsjson? You're looking at the log. My log shows the same. I'm asking for the output.