Synching to Google Photo doesn't stop

Help and Support
1

What is the problem you are having with rclone?

The syncing process never stops and the total amount of data to be transferred keep increasing

What is your rclone version (output from rclone version)

rclone v1.53.2
- os/arch: linux/arm
- go version: go1.15.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspberry Pi OS Debian 10

Which cloud storage system are you using? (eg Google Drive)

Google Photo

The command you were trying to run (eg rclone copy /tmp remote:tmp)

nohup rclone -vv --bwlimit "08:00,500 23:00,off" sync /mnt/usb/Photo/ gphoto:upload &

After reading the documentation about :upload I also tried uploading the files to :album/Photo but the result it's the same

The rclone config contents with secrets removed.

[gphoto]
type = google photos
client_id = ****
client_secret = ****
token = ****

A log from the command with the -vv flag

2020/12/08 12:27:54 DEBUG : Creating backend with remote "/mnt/usb/Photo/"
2020/12/08 12:27:54 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2020/12/08 12:27:54 INFO  : Starting bandwidth limiter at 500kBytes/s
2020/12/08 12:27:54 DEBUG : Creating backend with remote "gphoto:album/Photo"
2020/12/08 12:27:54 DEBUG : Google Photos path "album/Photo": List: dir=""
2020/12/08 12:27:56 DEBUG : Google Photos path "album/Photo": >List: err=directory not found
2020/12/08 12:27:56 DEBUG : Google Photos path "album/Photo": Put: src=Varie/Video.mp4
2020/12/08 12:27:56 DEBUG : Varie/Video.mp4: Update: src=Varie/Video.mp4
2020/12/08 12:27:56 DEBUG : Google Photos path "album/Photo": Put: src=iPhone/Video/IMG_0527.MOV
2020/12/08 12:27:56 DEBUG : iPhone/Video/IMG_0527.MOV: Update: src=iPhone/Video/IMG_0527.MOV
2020/12/08 12:27:56 DEBUG : Google Photos path "album/Photo": Put: src=iPhone/Video/IMG_0864.m4v
2020/12/08 12:27:56 DEBUG : iPhone/Video/IMG_0864.m4v: Update: src=iPhone/Video/IMG_0864.m4v
2020/12/08 12:27:56 DEBUG : Google Photos path "album/Photo": Put: src=iPhone/Video/IMG_2655.MOV
2020/12/08 12:27:56 DEBUG : iPhone/Video/IMG_2655.MOV: Update: src=iPhone/Video/IMG_2655.MOV
2020/12/08 12:27:57 DEBUG : Google Photos path "album/Photo": Waiting for checks to finish
2020/12/08 12:27:57 DEBUG : Google Photos path "album/Photo": Waiting for transfers to finish
2020/12/08 12:28:18 DEBUG : Varie/Video.mp4: >Update: err=<nil>
2020/12/08 12:28:18 DEBUG : Google Photos path "album/Photo": >Put:
2020/12/08 12:28:18 DEBUG : Varie/Video.mp4: Size:
2020/12/08 12:28:18 DEBUG : Varie/Video.mp4: >Size:
2020/12/08 12:28:18 INFO  : Varie/Video.mp4: Copied (new)
2020/12/08 12:28:18 DEBUG : Google Photos path "album/Photo": Put: src=Pictures/done
2020/12/08 12:28:18 DEBUG : Pictures/done: Update: src=Pictures/done
2020/12/08 12:28:35 DEBUG : Pictures/done: >Update: err=upload failed: Failed: There was an error while trying to create this media item. (3)
2020/12/08 12:28:35 DEBUG : Google Photos path "album/Photo": >Put:
2020/12/08 12:28:35 ERROR : Pictures/done: Failed to copy: upload failed: Failed: There was an error while trying to create this media item. (3)
2020/12/08 12:28:35 DEBUG : Google Photos path "album/Photo": Put: src=iPhone/Video/IMG_3806.MOV
2020/12/08 12:28:35 DEBUG : iPhone/Video/IMG_3806.MOV: Update: src=iPhone/Video/IMG_3806.MOV
2020/12/08 12:28:54 NOTICE: Scheduled bandwidth change. Limit set to 500kBytes/s
2020/12/08 12:28:54 INFO  :
Transferred:       17.968M / 22.639 GBytes, 0%, 314.187 kBytes/s, ETA 20h58m18s
Errors:                 1 (retrying may help)
Transferred:            1 / 819, 0%
Elapsed time:       1m0.1s
Transferring:
 *                     iPhone/Video/IMG_0527.MOV:  6% /78.792M, 85.202k/s, 14m46s
 *                     iPhone/Video/IMG_0864.m4v:  9% /52.157M, 85.149k/s, 9m24s
 *                     iPhone/Video/IMG_2655.MOV:  0% /518.698M, 84.684k/s, 1h43m31s
 *                     iPhone/Video/IMG_3806.MOV:  5% /25.567M, 72.794k/s, 5m40s

it goes on uploading until it's basically done, and then

2020/12/09 22:09:53 INFO  :
Transferred:       23.354G / 23.362 GBytes, 100%, 201.850 kBytes/s, ETA 44s
Errors:                 6 (retrying may help)
Transferred:          813 / 814, 100%
Elapsed time:   33h42m0.1s
Transferring:
 * Pictures/Rullino iPhon…iPhone-fm/IMG_4275.MOV: 96% /245.067M, 308.932k/s, 29s

2020/12/09 22:10:27 DEBUG : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: >Update: err=<nil>
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": >Put:
2020/12/09 22:10:27 DEBUG : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: Size:
2020/12/09 22:10:27 DEBUG : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: >Size:
2020/12/09 22:10:27 INFO  : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: Copied (new)
2020/12/09 22:10:27 ERROR : Google Photos path "album/Photo": not deleting files as there were IO errors
2020/12/09 22:10:27 ERROR : Google Photos path "album/Photo": not deleting directories as there were IO errors
2020/12/09 22:10:27 ERROR : Attempt 1/3 failed with 6 errors and: couldn't upload file: Post "https://photoslibrary.googleapis.com/v1/uploads": http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=NO_ERROR, debug="max_age"
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": List: dir=""
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": >List: err=<nil>
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": List: dir="iPhone"
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/12/09 22:10:27 DEBUG : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: >Update: err=<nil>
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": >Put:
2020/12/09 22:10:27 DEBUG : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: Size:
2020/12/09 22:10:27 DEBUG : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: >Size:
2020/12/09 22:10:27 INFO  : Pictures/Rullino iPhone/Camera Roll from iPhone/IMG_4275.MOV: Copied (new)
2020/12/09 22:10:27 ERROR : Google Photos path "album/Photo": not deleting files as there were IO errors
2020/12/09 22:10:27 ERROR : Google Photos path "album/Photo": not deleting directories as there were IO errors
2020/12/09 22:10:27 ERROR : Attempt 1/3 failed with 6 errors and: couldn't upload file: Post "https://photoslibrary.googleapis.com/v1/uploads": http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=NO_ERROR, debug="max_age"
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": List: dir=""
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": Error reading media count: strconv.ParseInt: parsing "": invalid syntax
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": >List: err=<nil>
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": List: dir="iPhone"
2020/12/09 22:10:27 DEBUG : Google Photos path "album/Photo": Error reading media count: strconv.ParseInt: parsing "": invalid syntax

...
listing files
...

2020/12/09 22:10:50 DEBUG : Google Photos path "album/Photo": Waiting for checks to finish
2020/12/09 22:10:50 DEBUG : Google Photos path "album/Photo": Waiting for transfers to finish
2020/12/09 22:10:53 INFO  :
Transferred:       23.371G / 45.999 GBytes, 51%, 201.902 kBytes/s, ETA 1d8h38m34s
Errors:                 1 (retrying may help)
Checks:                 3 / 3, 100%
Transferred:          814 / 1630, 50%
Elapsed time:   33h43m0.1s
Transferring:
 *                     iPhone/Video/IMG_0527.MOV:  3% /78.792M, 102.780k/s, 12m40s
 *                     iPhone/Video/IMG_0864.m4v:  4% /52.157M, 106.628k/s, 7m56s
 *                     iPhone/Video/IMG_2655.MOV:  0% /518.698M, 94.692k/s, 1h33m4s
 *                     iPhone/Video/IMG_3806.MOV:  6% /25.567M, 84.946k/s, 4m47s

And it restarts, I tried waiting this second time but it goes on another time. At that point I stopped trying.

2

Rclone doesn't restart.

I'd imagine with nohup, you have more than 1 going on and you'd want make sure you stop any others.

3

I just have the one running, I use nohup rather than making a cronjob or whatever.

The process doesn’t restart per se, but it seems it starts over uploading the same data.

Am I missing something? Would rclone stop once the data has been copied?

4

Yes, once everything is copied and the process is done, it would exit.

5

So it counts the total files to be transferred, it's 819 and it's correct (the first log snippet). But then, when it should stop because all files have been uploaded, the number of files to be transferred goes up to 1630.

That's the same files being uploaded twice. Do you have any idea what could be the reason?

6

Yes, you'd have to include a full debug log and the information is there.

7

Rclone will retry the sync 3 times by default if there are errors (see the --retries flag).

Did the files get uploaded? Or did the upload fail?

Try listing the folder with rclone lsf - does it look correct or are there lots of photos with ID strings.

Try letting rclone run to completion - it will finish eventually.

8

I didn't realize about the 3 times retry, I'll wait for it to end and see what happens.

1 Like
9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.