Copy new album to Google Photos is too slow

Filippo_Bottega · February 8, 2021, 11:18pm

I need to create new albums to Google Photos but I want to call a new instance of rclone for each album.
Before loading files to Google Photos, rclone seams to list all the files in cloud. Is it possible to skip the listing of remote files?
I'm using these commands:

rclone mkdir "google_photos:album/AlbumName"
rclone copy AlbumPath google_photos:album/AlbumName --verbose --include *.jpg --include *.mp4 --order-by name,ascending --transfers 1

Thank you!

asdffdsa · February 8, 2021, 11:33pm

hello and welcome to the forum,

rclone has to compare the source and dest, to know which files need to be copied.

there are some flags that may help, depending on your use-case
https://rclone.org/docs/#no-check-dest
https://rclone.org/docs/#no-traverse

Filippo_Bottega · February 9, 2021, 9:18am

Hi,
thank you for your answer but nothing changed.
This is my log. In bold the slow events (20 and 60 seconds). Do you know if the delay is due to the Google API calls?

Thank you!

Loading ".\2018-04-23" in "google_photos:album/2018-04-23"
2021/02/09 10:02:26 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone.exe" "mkdir" "google_photos:album/2018-04-23" "--no-check-dest" "--no-traverse" "-vv"]
2021/02/09 10:02:26 DEBUG : Using config file from "C:\Users\\.config\rclone\rclone.conf"
2021/02/09 10:02:26 DEBUG : Google Photos path "album/2018-04-23": Making directory
2021/02/09 10:02:26 DEBUG : Google Photos path "album/2018-04-23": Mkdir: dir=""
2021/02/09 10:02:26 DEBUG : google_photos: Loaded invalid token from config file - ignoring
2021/02/09 10:02:26 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2021/02/09 10:02:26 DEBUG : google_photos: Saved new token in config file
2021/02/09 10:03:46 DEBUG : Google Photos path "album/2018-04-23": >Mkdir: err=
2021/02/09 10:03:46 DEBUG : 3 go routines active
2021/02/09 10:03:46 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rclone.exe" "copy" ".\2018-04-23" "google_photos:album/2018-04-23" "-vv" "--include" ".jpg" "--include" ".mp4" "--order-by" "name,ascending" "--transfers" "1" "--no-check-dest" "--no-traverse"]
2021/02/09 10:03:46 DEBUG : Using config file from "C:\Users\\.config\rclone\rclone.conf"
2021/02/09 10:03:46 DEBUG : fs cache: renaming cache item ".\2018-04-23" to be canonical "//?/./2018-04-23"
2021/02/09 10:03:46 DEBUG : Google Photos path "album/2018-04-23": Waiting for checks to finish
2021/02/09 10:03:46 DEBUG : Google Photos path "album/2018-04-23": Waiting for transfers to finish
2021/02/09 10:03:46 DEBUG : Google Photos path "album/2018-04-23": Put: src=IMG_20180423_110118.jpg
2021/02/09 10:03:46 DEBUG : IMG_20180423_110118.jpg: Update: src=IMG_20180423_110118.jpg
2021/02/09 10:04:46 INFO :
Transferred: 0 / 1.010 GBytes, 0%, 0 Bytes/s, ETA -
Transferred: 0 / 214, 0%
Elapsed time: 59.8s
Transferring:
* IMG_20180423_110118.jpg: 0% /6.038M, 0/s, -
2021/02/09 10:05:02 DEBUG : IMG_20180423_110118.jpg: >Update: err=
2021/02/09 10:05:02 DEBUG : Google Photos path "album/2018-04-23": >Put:
2021/02/09 10:05:02 DEBUG : IMG_20180423_110118.jpg: Size:
2021/02/09 10:05:02 DEBUG : IMG_20180423_110118.jpg: >Size:
2021/02/09 10:05:02 INFO : IMG_20180423_110118.jpg: Copied (new)
2021/02/09 10:05:02 DEBUG : Google Photos path "album/2018-04-23": Put: src=IMG_20180423_110127.jpg
2021/02/09 10:05:02 DEBUG : IMG_20180423_110127.jpg: Update: src=IMG_20180423_110127.jpg
2021/02/09 10:05:09 DEBUG : IMG_20180423_110127.jpg: >Update: err=
2021/02/09 10:05:09 DEBUG : Google Photos path "album/2018-04-23": >Put:
2021/02/09 10:05:09 DEBUG : IMG_20180423_110127.jpg: Size:
2021/02/09 10:05:09 DEBUG : IMG_20180423_110127.jpg: >Size:
2021/02/09 10:05:09 INFO : IMG_20180423_110127.jpg: Copied (new)
2021/02/09 10:05:09 DEBUG : Google Photos path "album/2018-04-23": Put: src=IMG_20180423_110150.jpg
2021/02/09 10:05:09 DEBUG : IMG_20180423_110150.jpg: Update: src=IMG_20180423_110150.jpg

ncw · February 9, 2021, 10:49am

Rclone is listing all the albums. Do you have a huge number of albums?

The only way in the google photos API to look up an album by name is to look them all up as far as I know.

Note that you don't need to do the rclone mkdir - the rclone copy will do it for you.

Filippo_Bottega · February 9, 2021, 11:06am

I have a huge number of albums.
I know that the album is new, why look up the album? I just need to create a new one without checking if it already exists.

Thank you for the tip!

ncw · February 9, 2021, 11:22am

That explains the delay.

Rclone needs to see if the album exists already to avoid creating it again.

What you can do is use rclone rcd to set up an persistent rclone then use rclone rc sync/copy to upload the albums.

If you are doing a whole series of albums this will cache the album listings for 5 minutes.

Filippo_Bottega · February 10, 2021, 11:18am

Thank you for the tip, I did'n know the rcd mode.

Is it possible to change the expiration time to one hour? My workflow may require some days to finish.

ncw · February 10, 2021, 3:18pm

Not currently, but this is a requested feature which will go into 1.55 I think

Filippo_Bottega · February 10, 2021, 3:44pm

If it were possible it would be very useful for me to customise the cache expiration time.
Thank you!

ncw · February 10, 2021, 4:12pm

The default is here if you want to patch the code

github.com

rclone/rclone/blob/bf8542c67004428ad25b8eab1fa7e34608ccb908/lib/cache/cache.go#L25


	expireRunning  bool
	expireDuration time.Duration // expire the cache entry when it is older than this
	expireInterval time.Duration // interval to run the cache expire
}
// New creates a new cache with the default expire duration and interval
func New() *Cache {
	return &Cache{
		cache:          map[string]*cacheEntry{},
		expireRunning:  false,
		expireDuration: 300 * time.Second,
		expireInterval: 60 * time.Second,
	}
}
// cacheEntry is stored in the cache
type cacheEntry struct {
	value    interface{} // cached item
	err      error       // creation error
	key      string      // key
	lastUsed time.Time   // time used for expiry

It really needs to be on a flag though!

system · April 12, 2021, 12:12pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.