Rclone errors accessing files

What is the problem you are having with rclone?

I have Rclone, a google drive, plex, and I'm using winwsp to mount the gcrypt to X:
Rclone intermittently can not access specific files. Which files can or can not be access is hit or miss. I have a ton (just a little over 1000) of movies in "grypt2:\Movies". I access them through plex normally, which works well for the most part. However, plex gives me an error sometimes that it can't play back the file. I tried to figure out the issue, and it really isn't a plex issue, as I cant play the files in VLC or any other program, even with gcrypt2 mounted and accessing it in file explorer. But then another day some of those files are accessible and some others aren't. Right now for example, I can not play "The Invisible Man" but in that same folder, using all the same settings other movies will open and play correctly for example "Fast 5".

What is your rclone version (output from rclone version)

1.53.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 x64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I tried to copy a file from the encrypted drive to another to see if it had anything to do with the encryption keys or something using

rclone copy "gcrypt2:\Movies\The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv" F: -P -v

I got back the following error in rclone

2020-11-21 20:22:31 ERROR : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:22:31 ERROR : Attempt 1/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:22:31 ERROR : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:22:31 ERROR : Attempt 2/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:22:32 ERROR : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:22:32 ERROR : Attempt 3/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

I used the same command with Fast 5 and it copied flawlessly taking around 3.5 minutes.

rclone copy "gcrypt2:\Movies\The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv" F: -P -v
2020/11/21 20:39:54 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "gcrypt2:\\Movies\\The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv" "F:" "-P" "-v" "-vv"]
2020/11/21 20:39:54 DEBUG : Using config file from "C:\\Users\\Jeff\\.config\\rclone\\rclone.conf"
2020/11/21 20:39:54 DEBUG : Creating backend with remote "gcrypt2:\\Movies\\The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv"
2020/11/21 20:39:54 DEBUG : Creating backend with remote "gdrive:gdrive/6nu5akjq0k1v3udplr4igmammo/r6ik3mv2eql49jekcmi77hbuc9g61g8kckcb0cdsolu7inin9r77hnf46tquimd48neikmqudedtr8v4to5gigjrocguubklqap24v8"
2020/11/21 20:39:56 DEBUG : fs cache: adding new entry for parent of "gdrive:gdrive/6nu5akjq0k1v3udplr4igmammo/r6ik3mv2eql49jekcmi77hbuc9g61g8kckcb0cdsolu7inin9r77hnf46tquimd48neikmqudedtr8v4to5gigjrocguubklqap24v8", "gdrive:gdrive/6nu5akjq0k1v3udplr4igmammo"
2020/11/21 20:39:56 DEBUG : fs cache: adding new entry for parent of "gcrypt2:\\Movies\\The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv", "gcrypt2:/Movies/The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv"
2020/11/21 20:39:56 DEBUG : Creating backend with remote "F:"
2020/11/21 20:39:56 DEBUG : fs cache: renaming cache item "F:" to be canonical "//?/F:/Users/Jeff/Downloads/rclone"
2020-11-21 20:39:56 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: Need to transfer - File not found at Destination
2020-11-21 20:39:56 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2020-11-21 20:39:56 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: Starting multi-thread copy with 4 parts of size 2.460G
2020-11-21 20:39:56 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 1/4 (0-2641100800) size 2.460G starting
2020-11-21 20:39:56 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 2/4 (2641100800-5282201600) size 2.460G starting
2020-11-21 20:39:56 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 3/4 (5282201600-7923302400) size 2.460G starting
2020-11-21 20:39:56 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 4/4 (7923302400-10564302651) size 2.460G starting
2020-11-21 20:41:16 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 4/4 (7923302400-10564302651) size 2.460G finished
2020-11-21 20:42:21 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 3/4 (5282201600-7923302400) size 2.460G finished
2020-11-21 20:43:09 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 2/4 (2641100800-5282201600) size 2.460G finished
2020-11-21 20:43:28 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: multi-thread copy: stream 1/4 (0-2641100800) size 2.460G finished
2020-11-21 20:43:32 DEBUG : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: Finished multi-thread copy with 4 parts of size 2.460G
2020-11-21 20:43:32 INFO  : The Fast and The Furious 5 - Fast Five (2011) 1080p.mkv: Multi-thread Copied (new)
Transferred:        9.839G / 9.839 GBytes, 100%, 46.748 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      3m37.8s
2020/11/21 20:43:32 INFO  :
Transferred:        9.839G / 9.839 GBytes, 100%, 46.748 MBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      3m37.8s

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = 
client_secret = 
scope = drive
use_trash = false
chunk_size = 16M
token = 
root_folder_id = 

[gcrypt2]
type = crypt
remote = gdrive:gdrive
filename_encryption = standard
directory_name_encryption = true
password = 
password2 = 


A log from the command with the -vv flag

2020/11/21 20:54:09 DEBUG : rclone: Version "v1.53.1" starting with parameters ["rclone" "copy" "gcrypt2:\\Movies\\The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv" "F:" "-P" "-v" "-vv"]
2020/11/21 20:54:09 DEBUG : Using config file from "C:\\Users\\Jeff\\.config\\rclone\\rclone.conf"
2020/11/21 20:54:09 DEBUG : Creating backend with remote "gcrypt2:\\Movies\\The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv"
2020/11/21 20:54:09 DEBUG : Creating backend with remote "gdrive:gdrive/6nu5akjq0k1v3udplr4igmammo/q9mqtjs2egoenitkv1460smqnmustj3d3jjq9rbvpm1bhiijhsv6tdhdihn1obra1sr4esmbhb9806c7cgvi475g3qsuc18hlgq2ik8"
2020/11/21 20:54:11 DEBUG : fs cache: adding new entry for parent of "gdrive:gdrive/6nu5akjq0k1v3udplr4igmammo/q9mqtjs2egoenitkv1460smqnmustj3d3jjq9rbvpm1bhiijhsv6tdhdihn1obra1sr4esmbhb9806c7cgvi475g3qsuc18hlgq2ik8", "gdrive:gdrive/6nu5akjq0k1v3udplr4igmammo"
2020/11/21 20:54:11 DEBUG : fs cache: adding new entry for parent of "gcrypt2:\\Movies\\The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv", "gcrypt2:/Movies/The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv"
2020/11/21 20:54:11 DEBUG : Creating backend with remote "F:"
2020/11/21 20:54:11 DEBUG : fs cache: renaming cache item "F:" to be canonical "//?/F:/Users/Jeff/Downloads/rclone"
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Sizes differ (src 3436566718 vs dst 0)
2020-11-21 20:54:11 INFO  : Writing sparse files: use --local-no-sparse or --multi-thread-streams 0 to disable
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Starting multi-thread copy with 4 parts of size 819.375M
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 4/4 (2577530880-3436566718) size 819.240M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 2/4 (859176960-1718353920) size 819.375M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 1/4 (0-859176960) size 819.375M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 3/4 (1718353920-2577530880) size 819.375M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 4/4 failed: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 1/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 3/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 2/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:11 ERROR : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:11 ERROR : Attempt 1/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Sizes differ (src 3436566718 vs dst 0)
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Starting multi-thread copy with 4 parts of size 819.375M
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 4/4 (2577530880-3436566718) size 819.240M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 2/4 (859176960-1718353920) size 819.375M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 1/4 (0-859176960) size 819.375M starting
2020-11-21 20:54:11 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 3/4 (1718353920-2577530880) size 819.375M starting
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 4/4 failed: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 2/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 3/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 1/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:12 ERROR : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:12 ERROR : Attempt 2/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Sizes differ (src 3436566718 vs dst 0)
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Starting multi-thread copy with 4 parts of size 819.375M
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 4/4 (2577530880-3436566718) size 819.240M starting
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 1/4 (0-859176960) size 819.375M starting
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 2/4 (859176960-1718353920) size 819.375M starting
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 3/4 (1718353920-2577530880) size 819.375M starting
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 4/4 failed: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 2/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 1/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:12 DEBUG : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: multi-thread copy: stream 3/4 failed: multpart copy: failed to open source: open file failed: Get "https://www.googleapis.com/drive/v3/files/1ouvqCTfqSm7ec4JT782-EyD_vH7BGiT6?alt=media": context canceled
2020-11-21 20:54:12 ERROR : The Invisible Man (2020) (1080p BluRay x265 10bit Tigole).mkv: Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
2020-11-21 20:54:12 ERROR : Attempt 3/3 failed with 1 errors and: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.4s
2020/11/21 20:54:12 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.4s

2020/11/21 20:54:12 DEBUG : 4 go routines active
2020/11/21 20:54:12 Failed to copy: multpart copy: failed to open source: open file failed: googleapi: Error 403: The download quota for this file has been exceeded., downloadQuotaExceeded

I will update rclone as soon as I am done posting this to 1.53.3 but I doubt this will fix the issue as I was on a previous version and updated to 1.53.1 and the problem persisted.

Updated to 1.53.3 - still same issue

I just created a new API and updated it in rclone to see if that would resolve the issue, and the same issue continues to persist.

Looks like you hit your quota so retry again after 24 hours.

thanks for your reply. I was actually hoping that you would chime in. I'm currently studying your github. The question remains: WHY did I hit the quota? The file was added a week or 2 ago, and hasn't been played at all. I'm thinking there is a setting in Plex that I need to adjust after reading more on your github.

Google provides zero way to answer that question unfortunately. You can always email them for support and ask.

Thanks for the reassurance and the work you put in. I'll continue to work on it while comparing to your own setup, and report my results.

@Animosity022 - so I noticed in your github you put to disable "Perform extensive media analysis during maintenance" under scheduled tasks. Mine was on, do you believe this alone could be what was causing the issue? The issue definitely is much more present with newer files that are added if that helps.

Are any of the other items in scheduled tasks an issue? Here is what I have:

	Scheduled Tasks
		Backup database every three days: yes
		Backup Directory: X:\Plex\Database Backup  (Where X: is my gdrive mounted to windows using winfsp)
		Optimize database every week: yes
		Remove old bundles every week: yes
		Remove old cache files every week: yes
		Refresh local metadata every three days: yes
		Upgrade media analysis during maintenane: yes
		Refresh music library metadata periodically: yes
		Perform extensive media analysis during maintenance: yes

extensive media analysis is going to generate more API hits and data transfer and defintiely could cause an API ban.

That's downloading entire files as it's only used if you have bandwidth restriction on, which I do not use. That would definitely cause some download usage.

Thanks, I just turned that off after seeing Animosity's github. Do you see any other settings there that could be an issue? Refresh local metadata every 3 days for example?

Thanks, I've turned it off. I guess we will see in 24 hours if I am still having some issues.

Not trying to be picky, but you don't get API bans as you have plenty of API on a daily basis.

You hit download and/or upload quotas daily.

Just trying to get this straight in my head. So a google account has download/upload limits of 750GB/day - if I exceed this I can not access any files on google drive at all. But an API also has download limits? And each individual file has limits, which is why some files wont play and others are free to work as usual?

API limits are easy to see and in the Google Admin console and have values in there. I'm a light API user by all means:

There are download/upload quotas and there is little to documentation other than the 750GB upload one:

As for download, good luck as I don't think there is much documented as there seems to be a total size, per file and that seems to change if you have a edu / team drive / etc. I've never hit the limit, but I'm a light user as I only hit maybe 4-5TB download in a month.

Just trying to help and there was a time where API bans did happen which is why we have cache and vfs setups now in rclone to avoid them.

That's actually not correct. They had download quota issues as download uses to count as full downloads, which triggered the download quota for the day.

I have a screenshot above of the API limits and you don't get banned for exceeding them as rclone is just asked to slow down.

I was only trying to help. Obviously I don't know what I'm talking about so I'll refrain from doing so in the future.

I'm sorry.

Hmmmm...... my numbers seem to be similar to yours, actually even slightly lower:


Even per 100 seconds seems to be less than half what the limit is, unless I'm reading this wrong.

Yep, that's why API hits vs Download/Upload is a bit confusing.

You aren't hitting any API issues, which is easy to see and it's documented nice.

It's the daily download / upload quotas per day that you don't have any visibility into unless you monitor/watch it on your side.