Still unable to move files on GDrive

Help and Support
1

I posted this a few months back, and I am still seeing this problem with the latest version. It does not always occur, but I cannot figure out what conditions trigger it. It does not appear to be peculiarities with file or directory names or sizes. When the issue does occur, it occurs consistently for the same files.

I am running rclone using the backup-dir option. I am getting errors that files can't be moved on GDrive, because they're not the same remote type, even though the files should be moving within the same remote. As a result, the files are being uploaded again (presumably after being downloaded). Any ideas why I'm seeing this behavior?

I've included an excerpt of my log file below, and my command line is:
rclone -vv -P --drive-impersonate
foo@bar.com --config c:\rclone\rclone.conf
--log-file c:\temp\20190907-212931-rclone.log sync
c:\temp wsGDcrypt-luminor:/current/temp --fast-list
--create-empty-src-dirs --backup-dir
wsGDcrypt-luminor:/backups/backup-20190907-212931
--exclude 20190907-212931-rclone.log --exclude-from
c:\rclone\excludes.txt

2019/09/07 21:30:37 DEBUG : Encrypted drive 'wsGDcrypt-luminor:/current/temp': copied 21 directories
2019/09/07 21:30:37 INFO : Waiting for deletions to finish
2019/09/07 21:30:37 DEBUG : Cache remote wsGDcache-luminor:p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410: new object 'oefa3ocd4oeupvuq7thqs0msm4/ghkd8j8511h8mctf9o7eup8nj8k1ro38rfa24d4cd7fiqvm4apa0'
2019/09/07 21:30:37 DEBUG : Cache remote wsGDcache-luminor:p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410: new object 'oefa3ocd4oeupvuq7thqs0msm4/m65bt38b4psmjh1e4o4mnrqg8upkd4m4ao4qs18nmndr1s7pjmgg'
2019/09/07 21:30:37 DEBUG : oefa3ocd4oeupvuq7thqs0msm4/ghkd8j8511h8mctf9o7eup8nj8k1ro38rfa24d4cd7fiqvm4apa0: find: error: couldn't open parent bucket for p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410/oefa3ocd4oeupvuq7thqs0msm4
2019/09/07 21:30:37 DEBUG : Cache remote wsGDcache-luminor:p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410: new object 'oefa3ocd4oeupvuq7thqs0msm4/79iqb07amhlissidlb6trdl8g9lqmeafeafgpm24ilkdf55kcuet40pt4aj0qgqbaqosb81b1rgb0'
2019/09/07 21:30:37 DEBUG : Cache remote wsGDcache-luminor:p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410: new object 'oefa3ocd4oeupvuq7thqs0msm4/nvgs6nahn3cfdphl9sb3uuomjpp0b6f974sl7uhqcq8vnju24efg'
2019/09/07 21:30:37 DEBUG : oefa3ocd4oeupvuq7thqs0msm4/m65bt38b4psmjh1e4o4mnrqg8upkd4m4ao4qs18nmndr1s7pjmgg: find: error: couldn't open parent bucket for p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410/oefa3ocd4oeupvuq7thqs0msm4
2019/09/07 21:30:37 DEBUG : oefa3ocd4oeupvuq7thqs0msm4/79iqb07amhlissidlb6trdl8g9lqmeafeafgpm24ilkdf55kcuet40pt4aj0qgqbaqosb81b1rgb0: find: error: couldn't open parent bucket for p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410/oefa3ocd4oeupvuq7thqs0msm4
2019/09/07 21:30:37 DEBUG : oefa3ocd4oeupvuq7thqs0msm4/nvgs6nahn3cfdphl9sb3uuomjpp0b6f974sl7uhqcq8vnju24efg: find: error: couldn't open parent bucket for p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410/oefa3ocd4oeupvuq7thqs0msm4
2019/09/07 21:30:38 DEBUG : find failed: not found in either local or remote fs
2019/09/07 21:30:38 DEBUG : Cache remote wsGDcache-luminor:p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410: moving obj 'oefa3ocd4oeupvuq7thqs0msm4/ghkd8j8511h8mctf9o7eup8nj8k1ro38rfa24d4cd7fiqvm4apa0' -> oefa3ocd4oeupvuq7thqs0msm4/ghkd8j8511h8mctf9o7eup8nj8k1ro38rfa24d4cd7fiqvm4apa0
2019/09/07 21:30:38 ERROR : : can't move - not same remote type
2019/09/07 21:30:38 DEBUG : old/S03E03 - Nesting.mkv: Can't move, switching to copy
2019/09/07 21:30:38 DEBUG : Cache remote wsGDcache-luminor:p07digl3gdh1vbnm1pi7slc158/f163u0m1kh2u66jsu1ncaobbkbj8l5j47c8jnkdobloun2vp4410: copy obj 'oefa3ocd4oeupvuq7thqs0msm4/ghkd8j8511h8mctf9o7eup8nj8k1ro38rfa24d4cd7fiqvm4apa0' -> 'oefa3ocd4oeupvuq7thqs0msm4/ghkd8j8511h8mctf9o7eup8nj8k1ro38rfa24d4cd7fiqvm4apa0'

2

This must be something to do with the cache backend. Are the failing files cached maybe?

3

Thanks for your reply. The crypt is backed by a cache, yes.

What info can I provide to help diagnose this? What could be causing the issue with the cache? The only time I use rclone is to run the sync command (i.e., I'm not exposing my gDrive files to Plex or anything like that).

4

Can you paste your config with the secrets XXXed out?

It might be worth clearing the cache db - run rclone once with --cache-db-purge

5

Will do as soon as I'm home. I appreciate your attention to this issue.

1 Like
6

I tried the --cache-db-purge and am still seeing the same behavior. It uploads many times. Right now there are only 2 files to be uploaded, totaling 1385MB in size. But the transferred stats show that 4.4G have already been transferred.

Here is my config file...appreciate any help and insight!

[xxxGDrive]
type = drive
client_id = xxx.apps.googleusercontent.com
client_secret = xxx
scope = drive
service_account_file = c:\rclone\rclone-xxx-SA-priv.json
root_folder_id = xxx

[wsGDcache-luminor]
type = cache
remote = xxxGDrive:/luminor
info_age = 2d

[wsGDcrypt-luminor]
type = crypt
remote = wsGDcache-luminor:/
filename_encryption = standard
directory_name_encryption = true
password = xxx

7

If you make another remote without the cache in there, does it work? Like this?

[xxxGDrive]
type = drive
client_id = xxx.apps.googleusercontent.com
client_secret = xxx
scope = drive
service_account_file = c:\rclone\rclone-xxx-SA-priv.json
root_folder_id = xxx

[wsGDcrypt-luminor-direct]
type = crypt
remote = xxxGDrive:/luminor
filename_encryption = standard
directory_name_encryption = true
password = xxx
8

Good question. I will try that as soon as I can, which sadly may not be for a couple of weeks (I have only intermittent access to this machine :frowning: )

9

I had a chance to try it quickly before leaving. It appears to work correctly without the cache, because now instead of showing file activity on the files to be backed up, it is showing upload of the new files, and the log contains entries for the old files being moved (": Moved (server side)")

10

Good, so we have a work-around at least.

This is almost certainly a bug in the cache backend. I don't currently have a cache backend maintainer so I'm not sure when it will be fixed, but it is worth making a new issue on github about this with a link to this forum thread - thanks.

11

I appreciate all of your help, and admit I'm a bit embarrassed that I didn't think to try it without the cache. I will indeed make a bug report. Thanks!

1 Like
12

Hi, I have a similar problem, so not sure if this is the right place to post.
After configuring my drive with default options I was able to sync a folder to gdrive. The files appeared correctly there. However subsequent syncs are failing to propagate to gdrive. Command line output seems ok but the files are not showing up. I didn't set up a client id nor service accounts. My config is this:

[gdrive]
type = drive
scope = drive
root_folder_id = <blah>
token = {"access_token":"blah","token_type":"Bearer","refresh_token":"blah,"expiry":"2020-03-28T11:21:40.172306341Z"}

A simple copy command like this

rclone -v copy test2.txt gdrive
2020/03/28 10:39:33 INFO  : test2.txt: Copied (new)
2020/03/28 10:39:33 INFO  : 
Transferred:   	         5 / 5 Bytes, 100%, 791 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

But the file simple doesn't appear anywhere. Can someone please help ?

13

You left the : off the end of gdrive so rclone will have copied it to a local directory called gdrive.

14

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.