Inconsistent behavior for GDrive ls and copy/sync on Win10

Graphity · February 10, 2021, 12:29pm

What is the problem you are having with rclone?

I want to backup files from GDrive and tried rclone. I can list a directory (with rclone ls) but I can't copy or sync it.

What is your rclone version (output from `rclone version`)

v1.54.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 Bit
Microsoft Windows [Version 10.0.18363.1316]

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

I'm using PowerShell

.\rclone.exe copy GDrive-c:share_inxxx\_my-App\export_test D:\Download\rclone-test\ -vv

The rclone config contents with secrets removed.

[GDrive-xxxxxx]
type = drive
client_id = XXXX
client_secret = XXXX
scope = drive.readonly

A log from the command with the `-vv` flag

rclone copy

PS D:\opt\rclone\rclone-v1.54.0-windows-amd64> .\rclone.exe copy GDrive-xxxxxx:share_inxxx\_my-App\export_test D:\Download\rclone-test\ -vv
2021/02/10 13:16:22 DEBUG : rclone: Version "v1.54.0" starting with parameters ["D:\\opt\\rclone\\rclone-v1.54.0-windows-amd64\\rclone.exe" "copy" "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test" "D:\\Download\\rclone-test\\" "-vv"]
2021/02/10 13:16:22 DEBUG : Using config file from "C:\\Users\\xxxx\\.config\\rclone\\rclone.conf"
2021/02/10 13:16:22 DEBUG : Creating backend with remote "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test"
2021/02/10 13:16:23 DEBUG : Google drive root 'share_inxxx/_my-App/export_test': root_folder_id = "0ANLD67QNQ2tbUk9PVA" - save this in the config to speed up startup
2021/02/10 13:16:25 DEBUG : fs cache: renaming cache item "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test" to be canonical "GDrive-xxxxxx:share_inxxx/_my-App/export_test"
2021/02/10 13:16:25 DEBUG : Creating backend with remote "D:\\Download\\rclone-test\\"
2021/02/10 13:16:25 DEBUG : fs cache: renaming cache item "D:\\Download\\rclone-test\\" to be canonical "//?/D:/Download/rclone-test/"
2021/02/10 13:16:25 ERROR : : error reading source directory: directory not found
2021/02/10 13:16:25 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 13:16:25 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 13:16:25 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2021/02/10 13:16:25 ERROR : : error reading source directory: directory not found
2021/02/10 13:16:25 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 13:16:25 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 13:16:25 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2021/02/10 13:16:26 ERROR : : error reading source directory: directory not found
2021/02/10 13:16:26 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 13:16:26 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 13:16:26 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2021/02/10 13:16:26 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.5s

2021/02/10 13:16:26 DEBUG : 4 go routines active
2021/02/10 13:16:26 Failed to copy: directory not found

rclone ls

PS D:\opt\rclone\rclone-v1.54.0-windows-amd64> .\rclone.exe ls GDrive-xxxxxx:share_inxxx\_my-App\export_test -vv
2021/02/10 13:24:42 DEBUG : rclone: Version "v1.54.0" starting with parameters ["D:\\opt\\rclone\\rclone-v1.54.0-windows-amd64\\rclone.exe" "ls" "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test" "-vv"]
2021/02/10 13:24:42 DEBUG : Using config file from "C:\\Users\\xxxx\\.config\\rclone\\rclone.conf"
2021/02/10 13:24:42 DEBUG : Creating backend with remote "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test"
2021/02/10 13:24:43 DEBUG : Google drive root 'share_inxxx/_my-App/export_test': root_folder_id = "0ANLD67QNQ2tbUk9PVA" - save this in the config to speed up startup
2021/02/10 13:24:44 DEBUG : fs cache: renaming cache item "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test" to be canonical "GDrive-xxxxxx:share_inxxx/_my-App/export_test"
       -1 Doküment von anderem User.docx
       -1 Export Täst Docüment.docx
     7917 besetztes Verzeichnis mit Umläuten/bild von anderen nützer.png
 16526816 besetztes Verzeichnis mit Umläuten/umläutün.png
2021/02/10 13:24:44 DEBUG : 6 go routines active

rclone sync

PS D:\opt\rclone\rclone-v1.54.0-windows-amd64> .\rclone.exe -i sync GDrive-xxxxxx:\share_inxxx\_my-App\export_test\ D:\Download\rclone-test\ -vv
2021/02/10 13:25:41 DEBUG : rclone: Version "v1.54.0" starting with parameters ["D:\\opt\\rclone\\rclone-v1.54.0-windows-amd64\\rclone.exe" "-i" "sync" "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test\\" "D:\\Download\\rclone-test\\" "-vv"]
2021/02/10 13:25:41 DEBUG : Using config file from "C:\\Users\\xxxx\\.config\\rclone\\rclone.conf"
2021/02/10 13:25:41 DEBUG : Creating backend with remote "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test\\"
2021/02/10 13:25:42 DEBUG : Google drive root 'share_inxxx/_my-App/export_test': root_folder_id = "0ANLD67QNQ2tbUk9PVA" - save this in the config to speed up startup
2021/02/10 13:25:43 DEBUG : fs cache: renaming cache item "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test\\" to be canonical "GDrive-xxxxxx:share_inxxx/_my-App/export_test"
2021/02/10 13:25:43 DEBUG : Creating backend with remote "D:\\Download\\rclone-test\\"
2021/02/10 13:25:43 DEBUG : fs cache: renaming cache item "D:\\Download\\rclone-test\\" to be canonical "//?/D:/Download/rclone-test/"
2021/02/10 13:25:44 ERROR : : error reading source directory: directory not found
2021/02/10 13:25:44 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 13:25:44 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 13:25:44 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting files as there were IO errors
2021/02/10 13:25:44 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting directories as there were IO errors
2021/02/10 13:25:44 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2021/02/10 13:25:44 ERROR : : error reading source directory: directory not found
2021/02/10 13:25:44 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 13:25:44 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 13:25:44 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting files as there were IO errors
2021/02/10 13:25:44 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting directories as there were IO errors
2021/02/10 13:25:44 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2021/02/10 13:25:45 ERROR : : error reading source directory: directory not found
2021/02/10 13:25:45 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 13:25:45 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 13:25:45 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting files as there were IO errors
2021/02/10 13:25:45 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting directories as there were IO errors
2021/02/10 13:25:45 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2021/02/10 13:25:45 NOTICE:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.5s

2021/02/10 13:25:45 DEBUG : 4 go routines active
2021/02/10 13:25:45 Failed to sync: directory not found

asdffdsa · February 10, 2021, 3:30pm

hello and welcome to the forum,

not sure what the exact problem is but for the two commands, the souce is not exactly the same.

the sync has extra back slashes
ls: GDrive-xxxxxx:share_inxxx\_my-App\export_test
sync GDrive-xxxxxx:\share_inxxx\_my-App\export_test\:

and as a result, corresponding backends created by rclone are not exactly the same
ls: Creating backend with remote "GDrive-xxxxxx:share_inxxx\\_my-App\\export_test"
sync: Creating backend with remote "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test\\"

Graphity · February 10, 2021, 3:58pm

Good point!

But still same error message with sync:

PS D:\opt\rclone\rclone-v1.54.0-windows-amd64> .\rclone.exe -i sync GDrive-xxxxxx:\share_inxxx\_my-App\export_test D:\Download\rclone-test\ -vv
2021/02/10 16:55:48 DEBUG : rclone: Version "v1.54.0" starting with parameters ["D:\\opt\\rclone\\rclone-v1.54.0-windows-amd64\\rclone.exe" "-i" "sync" "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test" "D:\\Download\\rclone-test\\" "-vv"]
2021/02/10 16:55:48 DEBUG : Using config file from "C:\\Users\\xxxx\\.config\\rclone\\rclone.conf"
2021/02/10 16:55:48 DEBUG : Creating backend with remote "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test"
2021/02/10 16:55:48 DEBUG : GDrive-xxxxxx: Loaded invalid token from config file - ignoring
2021/02/10 16:55:48 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2021/02/10 16:55:48 DEBUG : GDrive-xxxxxx: Saved new token in config file
2021/02/10 16:55:49 DEBUG : Google drive root 'share_inxxx/_my-App/export_test': root_folder_id = "0ANLD67QNQ2tbUk9PVA" - save this in the config to speed up startup
2021/02/10 16:55:51 DEBUG : fs cache: renaming cache item "GDrive-xxxxxx:\\share_inxxx\\_my-App\\export_test" to be canonical "GDrive-xxxxxx:share_inxxx/_my-App/export_test"
2021/02/10 16:55:51 DEBUG : Creating backend with remote "D:\\Download\\rclone-test\\"
2021/02/10 16:55:51 DEBUG : fs cache: renaming cache item "D:\\Download\\rclone-test\\" to be canonical "//?/D:/Download/rclone-test/"
2021/02/10 16:55:51 ERROR : : error reading source directory: directory not found
2021/02/10 16:55:51 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 16:55:51 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 16:55:51 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting files as there were IO errors
2021/02/10 16:55:51 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting directories as there were IO errors
2021/02/10 16:55:51 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2021/02/10 16:55:51 ERROR : : error reading source directory: directory not found
2021/02/10 16:55:51 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 16:55:51 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 16:55:51 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting files as there were IO errors
2021/02/10 16:55:51 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting directories as there were IO errors
2021/02/10 16:55:51 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2021/02/10 16:55:51 ERROR : : error reading source directory: directory not found
2021/02/10 16:55:51 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for checks to finish
2021/02/10 16:55:51 DEBUG : Local file system at //?/D:/Download/rclone-test/: Waiting for transfers to finish
2021/02/10 16:55:51 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting files as there were IO errors
2021/02/10 16:55:51 ERROR : Local file system at //?/D:/Download/rclone-test/: not deleting directories as there were IO errors
2021/02/10 16:55:51 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2021/02/10 16:55:51 NOTICE:
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         3.3s

2021/02/10 16:55:51 DEBUG : 6 go routines active
2021/02/10 16:55:51 Failed to sync: directory not found

asdffdsa · February 10, 2021, 4:32pm

i am not a gdrive expert, we have them and one should stop by soon.
you can just wait for them to stop by

or try my suggestion,
the source from the latest sync command still does not match the source from the ls command.

Graphity · February 10, 2021, 5:10pm

Wow, you pointed me into the right direction asdffdsa! Thank you!

Thanks to your input, I started comparing the lines "Creating backend with remote" and they were different!
As you might have guessed, I edited the output before I posted it here. In my console, one part of the remote path included a German Umlaut (ü, to be exact). It was there at the ls command, it wasn't there at the sync command.

How did that happen?

I'm using PowerShell running on Windows 10. I first did a rclone ls on a directory without Umlaut. I copy/pasted the directory name when composing the rclone ls command I posted.
When I composed the rclone sync command, I copy/pasted the path from the rclone ls command. This broke it!
Thanks to your input, asdffdsa, I composed the rclone sync command again, this time, I copy/pasted from result of rclone ls command. Now it worked!

asdffdsa · February 10, 2021, 7:45pm

well, i have a time machine, if you invite me to your country and offer me a beer, we can travel back in time together.

system · February 13, 2021, 7:45pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.