GDrive when rclone command is run from file rather than typed into terminal

stategery · August 20, 2021, 3:58am

What is the problem you are having with rclone?

I know I'm doing something wrong. rclone seems to run differently when I run a command directly in terminal vs from a script. Same command copy/pasted from the script. If run directly it works as expect. syncs a directory to GDrive. If run from a file, it creates a new directly in GDrive that rclone then can't find. rclone ls analytics-g-drive:analytics can find the folders I create in GDrive just fine. But if run on one of the folders rclone created when run from a file, it's like they don't exist.

What is your rclone version (output from `rclone version`)

rclone v1.57.0-beta.5635.3a2f748ae

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-1045-aws (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.16.7
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

--update -vv --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s "analytics/csvs" "analytics-g-drive:analytics"

The rclone config contents with secrets removed.

[analytics-g-drive]
type = drive
scope = drive
token = 
shared-with-me = false
team_drive =

A log from the command with the `-vv` flag

Log from when script is run directly.

rclone sync -vv --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s "analytics/csvs" "analytics-g-drive:analytics"
2021/08/20 03:48:12 DEBUG : rclone: Version "v1.57.0-beta.5635.3a2f748ae" starting with parameters ["rclone" "sync" "-vv" "--transfers" "30" "--checkers" "8" "--contimeout" "60s" "--timeout" "300s" "--retries" "3" "--low-level-retries" "10" "--stats" "1s" "analytics/csvs" "analytics-g-drive:analytics"]
2021/08/20 03:48:12 DEBUG : Creating backend with remote "analytics/csvs"
2021/08/20 03:48:12 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2021/08/20 03:48:12 DEBUG : fs cache: renaming cache item "analytics/csvs" to be canonical "/home/ubuntu/analytics/csvs"
2021/08/20 03:48:12 DEBUG : Creating backend with remote "analytics-g-drive:analytics"
2021/08/20 03:48:12 DEBUG : Google drive root 'analytics': 'root_folder_id = 0AMTA8vaeDlRIUk9PVA' - save this in the config to speed up startup
2021/08/20 03:48:13 DEBUG : Google drive root 'analytics': Waiting for checks to finish
2021/08/20 03:48:13 DEBUG : Google drive root 'analytics': Waiting for transfers to finish
2021/08/20 03:48:13 INFO  :
Transferred:      968.746Ki / 968.746 KiByte, 100%, 0 Byte/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:         1.5s
Transferring:
 *                               Daily Stats.csv:100% /968.746Ki, 0/s, -

2021/08/20 03:48:14 DEBUG : Daily Stats.csv: md5 = 7bac668fc207b07e638ec164543864a1 OK
2021/08/20 03:48:14 INFO  : Daily Stats.csv: Copied (new)
2021/08/20 03:48:14 DEBUG : Waiting for deletions to finish
2021/08/20 03:48:14 INFO  :
Transferred:      968.746Ki / 968.746 KiByte, 100%, 968.293 KiByte/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.8s

2021/08/20 03:48:14 DEBUG : 5 go routines active

Log from when the same exact string is run from a file:

source analytics/rclone-g-analytics.sh
2021/08/20 03:48:35 DEBUG : rclone: Version "v1.57.0-beta.5635.3a2f748ae" starting with parameters ["rclone" "sync" "-vv" "--transfers" "30" "--checkers" "8" "--contimeout" "60s" "--timeout" "300s" "--retries" "3" "--low-level-retries" "10" "--stats" "1s" "analytics/csvs" "analytics-g-drive:analytics\r"]
2021/08/20 03:48:35 DEBUG : Creating backend with remote "analytics/csvs"
2021/08/20 03:48:35 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2021/08/20 03:48:35 DEBUG : fs cache: renaming cache item "analytics/csvs" to be canonical "/home/ubuntu/analytics/csvs"
2021/08/20 03:48:35 DEBUG : Creating backend with remote "analytics-g-drive:analytics\r"
': 'root_folder_id = 0AMTA8vaeDlRIUk9PVA' - save this in the config to speed up startup
': Waiting for checks to finishgle drive root 'analytics
': Waiting for transfers to finish drive root 'analytics
2021/08/20 03:48:37 INFO  :
Transferred:              0 / 968.746 KiByte, 0%, 0 Byte/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:         1.7s
Transferring:
 *                               Daily Stats.csv:  0% /968.746Ki, 0/s, -

2021/08/20 03:48:38 DEBUG : Daily Stats.csv: md5 = 7bac668fc207b07e638ec164543864a1 OK
2021/08/20 03:48:38 INFO  : Daily Stats.csv: Copied (new)
2021/08/20 03:48:38 DEBUG : Waiting for deletions to finish
2021/08/20 03:48:38 INFO  :
Transferred:      968.746Ki / 968.746 KiByte, 100%, 968.730 KiByte/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         2.6s

2021/08/20 03:48:38 DEBUG : 5 go routines active

I see this as a difference - Creating backend with remote "analytics-g-drive:analytics**\r**" but I don't know what that means. Maybe read-only but I'm not sure why that would happen when it's run from a from a file rather than directly in terminal. I thought running like that was identical to typing a command in. Against obviously I'm missing something and hoping you all can help.

asdffdsa · August 20, 2021, 1:12pm

hello and welcome to the forum,

the destination folder for each command is different.
perhaps a simple typo, perhaps a copy/paste mistake.

choose which folder is correct and use that for both commands.

in the log, for each command, you can see which folder rclone is using
Creating backend with remote "analytics-g-drive:analytics"
Creating backend with remote "analytics-g-drive:analytics\r"

albertony · August 20, 2021, 1:34pm

Perhaps your script has Windows line endings \r\n in it? Since Unix uses \n only as line endings, the \r will be treated as a regular character.

asdffdsa · August 20, 2021, 2:02pm

good point,

as i was writing that post, i was thinking the same thing but by the time the post was finished, i forgot about it.

stategery · August 20, 2021, 2:57pm

Holy crap, that's it! I had written the file in notepad and dropped it into the server over SFTP. I would not have caught that for sooo long. Literally copy pasted the working code into a new file directly in the server and tada!

Thank you!!!

asdffdsa · August 20, 2021, 3:09pm

oy, notepad?

on windows, i use notepad++, which can

edit/run windows batch scripts and perform EOL conversions.
if WSL is installed, edit/run bash scripts
run bash commands over ssh on a remote linux server.

stategery · August 20, 2021, 3:22pm

I actually have notepad++ installed already. I use it for anything over a single line and rarely save from it. Just use it as an extended clipboard but in this case I saved my file. I will now start using notepad++ for all things code related!

I was actually accessing my server through ssh and WSL. I didn't know notepad++ could do that directly. Thanks for the heads up!

asdffdsa · August 20, 2021, 3:32pm

about WSL,

system · August 23, 2021, 3:32pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.