Rclone moveto accidentally appended \r (carriage return, ^M)

#1

Hi guys,

I ran a bunch of rclone moveto commands with rclone v1.39 and now I am in trouble, remotely on my G Drive crypt. Basically, I had the following structure:

library:Movies/My Movie (2019).mkv

I wanted to move it to:

library:Movies/My Movie (2019)/My Movie (2019).mkv

To do this, I tried running:

rclone moveto -vv --log-file “home/logfile.txt” “library:Movies/My Movie (2019).mkv” “library:Movies/My Movie (2019)/My Movie (2019).mkv”

Unfortunately the bash file I created to do a bunch of these at once appended an \r to the end of each command, so in my logfile I see:

2019/04/10 17:48:27 DEBUG : rclone: Version “v1.39” starting with parameters [“rclone” “moveto” “-vv” “–log-file” “/home/rd/movelog.txt” “library:Movies/My Movie (2019).mkv” “library:Movies/My Movie (2019)/My Movie (2019).mkv\r”] ((((emphasis mine — notice the \r))))
2019/04/10 17:48:31 INFO : Encrypted drive ‘library:Movies/My Movie (2019)/’: Modify window is 1ms
2019/04/10 17:48:32 DEBUG : My Movie (2019).mkv: Couldn’t find file - need to transfer
2019/04/10 17:48:33 INFO : My Movie (2019).mkv: Moved (server side)
2019/04/10 17:48:33 INFO :
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 0
Transferred: 1
Elapsed time: 6.5s

2019/04/10 17:48:33 DEBUG : Go routines at exit 5
2019/04/10 17:48:33 DEBUG : rclone: Version “v1.39” finishing with parameters [“rclone” “moveto” “-vv” “–log-file” “/home/rd/movelog.txt” “library:Movies/My Movie (2019).mkv” “library:Movies/My Movie (2019)/My Movie (2019).mkv**\r**”]

Does anyone have any idea how I can reverse these actions or fix them??

rclone lsd “library:Movies/My Movie (2019)” and rclone ls “library:Movies/My Movie (2019)” both return nothing. Running the rclone moveto in reverse (ie: rclone moveto “library:Movies/My Movie (2019)/My Movie (2019).mkv\r” “library:Movies/My Movie (2019).mkv”) says it can’t find the directory.

EDIT: Was able to get all fixed by removing the check from rclone cipher, using that version of rclone to mount, and then doing a bunch of mv/rename commands. You guys are incredible <3 <3 <3 <3

0 Likes

#2

1.39 is super super old. Are you able to mount it? Easier to do it that way imo.

If not, let me know.

0 Likes

#3

I have 1.45 installed on another server if that would be better?

And if you can teach me how/show me what commands to run, I have root access :smile: I’m not the best at Linux but I’m pretty OK at following directions!

0 Likes

#4

If you do a rclone ls, do you get something like this?

 rclone ls gcrypt:Test
        0 blah.mkv\r
        0 test.mkv\r

0 Likes

#5

I just did something a bit dirty, but seems to work:

rclone ls gcrypt:Test  | awk {' print "rclone move \"gcrypt:Test\/"$2 "\" \"gcrypt:Test\/" $2 "\"" '} | sed  's/\\r//2'awk: cmd. line:1: warning: escape sequence `\/' treated as plain `/'
rclone move "gcrypt:Test/blah.mkv\r" "gcrypt:Test/blah.mkv"
rclone move "gcrypt:Test/test.mkv\r" "gcrypt:Test/test.mkv"
[felix@gemini Test]$ rclone move "gcrypt:Test/blah.mkv\r" "gcrypt:Test/blah.mkv" -vv
2019/04/10 20:37:24 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "move" "gcrypt:Test/blah.mkv\\r" "gcrypt:Test/blah.mkv" "-vv"]
2019/04/10 20:37:24 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019/04/10 20:37:28 DEBUG : blah.mkv\r: Couldn't find file - need to transfer
2019/04/10 20:37:30 INFO  : blah.mkv\r: Moved (server side)
2019/04/10 20:37:30 INFO  :
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        5.9s

2019/04/10 20:37:30 DEBUG : 4 go routines active
2019/04/10 20:37:30 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "move" "gcrypt:Test/blah.mkv\\r" "gcrypt:Test/blah.mkv" "-vv"]
[felix@gemini Test]$ ls
 blah.mkv  'test.mkv\r'
0 Likes

#6

Hm, unfortunately I get:

rclone ls -vv “library:Movies/My Movie (2019)”
2019/04/10 22:35:52 DEBUG : Using config file from “/home/.config/rclone/rclone.conf”
2019/04/10 22:35:52 DEBUG : rclone: Version “v1.39” starting with parameters [“rclone” “ls” “-vv” “library:Movies/My Movie (2019)”]
2019/04/10 22:35:55 INFO : Encrypted drive ‘library:Movies/My Movie (2019)’: Modify window is 1ms
2019/04/10 22:35:56 DEBUG : qdgmq08ktoq7kd9vuq7m162uu5j3a8h5ji2ab1gpl6e4dq2mdkremrcjmf0cskhrp8mhpna7093tl09bdls89ui834vunburjdrqmf8qr26gtjcub6chiar95eg8cctp: Skipping undecryptable file name: bad decryption - contains control chars
2019/04/10 22:35:56 DEBUG : Go routines at exit 5
2019/04/10 22:35:56 DEBUG : rclone: Version “v1.39” finishing with parameters [“rclone” “ls” “-vv” “library:Movies/My Movie (2019)”]

When I try an rclone ls -vv (just ls returns nothing)

Control character, so I guess it’s not actually \r but ^M or however a carriage return is displayed? Note: I tried 1.45 as well, got the same result.

0 Likes

#7

Not that I expect it to be different, but if you can use the latest stable to validate that as it’s 1.46.

If that error also pops up on 1.46, we probably need some help as if you can’t see it via the error being tossed from encryption, I’m not sure how to move it.

0 Likes

#8

Yup, same issue, same logs :frowning:

2019/04/10 22:59:58 DEBUG : qdgmq08ktoq7kd9vuq7m162uu5j3a8h5ji2ab1gpl6e4dq2mdkremrcjmf0cskhrp8mhpna7093tl09bdls89ui834vunburjdrqmf8qr26gtjcub6chiar95eg8cctp: Skipping undecryptable file name: bad decryption - contains control chars

0 Likes

#9

Guessing I’ll need to file an issue on Gitlab?

0 Likes

#10

Yeah, that’s so odd. Let me see if I can recreate it as well.

0 Likes

#11

I think you’ll need to comment this out so that rclone stops ignoring the files with /r in

diff --git a/backend/crypt/cipher.go b/backend/crypt/cipher.go
index c3dd87768..291198fe5 100644
--- a/backend/crypt/cipher.go
+++ b/backend/crypt/cipher.go
@@ -210,12 +210,12 @@ func (c *cipher) putBlock(buf []byte) {
 // check to see if the byte string is valid with no control characters
 // from 0x00 to 0x1F and is a valid UTF-8 string
 func checkValidString(buf []byte) error {
-	for i := range buf {
-		c := buf[i]
-		if c >= 0x00 && c < 0x20 || c == 0x7F {
-			return ErrorBadDecryptControlChar
-		}
-	}
+	// for i := range buf {
+	// 	c := buf[i]
+	// 	if c >= 0x00 && c < 0x20 || c == 0x7F {
+	// 		return ErrorBadDecryptControlChar
+	// 	}
+	// }
 	if !utf8.Valid(buf) {
 		return ErrorBadDecryptUTF8
 	}
1 Like

#12

Here’s how to reproduce:

I had a text file made on Windows, Notepad++, with a few commands like:
rclone moveto “library:Movies/My Movie (2019).mkv” “library:Movies/My Movie (2019)/My Movie (2019).mkv”
rclone moveto “library:Movies/My Movie2 (2019).mkv” “library:Movies/My Movie2 (2019)/My Movie (2019).mkv”
…etc.

I copied this file over to my server via Filezilla and ran it. I guess the carriage returns got added in the process.

@ncw I will try that, although this may get ugly LOL – will report back

0 Likes

#13

Trying to build go now, using:

git clone https://github.com/ncw/rclone.git
cd rclone [and make the suggested changes]
go build
./rclone version

NOTE: I don’t have GOPATH or GOROOT set. I placed the go binary (https://dl.google.com/go/go1.12.3.linux-amd64.tar.gz) at /usr/bin/go. Should I do something different?

But I’m getting when I try go build:
go: extracting cloud.google.com/go v0.36.0
build github.com/ncw/rclone: cannot load bufio: cannot find module providing package bufio

I’m sure there’s some very simple user error but I can’t figure out what I’m doing wrong. Never really worked with go before.

Will report back shortly…

EDIT: After installing with ‘umake go’ it all worked. Now let’s try the rclone commands…

0 Likes

#14

Awesome, after commenting out that check, I can find it in rclone ls.

Now, just need to figure out how to move it! rclone moveto with \r doesn’t work – seems I’m going to need an actual carriage return character in my rclone command.

Anyone have any ideas?

0 Likes

#15

What does the rclone ls look like?

You can also try to rclone mount it somewhere to see if you move them that way.

How many files did you impact?

0 Likes

#16

2019/04/11 12:08:30 DEBUG : rclone: Version “v1.46-DEV” starting with parameters ["./rclone-dev" “ls” “-vv” “library:Movies/My Movie (2019)”]
2019/04/11 12:08:30 DEBUG : Using config file from “/home/rd/.config/rclone/rclone.conf”
3282530834 My Movie (2019).mkv
2019/04/11 12:08:32 DEBUG : 4 go routines active
2019/04/11 12:08:32 DEBUG : rclone: Version “v1.46-DEV” finishing with parameters ["./rclone-dev" “ls” “-vv” “library:Movies/My Movie (2019)”]

How would you suggest I mount it?

Impacted about 1900 files, but I have all of the commands I ran…

0 Likes

#17

Just mount the entire drive and you might be able to browse it.

0 Likes

#18

So something like:

rclone mount library: /somemountpoint

Once it’s mounted, that gives us a bit flexibility I think to run some other commands and maybe solve it that way.

0 Likes

#19

Yup, it’s mounted and I was able to do a simple mv command and it seemed to work! Awesome!!!

Now we just need to come up with a command to fix all the broken files in one go… I’m afraid to do any more regex’ing LOL

0 Likes

#20

What is an example of ls -al on a few files from the mount?

0 Likes