Allow_import_name_change - not doing what I thought it would?

Help and Support
1

What is the problem you are having with rclone?

I have a new situation where I want to copy batches (65 in the first batch) of newly-created ODTs or DOCXs to G Drive - but of course they get copied with their extension intact . . I was under the impression that it was possible to do the copy and convert the docs into native Google Drive format during the process? I put "allow_import_name_change = true" in the config but it didn't make any difference - I am probably missing something . .

What is your rclone version (output from rclone version)

rclone v1.44-088-g63a2a935-beta

  • os/arch: linux/amd64
  • go version: go1.11.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Fedora Linux v31 x86_64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -L -v sync --size-only --drive-formats "ods,odt" odt/ gd:/0_Relatives/Diary_02/

The rclone config contents with secrets removed.

[gd]                                                                                                                                                                                                                 
type = drive                                                                                                                                                                                                         
client_id =                                                                                                                                                                                                          
client_secret =                                                                                                                                                                                                      
allow_import_name_change = true                                                                                                                                                                                      
token = {"access_token":"xxxx","token_type":
"Bearer","refresh_token":"xxxx","expiry":"2020-09-27T15:45:14.755278437+10:00"}

A log from the command with the -vv flag

2020/09/27 15:38:07 DEBUG : rclone: Version "v1.44-088-g63a2a935-beta" starting with parameters ["rclone" "-L" "-vv" "sync" "--size-only" "--drive-formats" "ods,odt" "odt/" "gd:/0_Relatives/Diary_02/"]
2020/09/27 15:38:07 DEBUG : Using config file from "/home/phr/.config/rclone/rclone.conf"
2020/09/27 15:38:09 INFO  : Google drive root '0_Relatives/Diary_02': Waiting for checks to finish
2020/09/27 15:38:09 DEBUG : IMG_5844.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5844.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5845.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5838.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5842.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5842.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5841.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5841.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5848.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5848.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5846.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5849.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5846.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5849.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5850.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5850.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5852.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5852.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5853.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5853.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5854.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5851.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5854.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5851.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5855.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5856.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5855.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5856.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5857.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5858.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5857.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5858.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5859.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5860.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5859.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5860.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5861.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5862.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5861.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5862.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5863.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5864.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5863.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5864.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5865.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5866.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5866.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5867.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5867.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5868.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5868.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5869.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5869.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5870.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5870.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5871.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5871.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5865.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5872.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5872.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5873.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5873.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5875.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5875.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5876.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5876.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5877.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5874.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5877.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5874.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5878.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5879.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5878.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5879.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5880.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5838.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5881.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5882.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5882.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5883.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5883.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5884.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5884.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5881.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5885.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5886.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5886.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5885.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5887.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5888.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5887.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5888.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5889.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5890.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5889.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5891.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5891.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5892.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5892.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5890.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5893.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5894.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5894.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5895.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5895.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5896.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5896.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5897.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5893.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5897.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5898.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5898.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5900.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5900.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5901.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5901.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5902.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5902.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5899.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5899.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5845.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5880.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5839.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5839.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5847.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5847.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5843.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5843.odt: Unchanged skipping
2020/09/27 15:38:09 DEBUG : IMG_5840.odt: Sizes identical
2020/09/27 15:38:09 DEBUG : IMG_5840.odt: Unchanged skipping
2020/09/27 15:38:09 INFO  : Google drive root '0_Relatives/Diary_02': Waiting for transfers to finish
2020/09/27 15:38:11 INFO  : IMG_5838.docx: Copied (new)
2020/09/27 15:38:11 INFO  : Waiting for deletions to finish
2020/09/27 15:38:11 INFO  : 
Transferred:   	  657.224k / 657.224 kBytes, 100%, 138.130 kBytes/s, ETA 0s
Errors:                 0
Checks:                65 / 65, 100%
Transferred:            1 / 1, 100%
Elapsed time:        4.7s

2020/09/27 15:38:11 DEBUG : 4 go routines active
2020/09/27 15:38:11 DEBUG : rclone: Version "v1.44-088-g63a2a935-beta" finishing with parameters ["rclone" "-L" "-vv" "sync" "--size-only" "--drive-formats" "ods,odt" "odt/" "gd:/0_Relatives/Diary_02/"]
2

That's an ancient version of rclone so option probably isn't there. Try to update and retest.

1 Like
3

@Animosity022 ,

Ah . . OK . . will do - I should have said also that when I try and upload a batch of files like this through the normal GD GUI - most of them fail - and I would much rather use rclone anyway . .

4

I just updated to:

rclone-v1.53.1-linux-amd64.rpm

but sadly no improvement . .

5

I think you want --drive-import-formats ods,odt as well as --drive-formats ods,odt - that will import the ods/odt files as google documents which is what I think you want.

At the moment, without --drive-import-formats, you are just uploading them as files.

1 Like
6

@ncw ,

That's it! Thanks so much!

Phil.

7

@ncw ,

I spoke too soon . . it works for one or two files but then I get errors . . I might try using a script and sending one file at a time to see if the improves the situation - odd though, I didn't have this problem when the files weren't getting converted to G native format . . hmm . .

I will report back with the single file copy exercise . .

P.

2020/09/27 20:35:29 DEBUG : rclone: Version "v1.53.1" starting with parameters ["/usr/bin/rclone" "-L" "-vv" "sync" "--size-only" "--drive-formats" "ods,odt" "--drive-import-formats" "ods,odt" "odt/" "gd:/0_Relatives/Diary_02/"]
2020/09/27 20:35:29 DEBUG : Creating backend with remote "odt/"
2020/09/27 20:35:29 DEBUG : Using config file from "/home/phr/.config/rclone/rclone.conf"
2020/09/27 20:35:29 DEBUG : fs cache: renaming cache item "odt/" to be canonical "/home/phr/relatives/MumDiaries/Diary_02/heic/odt"
2020/09/27 20:35:29 DEBUG : Creating backend with remote "gd:/0_Relatives/Diary_02/"
2020/09/27 20:35:30 DEBUG : Google drive root '0_Relatives/Diary_02': root_folder_id = "0AA-FsVZgP3GtUk9PVA" - save this in the config to speed up startup
2020/09/27 20:35:31 DEBUG : fs cache: renaming cache item "gd:/0_Relatives/Diary_02/" to be canonical "gd:0_Relatives/Diary_02"
2020/09/27 20:35:31 DEBUG : Google drive root '0_Relatives/Diary_02': Waiting for checks to finish
2020/09/27 20:35:31 DEBUG : Google drive root '0_Relatives/Diary_02': Waiting for transfers to finish
2020/09/27 20:35:34 DEBUG : IMG_5838.docx: MD5 = bcb5d124fdfee6866a3ec49a6408535a OK
2020/09/27 20:35:34 INFO  : IMG_5838.docx: Copied (new)
2020/09/27 20:35:37 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:37 DEBUG : pacer: Rate limited, increasing sleep to 1.443210327s
2020/09/27 20:35:37 DEBUG : IMG_5839.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 1/10
2020/09/27 20:35:37 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:37 DEBUG : pacer: Rate limited, increasing sleep to 2.74478956s
2020/09/27 20:35:37 DEBUG : IMG_5840.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 1/10
2020/09/27 20:35:37 DEBUG : pacer: Reducing sleep to 0s
2020/09/27 20:35:40 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:40 DEBUG : pacer: Rate limited, increasing sleep to 1.815101712s
2020/09/27 20:35:40 DEBUG : IMG_5841.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 1/10
2020/09/27 20:35:40 DEBUG : pacer: Reducing sleep to 0s
2020/09/27 20:35:40 INFO  : IMG_5838.odt: Copied (new)
2020/09/27 20:35:43 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:43 DEBUG : pacer: Rate limited, increasing sleep to 1.979492367s
2020/09/27 20:35:43 DEBUG : IMG_5839.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 2/10
2020/09/27 20:35:44 DEBUG : pacer: Reducing sleep to 0s
2020/09/27 20:35:44 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:44 DEBUG : pacer: Rate limited, increasing sleep to 1.001637048s
2020/09/27 20:35:44 DEBUG : IMG_5840.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 2/10
2020/09/27 20:35:46 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:46 DEBUG : pacer: Rate limited, increasing sleep to 2.634015664s
2020/09/27 20:35:46 DEBUG : IMG_5841.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 2/10
2020/09/27 20:35:47 DEBUG : pacer: Reducing sleep to 0s
2020/09/27 20:35:47 DEBUG : pacer: low level retry 1/1 (error googleapi: Error 500: Internal Error, internalError)
2020/09/27 20:35:47 DEBUG : pacer: Rate limited, increasing sleep to 1.661399384s
2020/09/27 20:35:47 DEBUG : IMG_5842.odt: Received error: googleapi: Error 500: Internal Error, internalError - low level retry 1/10
8

500 errors normally indicate something wrong at Google.

I think if you retry it will pick up where it left off.

You could try slowing it down a bit with --tpslimit 5 or smaller

9

About 10 hours later - OK, I will try that - I emptied my Google bin - my storage use went down from 91% full to 11 out of 17GB used. This is what I just tried:

 $ for i in odt/IM*odt ; do echo $i ; /usr/bin/rclone -L -v copy --tpslimit 5 --drive-formats "ods,odt" --drive-import-formats "ods,odt" $i gd:0_Relatives/Diary_02/; sleep 30 ;
done
odt/IMG_5838.odt
2020/09/28 06:46:06 INFO  : Starting HTTP transaction limiter: max 5 transactions/s with burst 1
2020/09/28 06:46:20 INFO  : IMG_5838.odt: Copied (replaced existing)
2020/09/28 06:46:20 INFO  :
Transferred:      656.044k / 656.044 kBytes, 100%, 59.658 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:        13.9s

odt/IMG_5839.odt
2020/09/28 06:46:50 INFO  : Starting HTTP transaction limiter: max 5 transactions/s with burst 1
2020/09/28 06:47:52 INFO  :
Transferred:        6.621M / 6.621 MBytes, 100%, 113.810 kBytes/s, ETA 0s
Transferred:            0 / 1, 0%
Elapsed time:       1m1.7s
Transferring:
 *                                  IMG_5839.odt:100% /677.945k, 0/s, -

2020/09/28 06:47:57 ERROR : IMG_5839.odt: Failed to copy: googleapi: Error 500: Internal Error, internalError
2020/09/28 06:47:57 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 500: Internal Error, internalError
10

@ncw ,

Is it worth doing some sort of more systematic debugging with various sized test files or something? - it seems a long time for there to be a serious Google problem?

11

I think it is probably a rate limit problem but I'm not sure...

If you run with -vv --dump responses rclone will print the body of the 500 error which may have some more info in it.

You could try slowing it down even more with --tpslimit 1 --checkers 1 --transfers 1 also.

12

Result posted in:

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.