First let me say that while I am new to rclone, I am very happy with it! Though still trying to get my head around the various filtering options and features.
My question:
rclone is trying to copy files with same name to destination and failing. Is there a way to have rclone overwrite files of the same name only if they are different size and/or date/time stamp?
I am working with rclone v1.53.3 on a Windows/amd64 system running Windows 10, using rclone to copy files to Box.
I am using this command to copy qualifying files from a mapped network drive (Z: in the command example) to a specific path on Box: rclone copy Z:\lab -P --skip-links --max-age "2020-11-22" --max-size 32G --exclude-from exclude-file.txt --log-file log.txt boxconfig:/DestFiles/DestBackups/Destination-1/Lab > rcOut.txt 2>&1
exclued-file.txt contains (each on its own line with no commas): ~$*, *.ini, .DS_St*
As you can see, I am piping stdout and stderr to a file.
Everything appears to be working as expected, but rclone tries (and fails) to overwrite a file on the destination if it already exists. As example, from the log file:
2020/11/25 04:24:40 ERROR : Attempt 3/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/25 04:24:40 Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists```
What I would like to do is set an option in the command that would overwrite files on the destination if the size and/or date/time has changed but if nothing has changed, ignore them.
I've looked at --ignore-existing, but it doesn't seem to allow for the granularity I would like.
Please suggest any available options.
Thanks!
Best to use the help template as a lot of effort went into formatting and collecting the right information up front.
Can you please share:
rclone version output
rclone.conf with passwords and keys removed
Run the command with -vv and share the full output of the log and just test with one problematic file.
A copy normally updates the remote on the other side or skips a file if they are the same size/date.
Sorry... I guess I was trying to simplify. I'll use the template from now on.
rclone version output:
rclone v1.53.3
os/arch: windows/amd64
go version: go1.15.5
I'm not entirely sure this is what you need in the way of rclone.conf, but this is the contents of my rclone.conf file with the access_token and refresh_token replaced by useless strings.
[boxconfig]
type = box
box_sub_type = user
token = {"access_token":"secretaccesstoken","token_type":"bearer","refresh_token":"secretrefreshtoken","expiry":"2020-11-25T04:37:10.0528737-08:00"}
Hmm, this isn't normal behaviour. Rclone should be able to write over files that exist just fine.
When you say same name, what do you mean? On the source and destination or something different?
I think that error could be caused by a sync which had both FILE.txt and file.txt in the same directory. I don't know whether your Z: drive allows that.
If you could attach a complete log with -vv that would be really helpful. The other things @Animosity022 asked for would be very useful too.
I don't think that is the same issue though because the command posted by @copyier doesn't appear to have any flags which would cause a server side copy or a server side move which is what that issue is about.
@ncw, By "same name", I mean that the file named Case 4.cas in the source directory is in the target directory (on Box) with the same name; except that Case 4.cas in the source directory is dated 11/24/2020 and in the target directory is dated 11/3/2020.
However, now that I've looked closer, the file in the source directory has a lowercased extension (.cas) whereas the file in the target directory has a proper-cased extension (.Cas). Not sure how rclone handles a situation like that...
I hope this information is what you are looking for, concerning rclone configuration and the log file:
rclone version:
rclone v1.53.3
os/arch: windows/amd64
go version: go1.15.5
rclone config show (with access tokens replaced):
[boxconfig]
type = box
box_sub_type = user
token = {"access_token":"secretaccesstoken","token_type":"bearer","refresh_token":"secretrefreshtoken","expiry":"2020-11-25T04:37:10.0528737-08:00"}
Log output:
2020/11/27 10:43:51 DEBUG : --max-age 5.780462848988426d to 2020-11-21 16:00:00 -0800 PST m=-499431.890436299
2020/11/27 10:43:51 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "Z:\lab\imageuser\aScratch_Folder\Ferdi_Lab\IRW_DB\Sadaf\Doses\89Zr\Case 4.cas" "-P" "-vv" "--skip-links" "--max-age" "2020-11-22" "--max-size" "32G" "--exclude-from" "exclude-file.txt" "--log-file" "log.txt" "boxconfig:/MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"]
2020/11/27 10:43:51 DEBUG : Creating backend with remote "Z:\lab\imageuser\aScratch_Folder\Ferdi_Lab\IRW_DB\Sadaf\Doses\89Zr\Case 4.cas"
2020/11/27 10:43:51 DEBUG : Using config file from "C:\Users\maxtm\.config\rclone\rclone.conf"
2020/11/27 10:43:52 DEBUG : fs cache: adding new entry for parent of "Z:\lab\imageuser\aScratch_Folder\Ferdi_Lab\IRW_DB\Sadaf\Doses\89Zr\Case 4.cas", "//?/Z:/lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"
2020/11/27 10:43:52 DEBUG : Creating backend with remote "boxconfig:/MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"
2020/11/27 10:43:52 DEBUG : boxconfig: Loaded invalid token from config file - ignoring
2020/11/27 10:43:52 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020/11/27 10:43:52 DEBUG : boxconfig: Saved new token in config file
2020/11/27 10:43:56 DEBUG : fs cache: renaming cache item "boxconfig:/MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr" to be canonical "boxconfig:MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"
2020/11/27 10:43:56 DEBUG : Case 4.cas: Need to transfer - File not found at Destination
2020/11/27 10:43:58 ERROR : Case 4.cas: Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/27 10:43:58 ERROR : Attempt 1/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/27 10:43:58 DEBUG : Case 4.cas: Need to transfer - File not found at Destination
2020/11/27 10:43:59 ERROR : Case 4.cas: Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/27 10:43:59 ERROR : Attempt 2/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/27 10:44:00 DEBUG : Case 4.cas: Need to transfer - File not found at Destination
2020/11/27 10:44:01 ERROR : Case 4.cas: Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/27 10:44:01 ERROR : Attempt 3/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/27 10:44:01 INFO :
Transferred: 4.151k / 4.151 kBytes, 100%, 990 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 9.5s
2020/11/27 10:44:01 DEBUG : 9 go routines active
2020/11/27 10:44:01 Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:40 DEBUG : rclone: Version "v1.54.0-beta.4929.b31fec4b0.finalise" starting with parameters ["rclone" "copyto" "-vv" "/tmp/hello.txt" "boxcom:file1.txt"]
2020/11/28 12:54:40 DEBUG : Creating backend with remote "/tmp/hello.txt"
2020/11/28 12:54:40 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2020/11/28 12:54:40 DEBUG : fs cache: adding new entry for parent of "/tmp/hello.txt", "/tmp"
2020/11/28 12:54:40 DEBUG : Creating backend with remote "boxcom:"
2020/11/28 12:54:40 DEBUG : hello.txt: Need to transfer - File not found at Destination
2020/11/28 12:54:42 DEBUG : hello.txt: SHA-1 = f572d396fae9206628714fb2ce00f72e94f2258f OK
2020/11/28 12:54:42 INFO : hello.txt: Copied (new) to: file1.txt
2020/11/28 12:54:42 INFO :
Transferred: 6 / 6 Bytes, 100%, 3 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 2.3s
2020/11/28 12:54:42 DEBUG : 8 go routines active
ncw@dogger:~$ rclone copyto -vv /tmp/hello.txt boxcom:FILE1.txt
2020/11/28 12:54:48 DEBUG : rclone: Version "v1.54.0-beta.4929.b31fec4b0.finalise" starting with parameters ["rclone" "copyto" "-vv" "/tmp/hello.txt" "boxcom:FILE1.txt"]
2020/11/28 12:54:48 DEBUG : Creating backend with remote "/tmp/hello.txt"
2020/11/28 12:54:48 DEBUG : Using config file from "/home/ncw/.rclone.conf"
2020/11/28 12:54:48 DEBUG : fs cache: adding new entry for parent of "/tmp/hello.txt", "/tmp"
2020/11/28 12:54:48 DEBUG : Creating backend with remote "boxcom:"
2020/11/28 12:54:49 DEBUG : hello.txt: Need to transfer - File not found at Destination
2020/11/28 12:54:51 ERROR : hello.txt: Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:51 ERROR : Attempt 1/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:51 DEBUG : hello.txt: Need to transfer - File not found at Destination
2020/11/28 12:54:52 ERROR : hello.txt: Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:52 ERROR : Attempt 2/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:53 DEBUG : hello.txt: Need to transfer - File not found at Destination
2020/11/28 12:54:54 ERROR : hello.txt: Failed to copy: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:54 ERROR : Attempt 3/3 failed with 1 errors and: Error "item_name_in_use" (409): Item with the same name already exists
2020/11/28 12:54:54 INFO :
Transferred: 18 / 18 Bytes, 100%, 3 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 5.7s
2020/11/28 12:54:54 DEBUG : 8 go routines active
2020/11/28 12:54:54 Failed to copyto: Error "item_name_in_use" (409): Item with the same name already exists
So what is happening here is that rclone is failing to notice it is updating a file and it uploads a new one.
This only fails in the specific case of mentioning a file on the command line, if you sync the directory above it should work I think.
It should also work if you use the same case for the file name that you used - can you try that?
I had a look at fixing this and it is harder than I thought!
@ncw changing the file extension (.cas -> .Cas) appears to have solved the immediate problem:
2020/11/30 07:47:26 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "copy" "Z:\\lab\\imageuser\\aScratch_Folder\\Ferdi_Lab\\IRW_DB\\Sadaf\\Doses\\89Zr\\Case 4.Cas" "-P" "-vv" "--skip-links" "--max-age" "2020-11-22" "--max-size" "32G" "--exclude-from" "exclude-file.txt" "--log-file" "log.txt" "boxconfig:/MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"]
2020/11/30 07:47:26 DEBUG : Creating backend with remote "Z:\\lab\\imageuser\\aScratch_Folder\\Ferdi_Lab\\IRW_DB\\Sadaf\\Doses\\89Zr\\Case 4.Cas"
2020/11/30 07:47:26 DEBUG : Using config file from "C:\\Users\\maxtm\\.config\\rclone\\rclone.conf"
2020/11/30 07:47:26 DEBUG : fs cache: adding new entry for parent of "Z:\\lab\\imageuser\\aScratch_Folder\\Ferdi_Lab\\IRW_DB\\Sadaf\\Doses\\89Zr\\Case 4.Cas", "//?/Z:/lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"
2020/11/30 07:47:26 DEBUG : Creating backend with remote "boxconfig:/MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"
2020/11/30 07:47:26 DEBUG : boxconfig: Loaded invalid token from config file - ignoring
2020/11/30 07:47:26 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020/11/30 07:47:26 DEBUG : boxconfig: Saved new token in config file
2020/11/30 07:47:30 DEBUG : fs cache: renaming cache item "boxconfig:/MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr" to be canonical "boxconfig:MyFiles/CustBackups/Cust-1/Lab/imageuser/aScratch_Folder/Ferdi_Lab/IRW_DB/Sadaf/Doses/89Zr"
2020/11/30 07:47:30 DEBUG : Case 4.Cas: Sizes differ (src 1417 vs dst 1416)
2020/11/30 07:47:32 DEBUG : Case 4.Cas: SHA-1 = 57a97671b46f19708c64de179e270e3611c7c9ca OK
2020/11/30 07:47:32 INFO : Case 4.Cas: Copied (replaced existing)
2020/11/30 07:47:32 INFO :
Transferred: 1.384k / 1.384 kBytes, 100%, 781 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 6.6s
2020/11/30 07:47:32 DEBUG : 9 go routines active
Now my issue is that I don't know what to do if/when a similar condition is encountered. These files are created by a test device, AFAIK, and I don't think the user has any influence on the file name. It could be that the code driving the test device is what causes the file extension differences (odd, but I suppose possible).
Do you have any recommendations on how I can automate catching and handling these differences, using rclone?
As a side note, I have been using python and python's ftplib to copy these files up to Box, but that method has proven unstable as it seems Box does not like the way I'm doing it.
The first step to that would be please make a new issue on github - can you link this forum post from the issue please?
The first part of the fix is easy - getting box to find the file if you mention it on the command line with a case insensitive compare.
The second part is harder though as copying a file on the command line internally uses the --files-from mechanism, and that is case sensitive at the moment.
The other way which will work is if you don't mention the file on the command line, so insteady of doing