What is the problem you are having with rclone?
rclone will not delete files from an encrypted swift backend
What is your rclone version (output from rclone version
)
rclone v1.55.1
os/type: linux
os/arch: amd64
go/version: go1.16.3
go/linking: static
go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 20.04 64bit
Which cloud storage system are you using? (eg Google Drive)
cloud-a (swift)
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone delete -vvv secret:current/[...redacted...]/Data.xml
The rclone config contents with secrets removed.
cat ~/.config/rclone/rclone.conf
[clouda]
type = swift
user = XXXXXXX
key = XXXXXXX
auth = https://ca-ns-1.bulkstorage.ca:8444/keys_auth/CurrentData
tenant = XXXXXXXXX
region = regionOne
auth_version = 2
endpoint_type = public
[secret]
type = crypt
remote = clouda:CurrentData
password = XXXX
A log from the command with the -vv
flag
2021/06/01 06:19:46 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 06:19:46 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "delete" "-vvv" "secret:current/[...redacted...]/Data.xml"]
2021/06/01 06:19:46 DEBUG : Creating backend with remote "secret:current/[...redacted...]Data.xml"
2021/06/01 06:19:46 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k"
2021/06/01 06:19:47 DEBUG : Waiting for deletions to finish
2021/06/01 06:19:48 DEBUG : 4 go routines active
Result:
rclone -vvv lsl secret:current/[...redacted...]
2021/06/01 06:22:02 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 06:22:02 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "lsl" "secret:current/[...redacted...]/"]
2021/06/01 06:22:02 DEBUG : Creating backend with remote "secret:current/[...redacted...]/"
2021/06/01 06:22:02 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk"
2021/06/01 06:22:05 DEBUG : sqnu1npjc2ojm9o1qjqnatj09k: Failed to read metadata: object not found
5007 2021-04-10 01:29:30.000000000 Data.xml
2021/06/01 06:22:05 DEBUG : 2 go routines active
asdffdsa
(jojothehumanmonkey)
June 1, 2021, 1:42pm
2
hello and welcome to the forum,
from clouda:
, can you delete the corresponding file from inside secret:
?
hello and thank you.
using delete
rclone -vvv delete clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k
2021/06/01 07:10:57 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 07:10:57 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "delete" "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k"]
2021/06/01 07:10:57 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k"
2021/06/01 07:10:59 DEBUG : Waiting for deletions to finish
2021/06/01 07:10:59 DEBUG : 2 go routines active
using deletefile
rclone -vvv deletefile clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k
2021/06/01 07:12:20 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 07:12:20 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "deletefile" "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k"]
2021/06/01 07:12:20 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k"
2021/06/01 07:12:21 ERROR : Attempt 1/3 failed with 1 errors and: clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k is a directory or doesn't exist
and the file is still there:
rclone -vvv lsl clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/
2021/06/01 07:13:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 07:13:13 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "lsl" "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/"]
2021/06/01 07:13:13 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/"
2021/06/01 07:13:14 DEBUG : fs cache: renaming cache item "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk/" to be canonical "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/jjskt1li4g8qbggr8f3nu0pg58/3m4r3llpfmu46omq0fdq0g096s/8599v9ugdvdjsr71etg36vbl3c/rfl0k38ve5ad9jgttl7mgt9odg/vtnu4tld71k0ruhgamd8vd6jao/1f1fm6aivbedm2tp20ms7d95l8/jpp4pmodv15ji7ac2tvujtcrfc/upkjdla6149cr98picqg18aqlk"
2021/06/01 07:13:15 DEBUG : sqnu1npjc2ojm9o1qjqnatj09k: Failed to read metadata: object not found
5055 2021-04-10 01:29:30.000000000 sqnu1npjc2ojm9o1qjqnatj09k
2021/06/01 07:13:15 DEBUG : 2 go routines active
I am very curious about the debug "failed to read metadata: object not found" statement
asdffdsa
(jojothehumanmonkey)
June 1, 2021, 2:29pm
4
to simplify this, as the crypted file names get confusing.
make a new folder in clouda:
, copy a file into it and then try to delete that file.
I can copy and delete files in clouda: .
The issue with deletion applies to a specific set of files only, so I hope you can forgive my truncation. No special characters in path/file names.
These files are spread out over a number of folders, and stand out as the only ones with modified times within the last month or two, all others being roughly the same time in 2019. Other archives have the original modify time for these files.
I cannot download the file from the encrypted backend to compare. When I try to rclone secret:/.../Data.xml file:
2021/06/01 09:39:01 INFO : There was nothing to transfer
The file is "there"
$ sudo rclone -vvv lsl secret:/current/.../Section60/
2021/06/01 09:05:28 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../upkjdla6149cr98picqg18aqlk"
2021/06/01 09:05:31 DEBUG : sqnu1npjc2ojm9o1qjqnatj09k: Failed to read metadata: object not found
5007 2021-04-10 01:29:30.000000000 Data.xml
2021/06/01 09:05:31 DEBUG : 2 go routines active
but cannot be deleted
$ sudo rclone -vvv deletefile secret:/current/.../Section60/Data.xml
2021/06/01 09:06:02 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 09:06:02 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "deletefile" "secret:/current/.../Section60/Data.xml"]
2021/06/01 09:06:02 DEBUG : Creating backend with remote "secret:/current/.../Section60/Data.xml"
2021/06/01 09:06:02 DEBUG : Creating backend with remote "clouda:CurrentData/.../upkjdla6149cr98picqg18aqlk/sqnu1npjc2ojm9o1qjqnatj09k"
2021/06/01 09:06:03 ERROR : Attempt 1/3 failed with 1 errors and: secret:/current/.../Section60/Data.xml is a directory or doesn't exist
...
2021/06/01 09:06:03 DEBUG : 2 go routines active
2021/06/01 09:06:03 Failed to deletefile: secret:/current/.../Section60/Data.xml is a directory or doesn't exist
Looking at the encrypted and obfuscated files, rclone delete ...
2021/06/01 09:33:54 Failed to deletefile: clouda:CurrentData/.../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k is a directory or doesn't exist
and rclone lsl ...
2021/06/01 09:47:25 DEBUG : sqnu1npjc2ojm9o1qjqnatj09k: Failed to read metadata: object not found
5020 2021-04-10 01:19:47.000000000 sqnu1npjc2ojm9o1qjqnatj09k
the file is there but not there
asdffdsa
(jojothehumanmonkey)
June 1, 2021, 5:33pm
6
so this only happens with files inside secret:
?
have you been using another software package that might have modified/accesses these files?
for the rclone deletefile
command, add the following and post it
--dump=bodies --retries=1 --low-level-retries=1
No, as shown in the last two snippets, I cannot delete from clouda: either.
*To clarify, secret is an encrypted backend for clouda
$ sudo rclone -vvv lsl secret:/current/.../Section10/
2021/06/01 11:40:11 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 11:40:11 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "lsl" "secret:/current/.../Section10/"]
2021/06/01 11:40:11 DEBUG : Creating backend with remote "secret:/current/.../Section10/"
2021/06/01 11:40:11 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4"
2021/06/01 11:40:14 DEBUG : sqnu1npjc2ojm9o1qjqnatj09k: Failed to read metadata: object not found
4972 2021-04-10 01:19:47.000000000 Data.xml
2021/06/01 11:40:14 DEBUG : 2 go routines active
$ sudo rclone -vvv --dump=bodies --retries=1 --low-level-retries=1 deletefile secret:/current/.../Section10/Data.xml
2021/06/01 11:40:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 11:40:36 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "--dump=bodies" "--retries=1" "--low-level-retries=1" "deletefile" "secret:/current/.../Section10/Data.xml"]
2021/06/01 11:40:36 DEBUG : Creating backend with remote "secret:/current/.../Section10/Data.xml"
2021/06/01 11:40:36 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k"
2021/06/01 11:40:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/06/01 11:40:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 11:40:36 DEBUG : HTTP REQUEST (req 0xc00051e500)
2021/06/01 11:40:36 DEBUG : POST /keys_auth/CurrentData/tokens HTTP/1.1
Host: ca-ns-1.bulkstorage.ca:8444
User-Agent: rclone/v1.55.1
Content-Length: 170
Content-Type: application/json
Accept-Encoding: gzip
{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"xxxx","apiKey":"xxxx"},"tenantName":"xxx"}}
2021/06/01 11:40:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 11:40:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 11:40:36 DEBUG : HTTP RESPONSE (req 0xc00051e500)
2021/06/01 11:40:36 DEBUG : HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 30
Content-Type: text/html; charset=UTF-8
Date: Tue, 01 Jun 2021 18:40:36 GMT
X-Trans-Id: txcb7a3b78178947428e0b5-0060b67f24
401 Unauthorized: Bad request
2021/06/01 11:40:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 11:40:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 11:40:36 DEBUG : HTTP REQUEST (req 0xc00051eb00)
2021/06/01 11:40:36 DEBUG : POST /keys_auth/CurrentData/tokens HTTP/1.1
Host: ca-ns-1.bulkstorage.ca:8444
User-Agent: rclone/v1.55.1
Content-Length: 164
Content-Type: application/json
Accept-Encoding: gzip
{"auth":{"passwordCredentials":{"username":"xxxx","password":"xxxx"},"tenantName":"xxxx"}}
2021/06/01 11:40:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 11:40:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 11:40:37 DEBUG : HTTP RESPONSE (req 0xc00051eb00)
2021/06/01 11:40:37 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 828
Content-Type: application/json
Date: Tue, 01 Jun 2021 18:40:37 GMT
X-Trans-Id: txf41f96e077a8448594c73-0060b67f25
{"access": {"token": {"issued_at": "2021-06-01T18:40:37.503848", "expires": "2021-06-02T18:40:37.503754", "id": "{\"key_type\": \"Full-Key\", \"container_key\": \"xxxx\", \"container_name\": \"CurrentData\"}", "tenant": {"enabled": true, "description": null, "name": "xxxx", "id": "xxxx"}}, "serviceCatalog": [{"endpoints_links": [], "endpoints": [{"region": "regionOne", "id": "xxxx", "publicURL": "https://ca-ns-1.bulkstorage.ca:8444/v1/AUTH_xxxx"}], "type": "object-store", "name": "swift"}], "user": {"username": "xxxx", "roles_links": [], "id": "xxxx", "roles": [], "name": "xxxx"}}}
2021/06/01 11:40:37 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 11:40:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 11:40:37 DEBUG : HTTP REQUEST (req 0xc0003f3500)
2021/06/01 11:40:37 DEBUG : HEAD /v1/AUTH_xxxx/CurrentData/3rvhq97hv8butfdprv0r7btcik/..../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k HTTP/1.1
Host: ca-ns-1.bulkstorage.ca:8444
User-Agent: rclone/v1.55.1
Transfer-Encoding: chunked
X-Auth-Token: XXXX
0
2021/06/01 11:40:37 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 11:40:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 11:40:38 DEBUG : HTTP RESPONSE (req 0xc0003f3500)
2021/06/01 11:40:38 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: text/html; charset=UTF-8
Date: Tue, 01 Jun 2021 18:40:37 GMT
X-Trans-Id: tx004f1900ea6649519b598-0060b67f25
Content-Length: 0
2021/06/01 11:40:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 11:40:38 ERROR : Attempt 1/1 failed with 1 errors and: secret:/current/.../Section10/Data.xml is a directory or doesn't exist
2021/06/01 11:40:38 DEBUG : 2 go routines active
2021/06/01 11:40:38 Failed to deletefile: secret:/current/..../Section10/Data.xml is a directory or doesn't exist
asdffdsa
(jojothehumanmonkey)
June 1, 2021, 6:55pm
8
let's not test with secret:
, it just make this all the more complex and confusing.
can you repost the commands using clouda
with
-vv --dump=bodies --retries=1 --low-level-retries=1
confirming it is there
$ sudo rclone -vvv lsl clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/
2021/06/01 11:58:51 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 11:58:51 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "lsl" "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/"]
2021/06/01 11:58:51 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/"
2021/06/01 11:58:53 DEBUG : fs cache: renaming cache item "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/../tku5qmqlqc25d3epefasklp8g4/" to be canonical "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/../tku5qmqlqc25d3epefasklp8g4"
2021/06/01 11:58:54 DEBUG : sqnu1npjc2ojm9o1qjqnatj09k: Failed to read metadata: object not found
5020 2021-04-10 01:19:47.000000000 sqnu1npjc2ojm9o1qjqnatj09k
2021/06/01 11:58:54 DEBUG : 2 go routines active
rclone deletefile
$ sudo rclone -vvv deletefile --dump=bodies --retries=1 --low-level-retries=1 clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k
2021/06/01 12:02:44 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/06/01 12:02:44 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "-vvv" "deletefile" "--dump=bodies" "--retries=1" "--low-level-retries=1" "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k"]
2021/06/01 12:02:44 DEBUG : Creating backend with remote "clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k"
2021/06/01 12:02:44 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/06/01 12:02:44 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 12:02:44 DEBUG : HTTP REQUEST (req 0xc00065db00)
2021/06/01 12:02:44 DEBUG : POST /keys_auth/CurrentData/tokens HTTP/1.1
Host: ca-ns-1.bulkstorage.ca:8444
User-Agent: rclone/v1.55.1
Content-Length: 170
Content-Type: application/json
Accept-Encoding: gzip
{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"xxxx","apiKey":"xxxx"},"tenantName":"xxxx"}}
2021/06/01 12:02:44 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 12:02:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 12:02:45 DEBUG : HTTP RESPONSE (req 0xc00065db00)
2021/06/01 12:02:45 DEBUG : HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 30
Content-Type: text/html; charset=UTF-8
Date: Tue, 01 Jun 2021 19:02:44 GMT
X-Trans-Id: txaf6f0163b39b4ce2a7a9b-0060b68454
401 Unauthorized: Bad request
2021/06/01 12:02:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 12:02:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 12:02:45 DEBUG : HTTP REQUEST (req 0xc00065c700)
2021/06/01 12:02:45 DEBUG : POST /keys_auth/CurrentData/tokens HTTP/1.1
Host: ca-ns-1.bulkstorage.ca:8444
User-Agent: rclone/v1.55.1
Content-Length: 164
Content-Type: application/json
Accept-Encoding: gzip
{"auth":{"passwordCredentials":{"username":"xxxx","password":"xxxx"},"tenantName":"xxxx"}}
2021/06/01 12:02:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 12:02:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 12:02:45 DEBUG : HTTP RESPONSE (req 0xc00065c700)
2021/06/01 12:02:45 DEBUG : HTTP/1.1 200 OK
Connection: close
Content-Length: 828
Content-Type: application/json
Date: Tue, 01 Jun 2021 19:02:45 GMT
X-Trans-Id: tx7d0089fe69d64092b2640-0060b68455
{"access": {"token": {"issued_at": "2021-06-01T19:02:45.421862", "expires": "2021-06-02T19:02:45.421846", "id": "{\"key_type\": \"Full-Key\", \"container_key\": \"xxxx\", \"container_name\": \"CurrentData\"}", "tenant": {"enabled": true, "description": null, "name": "xxxx", "id": "xxxx"}}, "serviceCatalog": [{"endpoints_links": [], "endpoints": [{"region": "regionOne", "id": "xxxx", "publicURL": "https://ca-ns-1.bulkstorage.ca:8444/v1/AUTH_xxxx"}], "type": "object-store", "name": "swift"}], "user": {"username": "xxxx", "roles_links": [], "id": "xxxx", "roles": [], "name": "xxxx"}}}
2021/06/01 12:02:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 12:02:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 12:02:45 DEBUG : HTTP REQUEST (req 0xc00065ce00)
2021/06/01 12:02:45 DEBUG : HEAD /v1/AUTH_xxxx/CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k HTTP/1.1
Host: ca-ns-1.bulkstorage.ca:8444
User-Agent: rclone/v1.55.1
Transfer-Encoding: chunked
X-Auth-Token: XXXX
0
2021/06/01 12:02:45 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/06/01 12:02:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 12:02:45 DEBUG : HTTP RESPONSE (req 0xc00065ce00)
2021/06/01 12:02:45 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: text/html; charset=UTF-8
Date: Tue, 01 Jun 2021 19:02:45 GMT
X-Trans-Id: tx6f93134c8ae74437a6e7c-0060b68455
Content-Length: 0
2021/06/01 12:02:45 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/06/01 12:02:45 ERROR : Attempt 1/1 failed with 1 errors and: clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k is a directory or doesn't exist
2021/06/01 12:02:45 DEBUG : 2 go routines active
2021/06/01 12:02:45 Failed to deletefile: clouda:CurrentData/3rvhq97hv8butfdprv0r7btcik/.../tku5qmqlqc25d3epefasklp8g4/sqnu1npjc2ojm9o1qjqnatj09k is a directory or doesn't exist
Okay, well, I did successfully rclone copy a version of the file in question back to the encrypted backend to overwrite the 'file', and was then able to delete it successfully.
rclone lsl no longer shows the file.
I don't know if that helps at all with diagnosis. I am still wondering, what happened, and how do I prevent this from happening in the future?
I did find this in the logs,
rclone_2021-04-08_00-00-05.log:2021/04/08 06:25:23 INFO : .../Section60/Data.xml: Copied (server side copy)
rclone_2021-04-08_00-00-05.log:2021/04/08 06:25:24 INFO : .../Section60/Data.xml: Deleted
rclone_2021-04-08_00-00-05.log:2021/04/08 06:25:24 INFO : .../Section60/Data.xml: Moved into backup dir
rclone_2021-04-10_00-00-05.log:2021/04/10 01:29:31 INFO : .../Section60/Data.xml: Copied (new)
so there was definitely an intentional change to the file; why, I don't know and am not sure it matters right now. I have a work-around, but if this happened because I didsomething, I would like to avoid doing it again in the future.
If it might help, these, and also some logging, are the parameters used:
--transfers 16
--maxsize 1G
--retries 8
--retrytime 60s
--fast-list
--copy-links
--b2-hard-delete
--backup-dir
asdffdsa
(jojothehumanmonkey)
June 1, 2021, 11:33pm
12
kingwarrick:
--b2-hard-delete
that flag is for b2, not swift
Error still arises even with the flag removed (missed it in the script originally).
I suspect it may be an issue with special characters in file/folder names (!&,% etc).
I've taken out the b2-hard-delete flag, re-copied the files causing the errors, and will have to wait for the next sync to see.
system
(system)
Closed
August 2, 2021, 11:17am
14
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.