Rclone sync fails - request wrongly url encoded?

What is the problem you are having with rclone?

rclone sync fails with the error Couldn't delete: Container Not Found
Other commands work correctly, even rclone deletefile when deleting the file that failed during rclone sync

I have an analysis of what seems to be going wrong, the rclone sync command seems to be url encoding the request that fails. When I manually execute the request using curl, the url encoded variant returns a 404 - Not found.

If I manually decode the request then the response is as you would expect.

Request that fails:

HTTP REQUEST (req 0xc0024db300)
2021/03/25 16:17:13 DEBUG : GET /production%255Fencrypted%255Fsegments?format=json&limit=1000&prefix=mariadbservers%2Fweekly%2Fpiwik%255F2021%252D02%252D10.gz.bin%2F1612992682.03079176%2F9221554274 HTTP/1.1

Same request decoded, succeeds:

curl -v -u XXX:XXXX "XXXX.xxx/production_encrypted_segments?format=json&limit=1000&prefix=
mariadbservers%2Fweekly%2Fpiwik%255F2021%252D02%252D10.gz.bin%2F1612992682.03079176%2F9221554274"      

What is your rclone version (output from rclone version)

rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7

But i also tried with a local build from the master branch last week.

Sorry about doing this in multiple replies. The system did not allow me to post in one go due to a restriction on the number of links in a post for new accounts.

Which OS you are using and how many bits (eg Windows 7, 64 bit)

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.7 LTS
Release:        16.04
Codename:       xenial

Which cloud storage system are you using? (eg Google Drive)

A combination of Openstack objectstore (Swift) and the encryption backend.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /mnt/backup/mysql backup_encrypted:mariadbservers --transfers=20 --skip-links --stats 0

The rclone config contents with secrets removed.

[backup]                                                                                                                                                                  
auth = XXX
key = XXXX
tenant = XXX
type = swift                                                                                                                                                                     
user = XXX

[backup_encrypted]                                                                                                                                                        
directory_name_encryption = false
filename_encryption = off
password = XXXXX
password2 = XXXX
remote = backup:/acceptance_encrypted/
type = crypt

A log from the command with the -vv flag

Paste  log here

The full log of the rclone sync command:
https://pastebin.com/edit/pM11bBwS

The log file does not work:

Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.

My mistake, that was the edit url.
This is the correct one:

What it looks like is that rclone failed to URL decode the Manifest in the previous response

X-Object-Manifest: production%5Fencrypted%5Fsegments/mariadbservers/weekly/piwik%5F2021%2D02%2D10.gz.bin/1612992682.03079176/9221554274

And URL encoding that again produces the rather odd

GET /production%255Fencrypted%255Fsegments

Can you try this with the latest beta and if it still doesn't work then please please make a new issue on github and we'll take a look at fixing it - thanks.

Thank you for the response.
The problem and log output are the same for the latest beta version.

Here is the bug i created on github: Rclone sync fails - request wrongly url encoded? · Issue #5180 · rclone/rclone · GitHub

1 Like