- rclone v1.37-157-g5a3a56abβ
- linux/amd64
- Openstack Swift backend
- rclone --dump-headers -vvv copy ./test/ vizlswift:second/
- Description of problem:
Hello
Have some issues and strange behavior of rclone when it copies subfolders.
Maybe this is ordinary way upload files to swift, but another clients like rclone (cyberduck, swftp) has another behavior.
I want to sync my local folder to swift storage with such structure:
$ ls -lR test/
test/:
folder_rclone
test/folder_rclone:
file_rclone
So when I copy it by rclone, it send only 1 PUT request and create object:
PUT /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second/folder_rclone/file_rclone HTTP/1.1
When I copy it by cyberduck, or swftp I see 2 PUT requests:
PUT /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second/folder_ftp HTTP/1.1.
PUT /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second/folder_ftp/file_ftp HTTP/1.1.
As you see, first PUT req creates pseudo-dir, and has headers which determine type of object as Directory.
Content-Length: 0.
Content-Type: application/directory.
In this case, swift clients doesn’t shown hierarchical structure inside my bucket, because rclone doesn’t create empty object with Content-Type: application/directory
Maybe you can notice this feature and change behavior of rclone.
I would be gratefull
- LOG:
`2017/09/22 16:19:28 DEBUG : Using config file from “/home/vizl/.config/rclone/rclone.conf”
2017/09/22 16:19:28 DEBUG : rclone: Version “v1.37-157-g5a3a56abβ” starting with parameters [“rclone” “–dump-headers” “-vv” “copy” “./test/” “vizlswift:second/”]
2017/09/22 16:19:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:28 DEBUG : HTTP REQUEST (req 0xc4200e9500)
2017/09/22 16:19:28 DEBUG : POST /v3/auth/tokens HTTP/1.1
Host: cs.advanced.host
User-Agent: rclone/v1.37-157-g5a3a56abβ
Content-Length: 207
Content-Type: application/json
Accept-Encoding: gzip
2017/09/22 16:19:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:28 DEBUG : HTTP RESPONSE (req 0xc4200e9500)
2017/09/22 16:19:28 DEBUG : HTTP/1.1 201 Created
Content-Length: 1526
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,X-Auth-Token,X-Openstack-Request-Id,X-Subject-Token
Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json
Date: Fri, 22 Sep 2017 13:19:28 GMT
Server: nginx/1.10.2
Vary: X-Auth-Token
X-Openstack-Request-Id: req-7d56dcf7-4906-432b-ac66-ddc479477cfc
X-Subject-Token: gAAAAABZxQ3gMld6R6fqIBAznlzaSi-UoDzeGo14wk5QlfH9DWiOgut3QLqYXcPJlOvYIzrtn3jurN9EVVyUwUH1VgYU_xXtZLEx_oY0Ye3bMxiX9v6NU73f1pdOdeT7fataf22cPHCXlBr-AX4bxOza3Kq4VYRT2UV47Yr4gQbdhp5m0322IpY
2017/09/22 16:19:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:28 INFO : Swift container second: Modify window is 1ns
2017/09/22 16:19:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:28 DEBUG : HTTP REQUEST (req 0xc4200e8c00)
2017/09/22 16:19:28 DEBUG : GET /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second?delimiter=%2F&format=json&limit=1000 HTTP/1.1
Host: cs.advanced.host
User-Agent: rclone/v1.37-157-g5a3a56abβ
X-Auth-Token: gAAAAABZxQ3gMld6R6fqIBAznlzaSi-UoDzeGo14wk5QlfH9DWiOgut3QLqYXcPJlOvYIzrtn3jurN9EVVyUwUH1VgYU_xXtZLEx_oY0Ye3bMxiX9v6NU73f1pdOdeT7fataf22cPHCXlBr-AX4bxOza3Kq4VYRT2UV47Yr4gQbdhp5m0322IpY
Accept-Encoding: gzip
2017/09/22 16:19:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : HTTP RESPONSE (req 0xc4200e8c00)
2017/09/22 16:19:29 DEBUG : HTTP/1.1 200 OK
Content-Length: 197
Accept-Ranges: bytes
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,X-Auth-Token,X-Openstack-Request-Id,X-Container-Meta-type,X-Container-Meta-Temp-URL-Key
Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Container-Meta-Temp-Url-Key,X-Container-Bytes-Used,Last-Modified,X-Timestamp
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Fri, 22 Sep 2017 13:19:29 GMT
Last-Modified: Thu, 07 Sep 2017 11:39:04 GMT
Server: nginx/1.10.2
X-Container-Bytes-Used: 0
X-Container-Meta-Temp-Url-Key: 5de3bcbb66bb48ad50ba23c29daa5a6
X-Container-Object-Count: 9
X-Openstack-Request-Id: tx705b8a377f5144cfb1e8b-0059c50de1
X-Storage-Policy: Policy-0
X-Timestamp: 1504778745.01642
X-Trans-Id: tx705b8a377f5144cfb1e8b-0059c50de1
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 INFO : Swift container second: Waiting for checks to finish
2017/09/22 16:19:29 INFO : Swift container second: Waiting for transfers to finish
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : HTTP REQUEST (req 0xc4200e9100)
2017/09/22 16:19:29 DEBUG : HEAD /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second HTTP/1.1
Host: cs.advanced.host
User-Agent: rclone/v1.37-157-g5a3a56abβ
X-Auth-Token: gAAAAABZxQ3gMld6R6fqIBAznlzaSi-UoDzeGo14wk5QlfH9DWiOgut3QLqYXcPJlOvYIzrtn3jurN9EVVyUwUH1VgYU_xXtZLEx_oY0Ye3bMxiX9v6NU73f1pdOdeT7fataf22cPHCXlBr-AX4bxOza3Kq4VYRT2UV47Yr4gQbdhp5m0322IpY
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : HTTP RESPONSE (req 0xc4200e9100)
2017/09/22 16:19:29 DEBUG : HTTP/1.1 204 No Content
Accept-Ranges: bytes
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,X-Auth-Token,X-Openstack-Request-Id,X-Container-Meta-type,X-Container-Meta-Temp-URL-Key
Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Container-Meta-Temp-Url-Key,X-Container-Bytes-Used,Last-Modified,X-Timestamp
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: keep-alive
Content-Type: text/plain; charset=utf-8
Date: Fri, 22 Sep 2017 13:19:29 GMT
Last-Modified: Thu, 07 Sep 2017 11:39:04 GMT
Server: nginx/1.10.2
X-Container-Bytes-Used: 0
X-Container-Meta-Temp-Url-Key: 5de3bcbb66bb48ad50ba23c29daa5a6
X-Container-Object-Count: 9
X-Openstack-Request-Id: txce9957ed33584bbd87c17-0059c50de1
X-Storage-Policy: Policy-0
X-Timestamp: 1504778745.01642
X-Trans-Id: txce9957ed33584bbd87c17-0059c50de1
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : HTTP REQUEST (req 0xc420616000)
2017/09/22 16:19:29 DEBUG : HEAD /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second/folder_rclone/file_rclone HTTP/1.1
Host: cs.advanced.host
User-Agent: rclone/v1.37-157-g5a3a56abβ
X-Auth-Token: gAAAAABZxQ3gMld6R6fqIBAznlzaSi-UoDzeGo14wk5QlfH9DWiOgut3QLqYXcPJlOvYIzrtn3jurN9EVVyUwUH1VgYU_xXtZLEx_oY0Ye3bMxiX9v6NU73f1pdOdeT7fataf22cPHCXlBr-AX4bxOza3Kq4VYRT2UV47Yr4gQbdhp5m0322IpY
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : HTTP RESPONSE (req 0xc420616000)
2017/09/22 16:19:29 DEBUG : HTTP/1.1 404 Not Found
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,X-Auth-Token,X-Openstack-Request-Id,X-Container-Meta-type,X-Container-Meta-Temp-URL-Key
Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Container-Meta-Temp-Url-Key,X-Container-Bytes-Used,Last-Modified,X-Timestamp
Connection: keep-alive
Content-Type: text/html; charset=UTF-8
Date: Fri, 22 Sep 2017 13:19:29 GMT
Server: nginx/1.10.2
X-Openstack-Request-Id: tx4df0fdae27eb40f9b2425-0059c50de1
X-Trans-Id: tx4df0fdae27eb40f9b2425-0059c50de1
Content-Length: 0
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : HTTP REQUEST (req 0xc4200e8300)
2017/09/22 16:19:29 DEBUG : PUT /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second/folder_rclone/file_rclone HTTP/1.1
Host: cs.advanced.host
User-Agent: rclone/v1.37-157-g5a3a56abβ
Content-Length: 5
Content-Type: application/octet-stream
X-Auth-Token: gAAAAABZxQ3gMld6R6fqIBAznlzaSi-UoDzeGo14wk5QlfH9DWiOgut3QLqYXcPJlOvYIzrtn3jurN9EVVyUwUH1VgYU_xXtZLEx_oY0Ye3bMxiX9v6NU73f1pdOdeT7fataf22cPHCXlBr-AX4bxOza3Kq4VYRT2UV47Yr4gQbdhp5m0322IpY
X-Object-Meta-Mtime: 1506082395.655482519
Accept-Encoding: gzip
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : HTTP RESPONSE (req 0xc4200e8300)
2017/09/22 16:19:29 DEBUG : HTTP/1.1 201 Created
Content-Length: 0
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,X-Auth-Token,X-Openstack-Request-Id,X-Container-Meta-type,X-Container-Meta-Temp-URL-Key
Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Container-Meta-Temp-Url-Key,X-Container-Bytes-Used,Last-Modified,X-Timestamp
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: keep-alive
Content-Type: text/html; charset=UTF-8
Date: Fri, 22 Sep 2017 13:19:29 GMT
Etag: d8e8fca2dc0f896fd7cb4cb0031ba249
Last-Modified: Fri, 22 Sep 2017 13:19:30 GMT
Server: nginx/1.10.2
X-Openstack-Request-Id: tx32e82a5edb7245edb2639-0059c50de1
X-Trans-Id: tx32e82a5edb7245edb2639-0059c50de1
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : HTTP REQUEST (req 0xc4200e8d00)
2017/09/22 16:19:29 DEBUG : HEAD /v1/AUTH_a008883e56004ab891ff1a79dc73a4ef/second/folder_rclone/file_rclone HTTP/1.1
Host: cs.advanced.host
User-Agent: rclone/v1.37-157-g5a3a56abβ
X-Auth-Token: gAAAAABZxQ3gMld6R6fqIBAznlzaSi-UoDzeGo14wk5QlfH9DWiOgut3QLqYXcPJlOvYIzrtn3jurN9EVVyUwUH1VgYU_xXtZLEx_oY0Ye3bMxiX9v6NU73f1pdOdeT7fataf22cPHCXlBr-AX4bxOza3Kq4VYRT2UV47Yr4gQbdhp5m0322IpY
2017/09/22 16:19:29 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 DEBUG : HTTP RESPONSE (req 0xc4200e8d00)
2017/09/22 16:19:29 DEBUG : HTTP/1.1 200 OK
Content-Length: 5
Accept-Ranges: bytes
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,X-Requested-With,X-Auth-Token,X-Openstack-Request-Id,X-Container-Meta-type,X-Container-Meta-Temp-URL-Key
Access-Control-Allow-Methods: HEAD,GET,POST,PUT,PATCH,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: X-Container-Meta-Temp-Url-Key,X-Container-Bytes-Used,Last-Modified,X-Timestamp
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: keep-alive
Content-Type: application/octet-stream
Date: Fri, 22 Sep 2017 13:19:29 GMT
Etag: d8e8fca2dc0f896fd7cb4cb0031ba249
Last-Modified: Fri, 22 Sep 2017 13:19:30 GMT
Server: nginx/1.10.2
X-Object-Meta-Mtime: 1506082395.655482519
X-Openstack-Request-Id: txc7a2aa279d55437289254-0059c50de1
X-Timestamp: 1506086369.64762
X-Trans-Id: txc7a2aa279d55437289254-0059c50de1
2017/09/22 16:19:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2017/09/22 16:19:29 INFO : folder_rclone/file_rclone: Copied (new)
2017/09/22 16:19:29 INFO :
Transferred: 5 Bytes (3 Bytes/s)
Errors: 0
Checks: 0
Transferred: 1
Elapsed time: 1.4s
2017/09/22 16:19:29 DEBUG : Go routines at exit 6
2017/09/22 16:19:29 DEBUG : rclone: Version “v1.37-157-g5a3a56abβ” finishing with parameters [“rclone” “–dump-headers” “-vv” “copy” “./test/” “vizlswift:second/”]`