`rclone touch s3:bucket/.` creates the object name with the unicode character Fullwidth Full Stop

jamshid · January 22, 2025, 8:54pm

What is the problem you are having with rclone?

Not a big problem but I was trying to create an object named foo/ with a trailing slash and that didn't work. I then tried foo/. and was surprised that did a PUT /foo/%EF%BC%8E. What's that? That's the unicode character “．” U+FF0E Fullwidth Full Stop Unicode Character.

Seems rclone should create a directory-like object with a trailing slash or create an object named with a period (.).

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.1

os/version: darwin 14.7.2 (64 bit)
os/kernel: 23.6.0 (arm64)
os/type: darwin
os/arch: arm64 (ARMv8 compatible)
go/version: go1.23.1
go/linking: dynamic
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3 (others show same issue)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -vv --dump headers touch s3:mahbucke/directorylike/.

The rclone config contents with secrets removed.

[s3]
type = s3
provider = AWS
access_key_id = X
secret_access_key = X

A log from the command with the `-vv` flag

2025/01/22 14:47:22 NOTICE: Automatically setting -vv as --dump is enabled
2025/01/22 14:47:22 DEBUG : rclone: Version "v1.68.1" starting with parameters ["rclone" "-vv" "--dump" "headers" "touch" "s3:mahbucke/directorylike/."]
2025/01/22 14:47:22 DEBUG : Creating backend with remote "s3:mahbucke/directorylike/"
2025/01/22 14:47:22 DEBUG : Using config file from "/Users/jamshidafshar/.rclone.conf"
2025/01/22 14:47:22 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/01/22 14:47:22 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/01/22 14:47:22 DEBUG : fs cache: renaming cache item "s3:mahbucke/directorylike/" to be canonical "s3:mahbucke/directorylike"
2025/01/22 14:47:22 DEBUG : Touch time 2025-01-22 14:47:22.724278 -0600 CST m=+0.164001668
2025/01/22 14:47:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:22 DEBUG : HTTP REQUEST (req 0x140001e68c0)
2025/01/22 14:47:22 DEBUG : HEAD /directorylike/%EF%BC%8E HTTP/1.1
Host: mahbucke.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 04f968a4-1c8e-4815-836d-9625aa0eaea7
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250122T204722Z

2025/01/22 14:47:22 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : HTTP RESPONSE (req 0x140001e68c0)
2025/01/22 14:47:23 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Wed, 22 Jan 2025 20:47:22 GMT
Server: AmazonS3
X-Amz-Id-2: 613cjoAluPkjV/t8MLgziIsM4Vus0BwD60uEL1j8P1jn888HoqdnPo70EjWIsqsD2YE8YAl7RRQ=
X-Amz-Request-Id: G8HYKPZFDYGH64P3


2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : S3 bucket mahbucke path directorylike: Touching (creating) "."
2025/01/22 14:47:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : HTTP REQUEST (req 0x140001e6f00)
2025/01/22 14:47:23 DEBUG : PUT / HTTP/1.1
Host: mahbucke.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.1
Content-Length: 0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: efeaeaea-7dcd-46a6-8967-2115bfac65ec
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250122T204723Z

2025/01/22 14:47:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : HTTP RESPONSE (req 0x140001e6f00)
2025/01/22 14:47:23 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 22 Jan 2025 20:47:24 GMT
Location: /mahbucke
Server: AmazonS3
X-Amz-Id-2: 3gUwu8bmcZ8W4DmxddQQtMdSAKST54QC63UFoy9cOHv4rUW0ZGi5pSnTv3Mh6hD2XQd8gYBVakY=
X-Amz-Request-Id: G8HY3AXHHZ72PKF5

2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 INFO  : S3 bucket mahbucke path directorylike: Bucket "mahbucke" created with ACL ""
2025/01/22 14:47:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : HTTP REQUEST (req 0x14000632500)
2025/01/22 14:47:23 DEBUG : PUT /directorylike/%EF%BC%8E?x-id=PutObject HTTP/1.1
Host: mahbucke.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.1
Content-Length: 0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: ff5cd5a2-1033-44d0-ba62-426005c59008
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20250122T204723Z
X-Amz-Meta-Mtime: 1737578842.724278

2025/01/22 14:47:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : HTTP RESPONSE (req 0x14000632500)
2025/01/22 14:47:23 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 22 Jan 2025 20:47:24 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Server: AmazonS3
X-Amz-Id-2: FzqBF8qWCqaBv78Yh1luKjWDsVlM05At6lXBszBpkQDnjc1IbUV687D0/5YChwQ/CA5k17Emhx4=
X-Amz-Request-Id: G8HZP7Z6VN40EKSM
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: t8Ji0wVffKDY.OLGK3V.LjNb.j4.ngU2

2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : HTTP REQUEST (req 0x1400014f900)
2025/01/22 14:47:23 DEBUG : HEAD /directorylike/%EF%BC%8E HTTP/1.1
Host: mahbucke.s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 80ccfd64-9524-410c-8747-6ef6a896e9cc
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20250122T204723Z

2025/01/22 14:47:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : HTTP RESPONSE (req 0x1400014f900)
2025/01/22 14:47:23 DEBUG : HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Wed, 22 Jan 2025 20:47:24 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Last-Modified: Wed, 22 Jan 2025 20:47:24 GMT
Server: AmazonS3
X-Amz-Id-2: y1nF/6ZZSbBFmlh1NAVGlFq+0HqoiloS+X0t3bHYWe70NBXVt7vU3JgZGWjnN5lVwkrjZ+UW5Wc=
X-Amz-Meta-Mtime: 1737578842.724278
X-Amz-Request-Id: G8HKW3JSAFG1JMT3
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: t8Ji0wVffKDY.OLGK3V.LjNb.j4.ngU2
Content-Length: 0

2025/01/22 14:47:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/01/22 14:47:23 DEBUG : 7 go routines active

asdffdsa · January 22, 2025, 9:04pm

@jamshid, afaik, rclone touch only works on files, not directories.

so it looks like rclone created a file named .
but . is one of the Restricted filename characters
so rclone converts . to U+FF0E

kapitainsky · January 22, 2025, 9:05pm

It is expected and documented behaviour:

jamshid · January 22, 2025, 9:38pm

Others have pointed out that translating a period to the special unicode character is documented behavior.

And I just found the way to create directory-like objects (` --s3-directory-markers):

$ echo hello > ./foo/bar/hello.txt
$ rclone -vv --dump headers copy --s3-directory-markers foo s3:mybucket
...
2025/01/22 21:35:52 DEBUG : PUT /mybucket/bar/ HTTP/1.1
...
2025/01/22 21:35:52 DEBUG : PUT /mybucket/bar/hello.txt HTTP/1.1
...

Ref:

github.com/rclone/rclone

s3: Support empty "directory" keys

opened 08:38PM - 30 Sep 16 UTC

closed 06:15PM - 02 Sep 23 UTC

felixbuenemann

enhancement Remote: S3 help wanted metadata

Many tools like the the AWS S3 Console, S3 network drives / fuse filesystems and… various other tools create "directories" on S3 by creating zero byte keys that end with a slash to emulate empty directories. rclone currently can't handle these keys, because it tries to treat them like a file instead of a directory and fails with an error if a local directory with the same name exist. Instead rclone should detect these keys and treat them like a directory. The keys are easy to detect, because they always end with a slash and have a content-length of 0 bytes. Here's an example key listed with the python aws cli: ``` aws s3api head-object --bucket bito-pim-images --key 'images/WebShop/ita_IT/' { "AcceptRanges": "bytes", "ContentType": "binary/octet-stream", "LastModified": "Fri, 22 Jul 2016 12:32:11 GMT", "ContentLength": 0, "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"", "Metadata": {} } ``` > What is your rclone version (eg output from `rclone -V`) v1.33-DEV > Which OS you are using and how many bits (eg Windows 7, 64 bit) macOS 10.12.1 > Which cloud storage system are you using? (eg Google Drive) s3 > The command you were trying to run (eg `rclone copy /tmp remote:tmp`) `rclone sync --size-only --delete-after remote:tmp /tmp` > A log from the command with the `-v` flag (eg output from `rclone -v copy /tmp remote:tmp`) ``` 2016/09/30 22:09:55 Local file system at /Volumes/EXT4TBA/images: Waiting for transfers to finish 2016/09/30 22:09:55 images/WebShop/ita_IT/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/ita_IT: is a directory 2016/09/30 22:09:55 images/WebShop/nld_BE/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/nld_BE: is a directory 2016/09/30 22:09:55 images/WebShop/nld_NL/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/nld_NL: is a directory 2016/09/30 22:09:55 images/WebShop/pol_PL/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/pol_PL: is a directory 2016/09/30 22:09:56 images/WebShop/slo_SK/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/slo_SK: is a directory 2016/09/30 22:09:56 images/WebShop/spa_ES/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/spa_ES: is a directory 2016/09/30 22:09:56 images/WebShop/ukrainian_ukraine/: Failed to copy: open /Volumes/EXT4TBA/images/WebShop/ukrainian_ukraine: is a directory 2016/09/30 22:09:56 Local file system at /Volumes/EXT4TBA/images: not deleting files as there were IO errors 2016/09/30 22:09:56 Attempt 3/3 failed with 30 errors and: not deleting files as there were IO errors 2016/09/30 22:09:56 Failed to sync: not deleting files as there were IO errors ``` It's very easy to create a test key by going into the AWS S3 Console and clicking on "Create Folder" to create an empty directory.

system · January 25, 2025, 9:39pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.