`rclone touch --no-check-dest --s3-no-check-bucket --s3-no-head` still does a HEAD

jamshid · February 29, 2024, 9:12pm

What is the problem you are having with rclone?

rclone touch always does a HEAD of the object even when using --no-check-dest and --s3-no-head.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.1
- os/version: darwin 14.3.1 (64 bit)
- os/kernel: 23.3.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.21.5
- go/linking: dynamic
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone -vv --dump=headers touch --no-check-dest --s3-no-check-bucket --s3-no-head --http-no-head s3:mahbucke/why

The rclone config contents with secrets removed.

[s3]
type = s3
access_key_id = X
secret_access_key = y
# Adding :443 used to break V4 sig with AWS but now rclone seems to drop the port in Host if it's default.
#endpoint = http://s3.amazonaws.com
part_size = 5000000

A log from the command with the `-vv` flag

[corrected]

2024/02/29 15:41:46 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "-vv" "--dump=headers" "touch" "--no-check-dest" "--s3-no-check-bucket" "--s3-no-head" "--http-no-head" "s3:mahbucke/why"]
2024/02/29 15:41:46 DEBUG : Creating backend with remote "s3:mahbucke/"
2024/02/29 15:41:46 DEBUG : Using config file from "/Users/jafshar/.rclone.conf"
2024/02/29 15:41:46 DEBUG : s3: detected overridden config - adding "{Xcof1}" suffix to name
2024/02/29 15:41:46 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/02/29 15:41:46 NOTICE: s3: s3 provider "" not known - please set correctly
2024/02/29 15:41:46 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/02/29 15:41:46 DEBUG : fs cache: renaming cache item "s3:mahbucke/" to be canonical "s3{Xcof1}:mahbucke"
2024/02/29 15:41:46 DEBUG : Touch time 2024-02-29 15:41:46.463672 -0600 CST m=+0.235979986
2024/02/29 15:41:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/02/29 15:41:46 DEBUG : HTTP REQUEST (req 0xc000730400)
2024/02/29 15:41:46 DEBUG : HEAD /mahbucke/why HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.65.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240229T214146Z

2024/02/29 15:41:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/02/29 15:41:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/02/29 15:41:46 DEBUG : HTTP RESPONSE (req 0xc000730400)
2024/02/29 15:41:46 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 29 Feb 2024 21:41:46 GMT
Server: AmazonS3
X-Amz-Id-2: Yj4P2Hn44teQ+UTtutVCVqj5YmQ0Ope5Z4QM2sFQsI7ll6puwdB6K6c+LVclPFQqluDZ+Y/+IE4=
X-Amz-Request-Id: T66W37CE6MKGKM9F

2024/02/29 15:41:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/02/29 15:41:46 DEBUG : S3 bucket mahbucke: Touching (creating) "why"
2024/02/29 15:41:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/02/29 15:41:46 DEBUG : HTTP REQUEST (req 0xc000a22300)
2024/02/29 15:41:46 DEBUG : PUT /mahbucke/why HTTP/1.1
Host: s3.us-east-1.amazonaws.com
User-Agent: rclone/v1.65.1
Content-Length: 0
Authorization: XXXX
Content-Md5: 1B2M2Y8AsgTpgAmY7PhCfg==
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20240229T214146Z
X-Amz-Meta-Mtime: 1709242906.463672
Accept-Encoding: gzip

2024/02/29 15:41:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/02/29 15:41:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/02/29 15:41:46 DEBUG : HTTP RESPONSE (req 0xc000a22300)
2024/02/29 15:41:46 DEBUG : HTTP/1.1 200 OK
Content-Length: 0
Date: Thu, 29 Feb 2024 21:41:47 GMT
Etag: "d41d8cd98f00b204e9800998ecf8427e"
Server: AmazonS3
X-Amz-Id-2: 83vfsNuA7llSWiWqIgdnoKSqkR1v4BLaR4ghMn1XA4go8JdhFAytMhLZPAD+X9HrzOiz268Sg8g=
X-Amz-Request-Id: T66WR0R3KYD4M20J
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: 539XtfAmDDYq6ESubNknFqvOrfM7t7fF

2024/02/29 15:41:46 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/02/29 15:41:46 DEBUG : 7 go routines active

asdffdsa · February 29, 2024, 9:21pm

hi,
try --s3-no-head-object

fwiw, i use --dump=headers --retries=1
helps keep the log smaller.

and what about this error?
NoSuchBucket: The specified bucket does not exist

asdffdsa · February 29, 2024, 9:35pm

i just did a quick test, --s3-no-head-object may not make a difference.

jamshid · February 29, 2024, 9:45pm

Sorry I typo'd the bucket. I redid the command and output with a valid bucket, but same behavior.

I can't seem to make rclone avoid the HEAD before the PUT. None of the below flags work with "touch".

Seems --no-check-dst should but maybe it only applies to copy.

  --no-check-dest                               Don't check the destination, copy regardless
  --http-no-head                                        Don't use HEAD requests
  --s3-no-check-bucket                                  If set, don't attempt to check the bucket exists or create it
  --s3-no-head                                          If set, don't HEAD uploaded objects to check integrity
  --s3-no-head-object                                   If set, do not do HEAD before GET when getting objects

asdffdsa · February 29, 2024, 10:03pm

i did more testing, tried additional flags, did not a difference.

as far as i can tell, none of those flags would apply to touch

imho, not seeing this as a bug, but a possible feature request.

jamshid · February 29, 2024, 10:15pm

Ok thanks I filed:

github.com/rclone/rclone

rclone touch always does a HEAD of the object even when using --no-check-dest

opened 10:15PM - 29 Feb 24 UTC

jamshid

#### The associated forum post URL from `https://forum.rclone.org` https://fo…rum.rclone.org/t/rclone-touch-no-check-dest-s3-no-check-bucket-s3-no-head-still-does-a-head/44859 #### What is the problem you are having with rclone? rclone touch always does a HEAD of the object even when using --no-check-dest and --s3-no-head and other options. #### What is your rclone version (output from `rclone version`) rclone v1.65.1 - os/version: darwin 14.3.1 (64 bit) - os/kernel: 23.3.0 (x86_64) - os/type: darwin - os/arch: amd64 - go/version: go1.21.5 - go/linking: dynamic - go/tags: cmount #### Which OS you are using and how many bits (e.g. Windows 7, 64 bit) macOS Sonoma 14.3.1 (23D60) #### Which cloud storage system are you using? (e.g. Google Drive) AWS S3 #### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`) `rclone -vv --dump=headers touch --no-check-dest --s3-no-check-bucket --s3-no-head --s3-no-head-object --http-no-head s3:mahbucke/why` #### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`) ``` 2024/02/29 16:13:14 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "-vv" "--dump=headers" "touch" "--no-check-dest" "--s3-no-check-bucket" "--s3-no-head" "--s3-no-head-object" "--http-no-head" "s3:mahbucke/why"] 2024/02/29 16:13:14 DEBUG : Creating backend with remote "s3:mahbucke/" 2024/02/29 16:13:14 DEBUG : Using config file from "/Users/jafshar/.rclone.conf" 2024/02/29 16:13:14 DEBUG : s3: detected overridden config - adding "{ECHgI}" suffix to name 2024/02/29 16:13:14 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it. 2024/02/29 16:13:14 NOTICE: s3: s3 provider "" not known - please set correctly 2024/02/29 16:13:14 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it. 2024/02/29 16:13:14 DEBUG : fs cache: renaming cache item "s3:mahbucke/" to be canonical "s3{ECHgI}:mahbucke" 2024/02/29 16:13:14 DEBUG : Touch time 2024-02-29 16:13:14.160928 -0600 CST m=+0.332722504 2024/02/29 16:13:14 DEBUG : S3 bucket mahbucke: Touching "why" 2024/02/29 16:13:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024/02/29 16:13:14 DEBUG : HTTP REQUEST (req 0xc0009d0c00) 2024/02/29 16:13:14 DEBUG : HEAD /mahbucke/why HTTP/1.1 Host: s3.us-east-1.amazonaws.com User-Agent: rclone/v1.65.1 Authorization: XXXX X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20240229T221314Z 2024/02/29 16:13:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024/02/29 16:13:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2024/02/29 16:13:14 DEBUG : HTTP RESPONSE (req 0xc0009d0c00) 2024/02/29 16:13:14 DEBUG : HTTP/1.1 200 OK Accept-Ranges: bytes Content-Type: application/octet-stream Date: Thu, 29 Feb 2024 22:13:15 GMT Etag: "d41d8cd98f00b204e9800998ecf8427e" Last-Modified: Thu, 29 Feb 2024 21:41:47 GMT Server: AmazonS3 X-Amz-Id-2: Hsozbf/FeYD7i5PDf30LJ5U4/pluTvbwLKB/Iy0m85FvtPWstSF8qm9+6nTuvExP+iEJ9pY42uk= X-Amz-Meta-Mtime: 1709242906.463672 X-Amz-Request-Id: NZ0FV3HQT6JDPM2E X-Amz-Server-Side-Encryption: AES256 X-Amz-Version-Id: 539XtfAmDDYq6ESubNknFqvOrfM7t7fF Content-Length: 0 2024/02/29 16:13:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2024/02/29 16:13:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024/02/29 16:13:14 DEBUG : HTTP REQUEST (req 0xc00077a300) 2024/02/29 16:13:14 DEBUG : PUT /mahbucke/why HTTP/1.1 Host: s3.us-east-1.amazonaws.com User-Agent: rclone/v1.65.1 Content-Length: 0 Authorization: XXXX Content-Type: application/octet-stream X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Copy-Source: mahbucke/why X-Amz-Date: 20240229T221314Z X-Amz-Meta-Mtime: 1709244794.160928 X-Amz-Metadata-Directive: REPLACE Accept-Encoding: gzip 2024/02/29 16:13:14 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2024/02/29 16:13:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2024/02/29 16:13:14 DEBUG : HTTP RESPONSE (req 0xc00077a300) 2024/02/29 16:13:14 DEBUG : HTTP/1.1 200 OK Content-Length: 224 Content-Type: application/xml Date: Thu, 29 Feb 2024 22:13:15 GMT Server: AmazonS3 X-Amz-Copy-Source-Version-Id: 539XtfAmDDYq6ESubNknFqvOrfM7t7fF X-Amz-Id-2: jseGFGZPSLOY+IpF6mhapz/Rxqap7ap85i+W+BtzHSc/dkt1mp96s027YIt/upoTQGxCQqmS1TM= X-Amz-Request-Id: NZ088VZWA4EH52WD X-Amz-Server-Side-Encryption: AES256 X-Amz-Version-Id: UCId7Uvtg3Qd8RWuxxzQ.bgLVmY4tZIr 2024/02/29 16:13:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2024/02/29 16:13:14 DEBUG : 7 go routines active ``` #### How to use GitHub * Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue. * Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue. * Subscribe to receive notifications on status change and new comments.

jamshid · March 1, 2024, 9:50pm

Has anyone else noticed that rclone touch does not work with --files-from? It reports:

Failed to touch: InvalidParameter: 1 validation error(s) found. - minimum field size of 1, HeadObjectInput.Key.

% rclone -vv --dump=headers touch --no-check-dest --s3-no-check-bucket s3:mahbucke/ --files-from /tmp/xx --no-traverse

2024/02/29 17:01:55 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "-vv" "--dump=headers" "touch" "--no-check-dest" "--s3-no-check-bucket" "s3:mahbucke/" "--files-from" "/tmp/xx" "--no-traverse"]
2024/02/29 17:01:55 DEBUG : Creating backend with remote "s3:mahbucke/"
2024/02/29 17:01:55 DEBUG : Using config file from "/Users/jafshar/.rclone.conf"
2024/02/29 17:01:55 DEBUG : s3: detected overridden config - adding "{Dn7qA}" suffix to name
2024/02/29 17:01:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/02/29 17:01:55 NOTICE: s3: s3 provider "" not known - please set correctly
2024/02/29 17:01:55 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/02/29 17:01:55 DEBUG : fs cache: renaming cache item "s3:mahbucke/" to be canonical "s3{Dn7qA}:mahbucke"
2024/02/29 17:01:55 DEBUG : Touch time 2024-02-29 17:01:55.127069 -0600 CST m=+0.094187058
2024/02/29 17:01:55 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2024/02/29 17:01:55 DEBUG : Touch time 2024-02-29 17:01:55.127289 -0600 CST m=+0.094407378
2024/02/29 17:01:55 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2024/02/29 17:01:55 DEBUG : Touch time 2024-02-29 17:01:55.127375 -0600 CST m=+0.094492831
2024/02/29 17:01:55 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2024/02/29 17:01:55 DEBUG : 6 go routines active
2024/02/29 17:01:55 Failed to touch: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

asdffdsa · March 1, 2024, 10:04pm

need to add provider = AWS

jamshid · March 1, 2024, 10:17pm

Thanks I'll add that but any idea what I have to do to avoid the InvalidParameter error?
My /tmp/xx just has one line with /test.

system · March 31, 2024, 10:18pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.