What is the problem you are having with rclone?
I'm trying to figure out what the current state of metadata support is, both on Openstack Swift and on S3. I understand that Swift metadata support may be limited or non-existent (although it seems like rclone is reading some metadata as part of the --swift-no-large-objects flag), so I'm trying first to step back and see if I can get these commands working on an S3-compatible service (Cloudflare R2).
What I'm trying to figure out:
- to remove Swift from the equation above, I'm trying to simply filter based on metadata on R2 first, which should (?) support it, given what I see on the service feature table for S3/S3-compatible
If I can get this working, what I ultimately want to know:
- is it possible to sync any metadata from Swift to R2 (i.e. the
X-Object-Manifestheader)? - is it possible to filter based on the presence of a specific header on Swift (i.e. if the
X-Object-Manifestheader is present) — the fact that this header factors in to--swift-no-large-objectsis what makes me curious
Run the command 'rclone version' and share the full output of the command.
$ rclone --version
rclone v1.63.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 4.4.0-1104-aws (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
- OVH (Openstack Swift)
- Cloudflare R2 (S3-compatible)
The command you were trying to run (eg rclone copy /tmp remote:tmp)
As a basic sanity check, I'm trying to get rclone to include a file based on its etag header. In my test bucket (rclone-debugging) exist two files:
testfile1.txtwith anEtagheader value of"59bcc3ad6775562f845953cf01624225"testfile2.txtwith anEtagheader value of"0f18fd4cf40bfb1dec646807c7fa5522"
To start:
$ rclone lsf r2-development:rclone-debugging
testfile1.txt
testfile2.txt
So far so good. Also:
$ rclone lsf r2-development:rclone-debugging -M --metadata-include "*"
testfile1.txt
testfile2.txt
Now I'm trying to match one of those known Etag headers a few different ways. None of the following commands return any files at all. In the last command I'm simply trying to match on the existence of the string etag.
$ rclone lsf r2-development:rclone-debugging -M --metadata-include "59bcc3ad6775562f845953cf01624225"
$ rclone lsf r2-development:rclone-debugging -M --metadata-include "*59bcc3ad6775562f845953cf01624225*"
$ rclone lsf r2-development:rclone-debugging -M --metadata-include "{{(?i).+59bcc3ad6775562f845953cf01624225.+}}"
$ rclone lsf r2-development:rclone-debugging -M --metadata-filter "+ *59bcc3ad6775562f845953cf01624225*" --metadata-filter "- *"
$ rclone lsf r2-development:rclone-debugging -M --metadata-include "{{(?i).+etag.+}}"
The rclone config contents with secrets removed.
[r2-development]
type = s3
provider = Cloudflare
env_auth = true
endpoint = XXXXXXXXXXXXXXXXXX
A log from the command with the -vv flag
$ rclone lsf r2-development:rclone-debugging -M --metadata-include "*59bcc3ad6775562f845953cf01624225*" --dump filters -vv
2023/07/25 19:45:52 DEBUG : Setting --config "/app/config/rclone.conf" from environment variable RCLONE_CONFIG="/app/config/rclone.conf"
--- start filters ---
--- File filter rules ---
--- Directory filter rules ---
--- Metadata filter rules ---
+ (^|/)[^/]*59bcc3ad6775562f845953cf01624225[^/]*$
- ^.*$
--- end filters ---
2023/07/25 19:45:52 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "lsf" "r2-development:rclone-debugging" "-M" "--metadata-include" "*59bcc3ad6775562f845953cf01624225*" "--dump" "filters" "-vv"]
2023/07/25 19:45:52 DEBUG : Creating backend with remote "r2-development:rclone-debugging"
2023/07/25 19:45:52 DEBUG : Using config file from "/app/config/rclone.conf"
2023/07/25 19:45:52 DEBUG : name = "r2-development", root = "rclone-debugging", opt = &s3.Options{Provider:"Cloudflare", EnvAuth:true, AccessKeyID:"", SecretAccessKey:"", Region:"", Endpoint:"XXXXXXXXXXXXXXXXX", STSEndpoint:"", LocationConstraint:"", ACL:"", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:5242880, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:false, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/25 19:45:52 DEBUG : Resolving service "s3" region "us-east-1"
2023/07/25 19:45:52 DEBUG : testfile1.txt: Excluded
2023/07/25 19:45:53 DEBUG : testfile2.txt: Excluded
And just in case it's relevant, here's an excerpt of the previous command, with --dump headers appended to it:
2023/07/25 19:48:54 DEBUG : HTTP REQUEST (req 0xc000902300)
2023/07/25 19:48:54 DEBUG : HEAD /rclone-debugging/testfile1.txt HTTP/1.1
Host: XXXXXXXXXXXXXXXXXXXX
User-Agent: rclone/v1.63.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230725T194854Z
2023/07/25 19:48:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/25 19:48:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/25 19:48:54 DEBUG : HTTP RESPONSE (req 0xc000902300)
2023/07/25 19:48:54 DEBUG : HTTP/1.1 200 OK
Content-Length: 4
Accept-Ranges: bytes
Cf-Ray: 7ec6ec4daf565a1b-IAD
Connection: keep-alive
Content-Type: text/plain
Date: Tue, 25 Jul 2023 19:48:54 GMT
Etag: "59bcc3ad6775562f845953cf01624225"
Last-Modified: Tue, 25 Jul 2023 19:28:31 GMT
Server: cloudflare
2023/07/25 19:48:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/25 19:48:54 DEBUG : testfile1.txt: Excluded
2023/07/25 19:48:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
