Rclone purge with files-from doesn't work

What is the problem you are having with rclone?

I'm trying to purge a list of "directories" from my AWS S3 bucket. The purging works good if I give call it directly with the path, but fails if I handover a list of paths with the "--files-from" command.

In the list there is only 1 entry "0003/".
The purge works with the list if the entry points to a file and not a folder. So the list would be "0003/1.txt"

What is your rclone version (output from rclone version)

1.56.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows Server 2019 64 bit

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone purge --files-from delete.txt aws:mybucket/ --interactive  --no-traverse 

The rclone config contents with secrets removed.

type = s3
provider = AWS
env_auth = false
access_key_id = 
secret_access_key = 
region = eu-central-1
location_constraint = eu-central-1
acl = private
server_side_encryption = AES256
storage_class = STANDARD

A log from the command with the -vv flag

2021/08/12 12:49:57 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "purge" "--files-from" "delete" "aws:mybucket/" "--interactive" "--no-traverse" "-vv" "--dump" "bodies"]
2021/08/12 12:49:57 DEBUG : Creating backend with remote "awsQA:mybucket/"
2021/08/12 12:49:57 DEBUG : Using config file from "C:\\Users\\Administrator\\.config\\rclone\\rclone.conf"
2021/08/12 12:49:57 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/08/12 12:49:57 DEBUG : fs cache: renaming cache item "awsQA:mybucket/" to be canonical "awsQA:mybucket"
2021/08/12 12:49:57 DEBUG : Waiting for deletions to finish
2021/08/12 12:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/12 12:49:57 DEBUG : HTTP REQUEST (req 0xc000700200)
2021/08/12 12:49:57 DEBUG : HEAD /0003 HTTP/1.1
Host: mybucket.s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.56.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210812T104957Z

2021/08/12 12:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/12 12:49:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/12 12:49:57 DEBUG : HTTP RESPONSE (req 0xc000700200)
2021/08/12 12:49:57 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 12 Aug 2021 10:49:57 GMT
Server: AmazonS3
X-Amz-Id-2: VIeDK7iY1qrDAcoXtQmVz3b+bB0vUkArsF0d198qIwADV6GcrYoQSdz+CyqdFVtpDGZIiU9os28=
X-Amz-Request-Id: 6VGQ0S6FJX0EH25Q

2021/08/12 12:49:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/12 12:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/12 12:49:57 DEBUG : HTTP REQUEST (req 0xc00013f200)
2021/08/12 12:49:57 DEBUG : HEAD /0003 HTTP/1.1
Host: mybucket.s3.eu-central-1.amazonaws.com
User-Agent: rclone/v1.56.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210812T104957Z

2021/08/12 12:49:57 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/08/12 12:49:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/12 12:49:57 DEBUG : HTTP RESPONSE (req 0xc00013f200)
2021/08/12 12:49:57 DEBUG : HTTP/1.1 404 Not Found
Connection: close
Content-Type: application/xml
Date: Thu, 12 Aug 2021 10:49:57 GMT
Server: AmazonS3
X-Amz-Id-2: KqCxQrH1gp6Aq2qXoAlYzndiUbCcBylSSfzpvym2KBPHDEjGJsiJMkVVLJpgHkz3IFbVy4A2Sso=
X-Amz-Request-Id: 6VGQ3NTSESXVHT4H

2021/08/12 12:49:57 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/08/12 12:49:57 DEBUG : 4 go routines active

Hi Ralf, welcome to the forum

The docs says: "--files-from expects a list of files as its input.", so it makes good sense if it it doesn't work with a list of directories.

I havn't tried, but --include-from or --filter-from may be better in your situation. Your can see all the filtering options here.

Please be cautious when trying them, use --interactive or --dry-run.

Tip: It may be easier to test filters with rclone lsl in a local temp folder

Hi Ole - just tried but fails as well :frowning:

Sorry, you are right.

The docs for rclone purge says: “Note that this does not obey include/exclude filters”, so this also makes sense.

Then I would make make a script with repeated calls to rclone purge. (remember --dry-run)

Yeah I guess I'll do so. I thought it might be an easy way with a built-in rclone feature, but nah.
Thanks Ole!

As Ole said, purge does not support filtering, you can use delete command for filtering. Takes longer, but works.

Just to add context as to the difference between purge and delete is that purge will try to use backend specific commands to delete the directory while delete will delete it file-by-file.

For some remotes such as B2 or S3, there is no real difference but for others, it can be a major speed up!

1 Like

For a list of directories you can make an --include-from file

This should contain

path/to/dir/**
path/to/dir2/**

Test first with --dry-run

You can also do rclone lsf --include-from to list the files which would be deleted.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.