Purge gets stuck with Storj

Suspected Bug
1

What is the problem you are having with rclone?

# rclone purge storj:u324186-bkp -vv --dump bodies --retries 1 --low-level-retries 1
2024/01/23 05:44:13 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "purge" "storj:u324186-bkp" "-vv" "--dump" "bodies" "--retries" "1" "--low-level-retries" "1"]
2024/01/23 05:44:13 DEBUG : Creating backend with remote "storj:u324186-bkp"
2024/01/23 05:44:13 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/23 05:44:13 DEBUG : FS sj://u324186-bkp: connecting...
2024/01/23 05:44:13 DEBUG : FS sj://u324186-bkp: connected: <nil>

Command gets stuck here. All other command like sync, ls, lsd, etc, succeed.

Run the command 'rclone version' and share the full output of the command.

# rclone version
rclone v1.65.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-91-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Storj

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone purge storj:u324186-bkp -vv --dump bodies --retries 1 --low-level-retries 1

The rclone config contents with secrets removed.

[storj]
type = storj
access_grant = 1z7iraMTbV81w
satellite_address = eu1.storj.io:7777

A log from the command with the -vv flag

2024/01/23 05:48:18 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "purge" "storj:u324186-bkp" "-vv" "--dump" "bodies" "--retries" "1" "--low-level-retries" "1"]
2024/01/23 05:48:18 DEBUG : Creating backend with remote "storj:u324186-bkp"
2024/01/23 05:48:18 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/23 05:48:18 DEBUG : FS sj://u324186-bkp: connecting...
2024/01/23 05:48:18 DEBUG : FS sj://u324186-bkp: connected: <nil>
2

from the docs, purge on a bucket should be efficient

for a deeper look, might try --dump=headers

3

Same output (stuck) with that option added (instead of bodies)

4

rclone purge storj:u324186-bkp/sub-folder -v

Works! I mean, there is the stdout verbose output of each file/folder being deleted. But not the bucket itself...

5

worked for me

rclone.exe purge storj:test01 -vv --config=c:\data\rclone\rclone.conf
2024/01/23 14:09:35 DEBUG : rclone: Version "v1.65.1" starting with parameters ["d:\\data\\rclone\\exe\\windows\\rclone-v1.65.1-windows-amd64\\rclone.exe" "purge" "storj:test01" "-vv" "--config=c:\\data\\rclone\\rclone.conf"]
2024/01/23 14:09:35 DEBUG : Creating backend with remote "storj:test01"
2024/01/23 14:09:35 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2024/01/23 14:09:35 DEBUG : FS sj://test01: connecting...
2024/01/23 14:09:35 DEBUG : FS sj://test01: connected: <nil>
2024/01/23 14:09:35 DEBUG : 5 go routines active
6

Where is the verbose output?

  1. Create a bucket
  2. Upload many folders and files
  3. Purge the bucket

I have no output on bucket, but an output on subfolder (output of each file and folder being deleted).

7

I was curious myself as I am using storj and keep an eye on any related rclone bugs.

Did exactly what you described above:

[storj-test]
type = storj
access_grant = XXX
satellite_address = eu1.storj.io:7777

$ rclone purge storj-test:test -vv
2024/01/23 16:55:32 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "purge" "storj-test:test" "-vv"]
2024/01/23 16:55:32 DEBUG : Creating backend with remote "storj-test:test"
2024/01/23 16:55:32 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/01/23 16:55:32 DEBUG : FS sj://test: connecting...
2024/01/23 16:55:32 DEBUG : FS sj://test: connected: <nil>
2024/01/23 16:55:33 DEBUG : 8 go routines active

All bucket test and its content have been deleted.

8

Once again: maybe the contents are effectively deleted, BUT, there is NO verbose output !
To test it yourselves:

  1. Create a bucket
  2. Upload many folders and files
  3. Purge the bucket (rclone purge storj:bucket -vv)

No verbose output. On a bucket with hundred of thousand of file, comme just seems to be "stuck".

  1. Create a bucket
  2. Upload many folders and files
  3. Purge a subfolder ((rclone purge storj:bucket/subfolder -vv)

There is verbose output. You can see on screen files being deleted, one by one.

:wink:

9

This is correct.

If you delete only subset of objects rclone has to delete them one by one (there are no really directories in S3 storage)

If you delete a bucket, rclone instructs s3 storage to delete only one "object" - bucket itself. Then S3 probably takes time to do whatever is has to do deleting all objects - it is not visible from S3 API perspective.

10

Thanks.

This does not apply for a (regular, non Storj) S3 bucket.
When deleting a AWS S3 bucket, there is verbose output.

Is this Storj specific?

11

Yes it is. Its normal mode of operations is storj propriety API (this is what you use when using access_grant. S3 gateway is alternative way to interact with storj, provided to make users life easier.

Here you are listed all specific backend-options.

1 Like
12

S3 does not support purge api call.
rclone always iterates over every file and deletes it, one by one.
so there is a lot of output, same as deleting a folder on storj

13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.