B2: rclone can download deleted files

Suspected Bug
1

What is the problem you are having with rclone?

Rclone can access deleted files on b2 (when b2 has versioning disabled).

What is your rclone version (output from rclone version)

rclone v1.57.0

  • os/version: linuxmint 20.2 (64 bit)
  • os/kernel: 5.11.0-41-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux Mint 20.2 Cinnamon

Which cloud storage system are you using? (eg Google Drive)

b2

The command you were trying to run (eg rclone copy /tmp remote:tmp)


rclone copy b2:x1-everything/test.deb ~/Desktop/

The rclone config contents with secrets removed.

[b2]
type = b2
account = [redacted]
key = [redacted]
hard_delete = true

A log from the command with the -vv flag

2021/12/15 20:38:24 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "b2:x1-everything/test.deb" "/home/[redacted]/Desktop/" "-vv"]
2021/12/15 20:38:24 DEBUG : Creating backend with remote "b2:x1-everything/test.deb"
2021/12/15 20:38:24 DEBUG : Using config file from "/home/[redacted]/.config/rclone/rclone.conf"
2021/12/15 20:38:25 DEBUG : fs cache: adding new entry for parent of "b2:x1-everything/test.deb", "b2:x1-everything"
2021/12/15 20:38:25 DEBUG : Creating backend with remote "/home/[redacted]/Desktop/"
2021/12/15 20:38:25 DEBUG : test.deb: Need to transfer - File not found at Destination
2021/12/15 20:38:27 DEBUG : test.deb: sha1 = b55958615b3531ebb4bc48834063c75ded673336 OK
2021/12/15 20:38:27 INFO  : test.deb: Copied (new)
2021/12/15 20:38:27 INFO  :
Transferred:   	    1.295 MiB / 1.295 MiB, 100%, 11.990 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         3.1s

Further information

I used b2-linux - BackBlaze's own command-line tool - to upload, and then delete, the file in question. After the file was uploaded, and before I deleted the file, I told b2-linux to download it; the download succeeded. A minute or so after I had deleted the file, I told b2-linux (again) to download it. That download failed - as it should. Yet, as I say, rclone was still able to download the file. So - unless I make some mistake - there is something odd here.

Here is a script that one can use for testing.

#!/usr/bin/env bash
declare -r file_localDestDir=~/Desktop/ # Edited to remove the quote marks, which were problematic.
declare -r file_leaf='test.deb'
declare -r -i sleepSecs=60

# Before running this script, do the following two things.
# [1] b2-linux upload-file [bucket name] [fullLocalPathOfPathToBeUploaded] [leafnameFileIsToHaveOnB2]
# [2] ./b2-linux delete-file-version [leafnameFileIsToHaveOnB2] [hideous long 'fileId' that is output by the upload-file command]
while true ; do
	date
	echo
	# Use Rclone to try to download the deleted file.
	if rclone copy b2:x1-everything/${file_leaf} ${file_localDestDir} -vv ; then
		echo 'File downloaded.'
	else
		echo 'File not downloaded. Exiting.'
		exit 0
	fi
	echo "Sleeping for ${sleepSecs} .."
	sleep ${sleepSecs}
	# Delete the downloaded file.
	rm -rf ${file_localDestDir}/${file_leaf} \
		&& echo 'Deleted local copy of file.'
	echo "Sleeping for ${sleepSecs} .."
	sleep ${sleepSecs}
	echo
	echo
done

I note finally that the rclone-browser program hangs when I use it to delete files and leave the files undeleted.

I find that rclone is able to download the file some twenty minutes, and counting, after I deleted it.

2

Can you make a simple rclone test?

  1. rclone upload a test file
  2. rclone ls the file
  3. rclone delete the file
  4. rclone ls the file again
3

Actually I think I found your thing:

4

Animosity: well done; that is helpful; thank you.

It seems to me that there is a real rclone problem here as against a mere peculiarity of b2. For, as I have said, the b2 tool deletes the file utterly, as the user expects, whereas rclone does not. EDIT: ah, no, I should say the following. The b2 tool fails, as it should, to download the deleted file - perhaps because that tool uses a highly specific 'file id' number - whereas rclone succeeds, as it should not, in downloading the deleted file.

I have a use-case for rclone that this problem really stuffs up.

5

I find that rclone's cleanup command solves the problem, but, on my setup, it takes over a minute. See this script.

6

I note also that I cannot use b2's tool - 'b2-linux' - to work around the problem. I had though that I could - that I could use that tool to check whether the file at issue exists. But I can't. For, I use Rclone to encrypt the remote. Thu - on pain of porting much of my rclone code to code that uses b2-linux - and losing encryption, I will be unable to tell b2 which file to check the existence of.

7

I discover a workaround. This workaround does work. Yet, it will do as one wants only if the file one wishes to delete (delete completely) is the only item in a directory. The workaround is: use purge to remove the entire directory.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.