Backblaze (crypt) server side move slow

Wanted to rename and move some folders around within my Backblaze remote. It consists of around 55k files, most small. Found that I should use the move command for that. It worked, but seemed to take an unreasonable amount of time, and I had to interupt the operation before long as it was rapidly approaching my daily class C transaction cap. At that rate I would be better off just syncing everything again from local storage, which barely uses any transactions thanks to --fast-list. (which I suspect does nothing with the move command) I thought I might try rclone move the few really big files, and then sync the rest, as the big files might be fast. Not so. I don't know whether it's an issue of Backblaze, encryption, rclone, or something else.

What is the problem you are having with rclone?

Server side move operation is slow or not working properly.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2
- os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
- os/kernel: 10.0.17763.4131 Build 17763.4131.4131 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using?

Backblaze B2

The command you were trying to run

rclone moveto --transfers 32 --no-traverse --fast-list -P backblaze-crypt:files/output-x264-crf17.mp4 backblaze-crypt:files/Storage1

The rclone config contents with secrets removed.

[backblaze]
type = b2
account = x
key = x

[backblaze-crypt]
type = crypt
remote = backblaze:/backup
password = x
directory_name_encryption = false

[storage]
type = combine
upstreams = Storage1=D:\Storage Storage2=E:\Storage Storage3=C:\Storage

A log from the command with the -vv flag

2023/04/26 18:49:26 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "moveto" "-P" "-vv" "backblaze-crypt:files/output-x264-crf17.mp4" "backblaze-crypt:files/Storage1"]
2023/04/26 18:49:26 DEBUG : Creating backend with remote "backblaze-crypt:files/output-x264-crf17.mp4"
2023/04/26 18:49:26 DEBUG : Using config file from "C:\\Users\\x\\AppData\\Roaming\\rclone\\rclone.conf"
2023/04/26 18:49:26 DEBUG : Creating backend with remote "backblaze:/backup/files/o78f0touv2j7gr67in571atg5163qnc4j8oah337kevd59stev6g"
2023/04/26 18:49:27 DEBUG : fs cache: adding new entry for parent of "backblaze:/backup/files/o78f0touv2j7gr67in571atg5163qnc4j8oah337kevd59stev6g", "backblaze:backup/files"
2023/04/26 18:49:27 DEBUG : Creating backend with remote "backblaze-crypt:files/"
2023/04/26 18:49:27 DEBUG : Creating backend with remote "backblaze:/backup/files"
2023/04/26 18:49:27 DEBUG : Couldn't decode error response: EOF
2023/04/26 18:49:27 DEBUG : fs cache: renaming cache item "backblaze:/backup/files" to be canonical "backblaze:backup/files"
2023/04/26 18:49:27 DEBUG : fs cache: switching user supplied name "backblaze:/backup/files" for canonical name "backblaze:backup/files"
2023-04-26 18:49:27 DEBUG : Couldn't decode error response: EOF
2023-04-26 18:49:27 DEBUG : output-x264-crf17.mp4: Need to transfer - File not found at Destination
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Starting copy of large file in 3 chunks (id "4_z9894cc2b102f4e368e750b13_f22435599c420be65_d20230426_m164928_c003_v0312011_t0001_u01682527768014")
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Copying chunk 3 length 3474184593
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Copying chunk 1 length 4294967296
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Copying chunk 2 length 4294967296
2023-04-26 18:51:17 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Done copying chunk 1
Transferred:              0 B / 11.233 GiB, 0%, 0 B/s, ETA -
Checks:                 0 / 1, 0%
Transferred:            0 / 1, 0%
Elapsed time:      3m22.3s
Checking:

Transferring:
 *                         output-x264-crf17.mp4:  0% /11.233Gi, 0/s, -

At this point I interupted the command. I removed some flags for this debugged command, but with the original command I waited for over 20 mins without any discernible progress.

Hi Lombra,

Backblaze B2 doesn't support move as you can see here, so what you see is rclone performing a copy followed by a delete.

Hi,

Sorry. Yes, I have understood that Backblaze move is actually a copy. My expectations are with that in mind. Still, that does not explain my very unsuccessful attempt to "move" this large file.

I see, yes that I cannot immediately explain.

Could be that the crypt starts an ordinary move and then stumbles when the underlying B2 can't.

Sounds like you have it all locally, so I suggest you just do all the changes locally and then do a sync, that will also save some downloads.

I tried again with a different, similarly sized file. It didn't seem to do much during the first few seconds and I did not watch the progress, but after three minutes it had finished. I also realised that you can't use moveto to move something into a folder, because that will replace said folder with the moved object. You have to specify the destination path including the name of the moved object. Perhaps that complicated things further.

Checking now, it seems like the first file may have gotten moved despite the reported non progress... I'm not sure. I've been moving stuff around so much that I've lost track.

Edit: Or maybe I was looking in the wrong folder... I'll just start over from scratch at this point.

Good to hear, sounds like you are almost in control and learned a lot - even if you have to start from scratch.

Happy moving!

B2 supports server-side-copy so it shouldn't be downloading, right?

Copy yes.
Move no.

The OP is moving files. I'd probably copy them and then delete imo :slight_smile:

Sorry, I just want to be clear. If the remote supports server-side-copy and I do a move of a file, it will server-side copy then delete? Or are you saying that it will do a non-server-side copy then delete?

I don't think so as the post suggests here.

I was saying it 'appears' to support server side copy by the overview list but I don't use Backblaze so I can't confirm.

If that is true, I was suggesting to copy and then delete.

You are absolutely right @jwink3101, rclone does a server-side copy followed by a delete.

Here confirmed on S3 with similar characteristics:

PS> rclone move myS3bucketCrypt:folder1 myS3bucketCrypt:folder2 -v 
2023/04/27 18:42:08 INFO  : hello: Copied (server-side copy)
2023/04/27 18:42:08 INFO  : hello: Deleted
2023/04/27 18:42:08 INFO  :
Transferred:             15 B / 15 B, 100%, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         0.5s

@Lombra This also explains the apparent lack of progress in the stats, they remain at 0% during the server-side copy and then suddenly jumps to 100% when it is complete. (There is no way for rclone to get continuous progress from the server)

Well spotted, thanks!

I'm confused then as the logs show a copy happening, nothing server side.

Where are you seeing a server side event ?

Interesting, the chunking happens when the size is above --b2-copy-cutoff implemented by this commit in PR #3991.

The description says:

Any files larger than this that need to be server-side copied will be copied in chunks of this size.

Please don't ask me if this is done client-side or server-side, or causes the crypt move to hang.

I see @Lombra typing, clarification may be incoming...

Here is the test I would do on B2:

echo "Hello Backblaze B2" | rclone rcat b2bucketCrypt:testfolder1/testfile
rclone move  b2bucketCrypt:testfolder1  b2bucketCrypt:testfolder2 --b2-copy-cutoff=4B -vv

Ah. I suppose that makes sense. Actually, I did not notice the Done copying chunk line sneakily inserted above the progress indicators until now.

Successive attempts seems to indicate that things are at least working.

It definitely is a server side operation. No network activity is going on during, and I certainly don't have these bandwidths at home. The server side copy line does not get logged until the operation has completed. The log also demonstrates the use of copy + delete to perform a move operation.

2023/04/27 20:53:18 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "moveto" "-P" "-vv" "--no-traverse" "backblaze-crypt:Storage1/rpi-boot.img" "backblaze-crypt:files/Storage1/rpi-boot.img"]
[...]
2023-04-27 20:53:20 DEBUG : rpi-boot.img: Need to transfer - File not found at Destination
2023-04-27 20:53:20 DEBUG : 3b38q99tb7su9n5094tfhveikg: Starting copy of large file in 2 chunks (id "4_z9894cc2b102f4e368e750b13_f2321d48a120b229a_d20230427_m185320_c003_v0312012_t0006_u01682621600420")
2023-04-27 20:53:20 DEBUG : 3b38q99tb7su9n5094tfhveikg: Copying chunk 2 length 3599591200
2023-04-27 20:53:20 DEBUG : 3b38q99tb7su9n5094tfhveikg: Copying chunk 1 length 4294967296
2023-04-27 20:55:06 DEBUG : 3b38q99tb7su9n5094tfhveikg: Done copying chunk 2
2023-04-27 20:55:40 DEBUG : 3b38q99tb7su9n5094tfhveikg: Done copying chunk 1
2023-04-27 20:55:40 DEBUG : 3b38q99tb7su9n5094tfhveikg: Finishing large file copy with 2 parts
2023-04-27 20:55:40 INFO  : rpi-boot.img: Copied (server-side copy)
2023-04-27 20:55:42 INFO  : rpi-boot.img: Deleted
Transferred:        7.351 GiB / 7.351 GiB, 100%, 442.225 MiB/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:      2m23.2s
1 Like

Ah, there we go as the full log shows the last bit that was missing from your original post.

I'd imagine it takes it a few to combine on the server side, which is the delay.

To be clear, the original attempt was interupted as I thought it was not doing anything. It was the full log from that attempt.

Oh trust me, I have almost negative patience so I'd have done the same. I didn't mean anything by it as I was just confused from the log at first.

1 Like

Still, my attempt before I started logging went on for 20 mins before I interupted it, so I have no idea what was going on there. (a single 11 GB file) I guess it will remain a mystery.

That does seem wonky. Hopefully a magical one off!

HI. We use B2 with Rclone on Windows. We had a similar issue with moves and copies and yes agreed B2 Copy and then Delete from the source for Move. In working with B2 support they indicated --b2-list-file-names is executed for about every 1,000 (if I recall correctly) files so the more files not to be confused with file size (which does not matter) the more time things take. We ended up; 1) Splitting up our buckets and 2) Changing the B2 default from save all file versions to save only the most recent. This helped speed things up significantly. Good luck.