Wanted to rename and move some folders around within my Backblaze remote. It consists of around 55k files, most small. Found that I should use the move command for that. It worked, but seemed to take an unreasonable amount of time, and I had to interupt the operation before long as it was rapidly approaching my daily class C transaction cap. At that rate I would be better off just syncing everything again from local storage, which barely uses any transactions thanks to --fast-list. (which I suspect does nothing with the move command) I thought I might try rclone move the few really big files, and then sync the rest, as the big files might be fast. Not so. I don't know whether it's an issue of Backblaze, encryption, rclone, or something else.
What is the problem you are having with rclone?
Server side move operation is slow or not working properly.
Run the command 'rclone version' and share the full output of the command.
rclone v1.62.2
- os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
- os/kernel: 10.0.17763.4131 Build 17763.4131.4131 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: cmount
[backblaze]
type = b2
account = x
key = x
[backblaze-crypt]
type = crypt
remote = backblaze:/backup
password = x
directory_name_encryption = false
[storage]
type = combine
upstreams = Storage1=D:\Storage Storage2=E:\Storage Storage3=C:\Storage
A log from the command with the -vv flag
2023/04/26 18:49:26 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "moveto" "-P" "-vv" "backblaze-crypt:files/output-x264-crf17.mp4" "backblaze-crypt:files/Storage1"]
2023/04/26 18:49:26 DEBUG : Creating backend with remote "backblaze-crypt:files/output-x264-crf17.mp4"
2023/04/26 18:49:26 DEBUG : Using config file from "C:\\Users\\x\\AppData\\Roaming\\rclone\\rclone.conf"
2023/04/26 18:49:26 DEBUG : Creating backend with remote "backblaze:/backup/files/o78f0touv2j7gr67in571atg5163qnc4j8oah337kevd59stev6g"
2023/04/26 18:49:27 DEBUG : fs cache: adding new entry for parent of "backblaze:/backup/files/o78f0touv2j7gr67in571atg5163qnc4j8oah337kevd59stev6g", "backblaze:backup/files"
2023/04/26 18:49:27 DEBUG : Creating backend with remote "backblaze-crypt:files/"
2023/04/26 18:49:27 DEBUG : Creating backend with remote "backblaze:/backup/files"
2023/04/26 18:49:27 DEBUG : Couldn't decode error response: EOF
2023/04/26 18:49:27 DEBUG : fs cache: renaming cache item "backblaze:/backup/files" to be canonical "backblaze:backup/files"
2023/04/26 18:49:27 DEBUG : fs cache: switching user supplied name "backblaze:/backup/files" for canonical name "backblaze:backup/files"
2023-04-26 18:49:27 DEBUG : Couldn't decode error response: EOF
2023-04-26 18:49:27 DEBUG : output-x264-crf17.mp4: Need to transfer - File not found at Destination
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Starting copy of large file in 3 chunks (id "4_z9894cc2b102f4e368e750b13_f22435599c420be65_d20230426_m164928_c003_v0312011_t0001_u01682527768014")
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Copying chunk 3 length 3474184593
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Copying chunk 1 length 4294967296
2023-04-26 18:49:28 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Copying chunk 2 length 4294967296
2023-04-26 18:51:17 DEBUG : 9utrbsjm70s6k5s8sku6d4uel4: Done copying chunk 1
Transferred: 0 B / 11.233 GiB, 0%, 0 B/s, ETA -
Checks: 0 / 1, 0%
Transferred: 0 / 1, 0%
Elapsed time: 3m22.3s
Checking:
Transferring:
* output-x264-crf17.mp4: 0% /11.233Gi, 0/s, -
At this point I interupted the command. I removed some flags for this debugged command, but with the original command I waited for over 20 mins without any discernible progress.
Sorry. Yes, I have understood that Backblaze move is actually a copy. My expectations are with that in mind. Still, that does not explain my very unsuccessful attempt to "move" this large file.
I tried again with a different, similarly sized file. It didn't seem to do much during the first few seconds and I did not watch the progress, but after three minutes it had finished. I also realised that you can't use moveto to move something into a folder, because that will replace said folder with the moved object. You have to specify the destination path including the name of the moved object. Perhaps that complicated things further.
Checking now, it seems like the first file may have gotten moved despite the reported non progress... I'm not sure. I've been moving stuff around so much that I've lost track.
Edit: Or maybe I was looking in the wrong folder... I'll just start over from scratch at this point.
Sorry, I just want to be clear. If the remote supports server-side-copy and I do a move of a file, it will server-side copy then delete? Or are you saying that it will do a non-server-side copy then delete?
@Lombra This also explains the apparent lack of progress in the stats, they remain at 0% during the server-side copy and then suddenly jumps to 100% when it is complete. (There is no way for rclone to get continuous progress from the server)
Ah. I suppose that makes sense. Actually, I did not notice the Done copying chunk line sneakily inserted above the progress indicators until now.
Successive attempts seems to indicate that things are at least working.
It definitely is a server side operation. No network activity is going on during, and I certainly don't have these bandwidths at home. The server side copy line does not get logged until the operation has completed. The log also demonstrates the use of copy + delete to perform a move operation.
2023/04/27 20:53:18 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "moveto" "-P" "-vv" "--no-traverse" "backblaze-crypt:Storage1/rpi-boot.img" "backblaze-crypt:files/Storage1/rpi-boot.img"]
[...]
2023-04-27 20:53:20 DEBUG : rpi-boot.img: Need to transfer - File not found at Destination
2023-04-27 20:53:20 DEBUG : 3b38q99tb7su9n5094tfhveikg: Starting copy of large file in 2 chunks (id "4_z9894cc2b102f4e368e750b13_f2321d48a120b229a_d20230427_m185320_c003_v0312012_t0006_u01682621600420")
2023-04-27 20:53:20 DEBUG : 3b38q99tb7su9n5094tfhveikg: Copying chunk 2 length 3599591200
2023-04-27 20:53:20 DEBUG : 3b38q99tb7su9n5094tfhveikg: Copying chunk 1 length 4294967296
2023-04-27 20:55:06 DEBUG : 3b38q99tb7su9n5094tfhveikg: Done copying chunk 2
2023-04-27 20:55:40 DEBUG : 3b38q99tb7su9n5094tfhveikg: Done copying chunk 1
2023-04-27 20:55:40 DEBUG : 3b38q99tb7su9n5094tfhveikg: Finishing large file copy with 2 parts
2023-04-27 20:55:40 INFO : rpi-boot.img: Copied (server-side copy)
2023-04-27 20:55:42 INFO : rpi-boot.img: Deleted
Transferred: 7.351 GiB / 7.351 GiB, 100%, 442.225 MiB/s, ETA 0s
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Renamed: 1
Transferred: 1 / 1, 100%
Elapsed time: 2m23.2s
Still, my attempt before I started logging went on for 20 mins before I interupted it, so I have no idea what was going on there. (a single 11 GB file) I guess it will remain a mystery.
HI. We use B2 with Rclone on Windows. We had a similar issue with moves and copies and yes agreed B2 Copy and then Delete from the source for Move. In working with B2 support they indicated --b2-list-file-names is executed for about every 1,000 (if I recall correctly) files so the more files not to be confused with file size (which does not matter) the more time things take. We ended up; 1) Splitting up our buckets and 2) Changing the B2 default from save all file versions to save only the most recent. This helped speed things up significantly. Good luck.