Hello,
since server-side copy with backblaze is multipe times slower than down and uploading the file again I'm wondering if it's possible to disable server-side copy.
Matthias
hi,
that is very strange, that server-side copy should be slower.
have you seen this?
https://rclone.org/docs/#disable-feature-feature
Thanks! I'll try that.
Slow means, renaming a 1GB file needs about 10 minutes. Now I'm renaming some folders with about 30 files and it's running for more than 2 hours.
not sure how that can be correct.
when you posted, there was a template of questions.
without that, we have no idea what is going on......
Well, my initial question was answered. If you think renaming should be faster, here's the template:
What is the problem you are having with rclone?
Server-side move appears too slow using the backblaze b2 storage. Renaming a bunch of files needs multiple hours.
What is your rclone version (output from rclone version)
$ rclone --version
rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Debian 10
Which cloud storage system are you using? (eg Google Drive)
Backblaze B2
The command you were trying to run (eg rclone copy /tmp remote:tmp)
$ rclone move --transfers 10 -vv b2eu01:test.bin b2eu01:test2.bin
The rclone config contents with secrets removed.
[b2eu01]
directory_name_encryption = true
filename_encryption = standard
password = redacted
password2 = redacted
remote = b2eu:redacted
type = crypt
[b2eu]
account = redacted
key = redacted
type = b2
A log from the command with the -vv flag
2021/06/19 21:10:12 DEBUG : Using config file from "/home/matthias/.config/rclone/rclone.conf"
2021/06/19 21:10:12 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "move" "-vv" "b2eu01:test.bin" "b2eu01:test2.bin"]
2021/06/19 21:10:12 DEBUG : Creating backend with remote "b2eu01:test.bin"
2021/06/19 21:10:12 DEBUG : Creating backend with remote "b2eu:<redacted:bucket>/<redacted:enc?(test.bin)"
2021/06/19 21:10:13 DEBUG : fs cache: adding new entry for parent of "b2eu:<redacted:bucket>/<redacted:enc?(test.bin)", "b2eu:<redacted:bucket>"
2021/06/19 21:10:13 DEBUG : Creating backend with remote "b2eu01:test2.bin"
2021/06/19 21:10:13 DEBUG : Creating backend with remote "b2eu:<redacted:bucket>/<redacted:enc?(test2.bin)"
2021/06/19 21:10:14 DEBUG : Couldn't decode error response: EOF
2021/06/19 21:10:14 DEBUG : Couldn't decode error response: EOF
2021/06/19 21:10:14 DEBUG : test.bin: Need to transfer - File not found at Destination
2021/06/19 21:11:14 INFO :
Transferred: 0 / 500 MBytes, 0%, 0 Bytes/s, ETA -
Checks: 0 / 1, 0%
Transferred: 0 / 1, 0%
Elapsed time: 1m1.5s
Checking:
Transferring:
* test.bin: 0% /500M, 0/s, -
2021/06/19 21:12:14 INFO :
Transferred: 0 / 500 MBytes, 0%, 0 Bytes/s, ETA -
Checks: 0 / 1, 0%
Transferred: 0 / 1, 0%
Elapsed time: 2m1.5s
Checking:
Transferring:
* test.bin: 0% /500M, 0/s, -
2021/06/19 21:13:14 INFO :
Transferred: 0 / 500 MBytes, 0%, 0 Bytes/s, ETA -
Checks: 0 / 1, 0%
Transferred: 0 / 1, 0%
Elapsed time: 3m1.5s
Checking:
Transferring:
* test.bin: 0% /500M, 0/s, -
2021/06/19 21:14:14 INFO :
Transferred: 0 / 500 MBytes, 0%, 0 Bytes/s, ETA -
Checks: 0 / 1, 0%
Transferred: 0 / 1, 0%
Elapsed time: 4m1.5s
Checking:
Transferring:
* test.bin: 0% /500M, 0/s, -
2021/06/19 21:15:00 INFO : test.bin: Copied (server-side copy)
2021/06/19 21:15:02 INFO : test.bin: Deleted
2021/06/19 21:15:02 INFO :
Transferred: 500M / 500 MBytes, 100%, 1.736 MBytes/s, ETA 0s
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Renamed: 1
Transferred: 1 / 1, 100%
Elapsed time: 4m49.6s
2021/06/19 21:15:02 DEBUG : 4 go routines active
well,
there is this Couldn't decode error response: EOF?
could be some kind of bug with rclone or b2
and if try to move a file using b2eu:, what happens?
Without the encryption layer it's faster:
rclone move -vv b2eu:<redacted:bucket>/test.bin b2eu:<redacted:bucket>/test2.bin
2021/06/19 21:30:54 DEBUG : Using config file from "/home/matthias/.config/rclone/rclone.conf"
2021/06/19 21:30:54 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "move" "-vv" "b2eu:<redacted:bucket>/test.bin" "b2eu:<redacted:bucket>/test2.bin"]
2021/06/19 21:30:54 DEBUG : Creating backend with remote "b2eu:<redacted:bucket>/test.bin"
2021/06/19 21:30:55 DEBUG : fs cache: adding new entry for parent of "b2eu:<redacted:bucket>/test.bin", "b2eu:<redacted:bucket>"
2021/06/19 21:30:55 DEBUG : Creating backend with remote "b2eu:<redacted:bucket>/test2.bin"
2021/06/19 21:30:56 DEBUG : Couldn't decode error response: EOF
2021/06/19 21:30:56 DEBUG : Couldn't decode error response: EOF
2021/06/19 21:30:56 DEBUG : test.bin: Need to transfer - File not found at Destination
2021/06/19 21:31:37 DEBUG : test.bin: SHA-1 = 0c8d3cc1a9dd96d1dc1ea35f3c3a13dc095bd0db OK
2021/06/19 21:31:37 INFO : test.bin: Copied (server-side copy)
2021/06/19 21:31:42 INFO : test.bin: Deleted
2021/06/19 21:31:42 INFO :
Transferred: 500M / 500 MBytes, 100%, 10.694 MBytes/s, ETA 0s
Checks: 2 / 2, 100%
Deleted: 1 (files), 0 (dirs)
Renamed: 1
Transferred: 1 / 1, 100%
Elapsed time: 48.2s
2021/06/19 21:31:42 DEBUG : 4 go routines active
but still those same errors,
can you add this to your command and post it?
--dump=bodies --retries=1 --low-level-retries=1
The log can be found here:
https://paste.debian.net/hidden/4ffd4e59/
this time it was slow again, but the server-side move was finally successful.
@ncw, hi, is there something wrong here, getting Couldn't decode error response: EOF in the log file?
look like you were trying to upload a file and rename it.
is that correct?
if so, the correct command is moveto
Yes, I uploaded it for the purpose of having a file to rename.
to rename a file, need to use moveto
you are trying to rename test.bin to test2.bin, correct?
if you look at the log, it does not look the file is getting renamed, rclone is creating a folder named test2.bin and then copying test.bin into test2.bin
HEAD /file/<redacted:bucket>/test2.bin/test.bin
Ah yes, you're right! But I guess the operation server side is the same?
I tried copyto in the US region. I copied the 500MB test.bin 5 times in a row:
Elapsed time: 20.7s
Elapsed time: 37.8s
Elapsed time: 13.0s
Elapsed time: 23.3s
Elapsed time: 24.3s
This still seems rather slow to me.
without a debug log, hard to comment.
was the copy, server-side or what?
slow as compared to what?
copyto within the same bucket. According to the logs it was a server-side copy.
for a server-side-copy, rclone can only wait until the operation has completed.
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.