i completely understand that it has to calculate checksum.
but my question went into direction if it already calculated it once (took 15 hours), then failed to upload in 3 seconds, then went caculating checksum again before trying to upload again.
i use rclone every day for over a year. many TB uploaded.
only a few times did i have internet problems that caused an upload to fail.
if you would run rclone on the synbox, that pain would be much less.
perhaps upgrade your internet speed.
maybe it had to do with older rclone version, you saw the logs.
this were not internet problems i think.
i'll let you know how it went with newer (up to date) clone.
yes you are right, i'll continue using rclone from synology for sure.
one thing i did was find ways to shrink the size of veeam backup files.
I would be eager to find out how you shrank Veeam files.
are you using agent or VBAR?
community or paid, and if paid what version?
what are you backing up?
physical servers?
virtual machines?
how often are you doing a incremental backup and full backups?
we are using community, backing up VMs (windows+linux).
incrementals once per day, active full monthly, keeping 30 points.
you can run a separate backup for each vm instead of one backup for all vms.
And again, this time with newest rclone running on Synology box I received:
2020/04/13 06:32:24 INFO :
Transferred: 5.159k / 3.745 TBytes, 0%, 0 Bytes/s, ETA 213y27w1d13h51m28s596ms
Checks: 63 / 63, 100%
Transferred: 2 / 4, 50%
Elapsed time: 10h56m59.8s
Transferring:
- 95/Backup95/Backup95D2020-03-06T210059.vbk: 0% /1.857T, 0/s, -
- 95/Backup95/Backup95D2020-04-03T210052.vbk: 0% /1.887T, 0/s, -
2020/04/13 06:32:39 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Multipart upload session started for 30920 parts of size 64M
2020/04/13 06:32:41 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 1/30920 offset 0/1.887T part size 64M
2020/04/13 06:32:42 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 2/30920 offset 64M/1.887T part size 64M
2020/04/13 06:32:43 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 3/30920 offset 128M/1.887T part size 64M
2020/04/13 06:32:44 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 4/30920 offset 192M/1.887T part size 64M
2020/04/13 06:32:45 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 5/30920 offset 256M/1.887T part size 64M
2020/04/13 06:32:46 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 6/30920 offset 320M/1.887T part size 64M
2020/04/13 06:32:48 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 7/30920 offset 384M/1.887T part size 64M
2020/04/13 06:32:50 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 8/30920 offset 448M/1.887T part size 64M
2020/04/13 06:33:06 DEBUG : 95/Backup95/Backup95D2020-04-03T210052.vbk: Uploading part 9/30920 offset 512M/1.887T part size 64M
2020/04/13 06:33:20 ERROR : 95/Backup95/Backup95D2020-04-03T210052.vbk: Failed to copy: multipart upload failed to upload part: -> github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /home/runner/work/rclone/src/github.com/rclone/rclone/vendor/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=InvalidBlobOrBlock) =====
Description=The specified blob or block content is invalid.
RequestId:1a357710-b01e-000d-294c-1151c4000000
Time:2020-04-13T04:33:20.9418119Z, Details:
Code: InvalidBlobOrBlock
PUT https://ipcveeam.blob.core.windows.net/veeam/GMT+02-2020.04.05-06.00.05/95/Backup95/Backup95D2020-04-03T210052.vbk?blockid=CQAAAAAAAAA%3D&comp=block&timeout=31536001
Authorization: REDACTED
Content-Length: [67108864]
Content-Md5: [zoKexzwyWC2syWl4p3LfGw==]
User-Agent: [rclone/v1.51.0]
X-Ms-Client-Request-Id: [e7877048-681e-42fc-6747-7c0288104d57]
X-Ms-Date: [Mon, 13 Apr 2020 04:33:06 GMT]
X-Ms-Version: [2018-11-09]
RESPONSE Status: 400 The specified blob or block content is invalid.
Content-Length: [234]
Content-Type: [application/xml]
Date: [Mon, 13 Apr 2020 04:33:20 GMT]
Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
X-Ms-Error-Code: [InvalidBlobOrBlock]
X-Ms-Request-Id: [1a357710-b01e-000d-294c-1151c4000000]
X-Ms-Version: [2018-11-09]
2020/04/13 06:33:24 INFO :
Transferred: 576.005M / 1.858 TBytes, 0%, 14.939 kBytes/s, ETA 4y12w23h13m47s
Errors: 1 (retrying may help)
Checks: 63 / 63, 100%
Transferred: 2 / 3, 67%
Elapsed time: 10h57m59.8s
Transferring:
- 95/Backup95/Backup95D2020-03-06T210059.vbk: 0% /1.857T, 0/s, -
Please advise.
I escalated to Microsoft support aswell, this is what they replied:
InvalidBlobOrBlock: “The specified blob or block content is invalid” can occur if the specified block size is not supported.
This error can also occur for various reasons but here is two of the most common reasons we see:
1 . The uncommitted block list for the blob is in a bad state. You will need to do one of the following:
-
Perform a GetBlockList (with blocklisttype=uncommitted) to retrieve the uncommitted block list, commit the block list, then delete the blob.
-
Create a dummy blob (can be of length zero) with the same blob name in the same container and transfer it using a non-blocked transfer
-
Wait 7 days for the uncommitted block list to garbage collect.
The issue may occurred because of the uncommitted blocks which were left from the previous upload operation. This means that such error comes when you’re trying to do concurrent upload commits. In such cases, the upload fails. To resolve this issue, we will have to delete the uncommitted blocks forcefully. If not, such uncommitted blocks might take up to 7 days to garbage collect.
You may able to see all the uncommitted blob, call perform the delete operation on it by making use of the Delete Blob Rest API. You can refer to the link: https://docs.microsoft.com/en-us/rest/api/storageservices/delete-blob
You can do either of (1) or (2) using an Azure Storage SDK, or you can perform (2) easily with a GUI-based tool like Azure Storage Explorer.
2. A timing/concurrency issue where the PUT operations are happening about the same time for a single blob. The Put Block List operation writes a blob by specifying the list of block IDs that make up the blob. In order to be written as part of a blob, a block must have been successfully written to the server in a prior Put Block operation.
Documentation reference:
https://docs.microsoft.com/en-us/rest/api/storageservices/put-block
This error can happen when doing concurrent upload commits after you have started the upload but before you commit. In that case, the upload fails. The application can retry this error or attempt some other recovery action based on the required scenario.
If you use REST API then, the maximum block size is 100 MB.
This will be valuable to rclone author I suppose.
The delay at the start of the upload is due to rclone calculating the checksum. However it is using that as a file integrity check sent as the x-ms-blob-content-md5 header. Rclone could skip this if required (there is a similar flag in the s3 backend which could be implemented for the azureblob backend)
--s3-disable-checksum Don't store MD5 checksum with object metadata
As for the other problem, it would be really helpful to see a complete log with the -vv flag. You can attach it here as a .log or .txt file, or email it to me nick@craig-wood.com with a link to this forum post.
Hi,
any news on this?
Sorry for the delay!
Thanks for the log - it was very helpful.
I made this which has two improvements.
- When it gets a
InvalidBlobOrBlockerror it should retry the chunk. Can you check this in the debug log with-vv- you should see the retries happening. - I added the --azureblob-disable-checksum flag - you can set this in the config as
disable_checksum = truealso.
Please let me know how the testing goes!
https://beta.rclone.org/branch/v1.51.0-229-g44f7204c-fix-azureblob-retry-beta/ (uploaded in 15-30 mins)
Hi,
so this is my command line now, trying with beta:
rclone --azureblob-disable-checksum --retries-sleep=5m --retries 3 --log-file=out.txt -vv sync ...source path... ...destination path... --transfers=8
- there is disable_checksum = true in rclone.conf
However there is still delay at start.
Now rclone broke, please take at log attached.out.txt (463.2 KB)
Looking at your log what appears to be happening is that the files you are syncing don't change in size. Is that right?
So what is happening is that rclone see's that their modification times are different, but their sizes are the same, so does a checksum test to see if the file has actually changed before uploading it.
As for the crash you ran out of memory apparently
runtime: out of memory: cannot allocate 67108864-byte block (1,542,356,992 in use)
fatal error: out of memory
Rclone was apparently using 1.5GB of memory...
Can you try this which has the previous patches plus another one which attempts to manage memory better (in the same way the s3 backend was patched recently)
https://beta.rclone.org/branch/v1.51.0-235-gf71f415d-fix-azureblob-retry-beta/ (uploaded in 15-30 mins)
Thanks!
Yes you are right, the files didnt change in size, neither in content, but their modificaiton time was different. No way to skip checksum check on this scenario?
Looks like its better now, rclone did whole sync without crash, no memory or other errors InvalidBlobOrBlock.
I'll try some more and let you know. Youll find the newest log here:
https://1drv.ms/u/s!AnWOHc0UWqgVqfoCRtY_MGnGXXeygA?e=7ocCB9
It is saving you uploading a 1TB file unnecessarily so it isn't all bad!
I think using the --no-check-dest flag will work and upload everything regardless! That might not be what you want though.
I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52
Let me know if it goes wrong again!
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.