I try to use Rclone for copying backup to swift-storage of my hoster Selectel.ru
And faced with re-uploading of large files:
- rclone upload file
- then check file size, got zero size
- delete file and upload it again, and so three retries
I tried --ignore-size options, rclone doesn’t checked size and file was succesfully uploaded.
So i saw that manifest has zero size, and rclone fail compare it with real size.
I wonder is it bug of rclone or of my hoster.
UPD: i successfully used python-swiftclient to upload that backups to storage
Rclone version is v1.36
OS is Debian GNU/Linux 8.6 (jessie)
Here is log:
/usr/sbin/rclone --bwlimit 20M -vv copy /mnt/mysql-dump/full/od_garden/inventory.ibd selectel:vmok_dumps_weekly/test_new
2017/05/30 17:47:29 INFO : Starting bandwidth limiter at 20MBytes/s
2017/05/30 17:47:29 DEBUG : rclone: Version “v1.36” starting with parameters ["/usr/sbin/rclone" “–bwlimit” “20M” “-vv” “copy” “/mnt/mysql-dump/full/od_garden/inventory.ibd” “selectel:vmok_dumps_weekly/test_new”]
2017/05/30 17:47:29 INFO : Swift container vmok_dumps_weekly path test_new/: Modify window is 1ns
2017/05/30 17:47:30 INFO : Swift container vmok_dumps_weekly path test_new/: Waiting for checks to finish
2017/05/30 17:47:30 INFO : Swift container vmok_dumps_weekly path test_new/: Waiting for transfers to finish
2017/05/30 17:47:30 DEBUG : inventory.ibd: Uploading segment file “test_new/inventory.ibd/1496155650.569747506/6287261696/00000000” into “vmok_dumps_weekly_segments”
2017/05/30 17:48:29 INFO :
Transferred: 1.170 GBytes (19.840 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 1m0.3s
Transferring:
-
inventory.ibd: 19% done, 20.328 MBytes/s, ETA: 3m56s
2017/05/30 17:49:29 INFO :
Transferred: 2.342 GBytes (19.920 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 2m0.3s
Transferring:
-
inventory.ibd: 39% done, 20.000 MBytes/s, ETA: 2m59s
2017/05/30 17:50:29 INFO :
Transferred: 3.514 GBytes (19.947 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 3m0.3s
Transferring:
-
inventory.ibd: 60% done, 19.992 MBytes/s, ETA: 1m59s
2017/05/30 17:51:29 INFO :
Transferred: 4.678 GBytes (19.927 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 4m0.3s
Transferring:
-
inventory.ibd: 79% done, 20.052 MBytes/s, ETA: 1m0s
2017/05/30 17:51:46 DEBUG : inventory.ibd: Uploading segment file “test_new/inventory.ibd/1496155650.569747506/6287261696/00000001” into “vmok_dumps_weekly_segments”
2017/05/30 17:52:29 INFO :
Transferred: 5.842 GBytes (19.915 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 5m0.3s
Transferring:
-
inventory.ibd: 99% done, 19.644 MBytes/s, ETA: 0s
2017/05/30 17:52:31 ERROR : inventory.ibd: corrupted on transfer: sizes differ 6287261696 vs 0
2017/05/30 17:52:31 INFO : inventory.ibd: Removing failed copy
2017/05/30 17:52:31 DEBUG : inventory.ibd: Removing segment file “test_new/inventory.ibd/1496155650.569747506/6287261696/00000000” in container “vmok_dumps_weekly_segments”
2017/05/30 17:52:43 DEBUG : inventory.ibd: Removing segment file “test_new/inventory.ibd/1496155650.569747506/6287261696/00000001” in container “vmok_dumps_weekly_segments”
2017/05/30 17:52:45 DEBUG : inventory.ibd: Removed empty container “vmok_dumps_weekly_segments”
2017/05/30 17:52:45 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 6287261696 vs 0
2017/05/30 17:52:45 INFO : Swift container vmok_dumps_weekly path test_new/: Waiting for checks to finish
2017/05/30 17:52:45 INFO : Swift container vmok_dumps_weekly path test_new/: Waiting for transfers to finish
2017/05/30 17:52:45 DEBUG : inventory.ibd: Uploading segment file “test_new/inventory.ibd/1496155965.574948005/6287261696/00000000” into “vmok_dumps_weekly_segments”
2017/05/30 17:53:29 INFO :
Transferred: 6.734 GBytes (19.133 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 6m0.3s
Transferring:
-
inventory.ibd: 14% done, 20.772 MBytes/s, ETA: 4m5s
2017/05/30 17:54:29 INFO :
Transferred: 7.899 GBytes (19.240 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 7m0.3s
Transferring:
-
inventory.ibd: 34% done, 19.840 MBytes/s, ETA: 3m16s
2017/05/30 17:55:29 INFO :
Transferred: 9.072 GBytes (19.339 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 8m0.3s
Transferring:
-
inventory.ibd: 54% done, 20.120 MBytes/s, ETA: 2m14s
2017/05/30 17:56:29 INFO :
Transferred: 10.231 GBytes (19.388 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 9m0.3s
Transferring:
-
inventory.ibd: 74% done, 19.282 MBytes/s, ETA: 1m18s
2017/05/30 17:57:00 DEBUG : inventory.ibd: Uploading segment file “test_new/inventory.ibd/1496155965.574948005/6287261696/00000001” into “vmok_dumps_weekly_segments”
2017/05/30 17:57:29 INFO :
Transferred: 11.416 GBytes (19.471 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 10m0.3s
Transferring:
-
inventory.ibd: 94% done, 20.097 MBytes/s, ETA: 15s
2017/05/30 17:57:44 ERROR : inventory.ibd: corrupted on transfer: sizes differ 6287261696 vs 0
2017/05/30 17:57:44 INFO : inventory.ibd: Removing failed copy
2017/05/30 17:57:44 DEBUG : inventory.ibd: Removing segment file “test_new/inventory.ibd/1496155965.574948005/6287261696/00000000” in container “vmok_dumps_weekly_segments”
2017/05/30 17:57:57 DEBUG : inventory.ibd: Removing segment file “test_new/inventory.ibd/1496155965.574948005/6287261696/00000001” in container “vmok_dumps_weekly_segments”
2017/05/30 17:57:59 DEBUG : inventory.ibd: Removed empty container “vmok_dumps_weekly_segments”
2017/05/30 17:57:59 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 6287261696 vs 0
2017/05/30 17:57:59 INFO : Swift container vmok_dumps_weekly path test_new/: Waiting for checks to finish
2017/05/30 17:57:59 INFO : Swift container vmok_dumps_weekly path test_new/: Waiting for transfers to finish
2017/05/30 17:57:59 DEBUG : inventory.ibd: Uploading segment file “test_new/inventory.ibd/1496156279.630483732/6287261696/00000000” into “vmok_dumps_weekly_segments”
2017/05/30 17:58:29 INFO :
Transferred: 12.314 GBytes (19.095 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 11m0.3s
Transferring:
-
inventory.ibd: 10% done, 22.647 MBytes/s, ETA: 3m57s
2017/05/30 17:59:29 INFO :
Transferred: 13.488 GBytes (19.173 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 12m0.3s
Transferring:
-
inventory.ibd: 30% done, 19.957 MBytes/s, ETA: 3m29s
2017/05/30 18:00:29 INFO :
Transferred: 14.658 GBytes (19.234 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 13m0.3s
Transferring:
-
inventory.ibd: 50% done, 20.057 MBytes/s, ETA: 2m28s
2017/05/30 18:01:29 INFO :
Transferred: 15.830 GBytes (19.289 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 14m0.3s
Transferring:
-
inventory.ibd: 70% done, 19.917 MBytes/s, ETA: 1m29s
2017/05/30 18:02:14 DEBUG : inventory.ibd: Uploading segment file “test_new/inventory.ibd/1496156279.630483732/6287261696/00000001” into “vmok_dumps_weekly_segments”
2017/05/30 18:02:29 INFO :
Transferred: 17.002 GBytes (19.336 MBytes/s)
Errors: 0
Checks: 0
Transferred: 0
Elapsed time: 15m0.3s
Transferring:
-
inventory.ibd: 90% done, 20.000 MBytes/s, ETA: 28s
2017/05/30 18:02:58 ERROR : inventory.ibd: corrupted on transfer: sizes differ 6287261696 vs 0
2017/05/30 18:02:58 INFO : inventory.ibd: Removing failed copy
2017/05/30 18:02:58 DEBUG : inventory.ibd: Removing segment file “test_new/inventory.ibd/1496156279.630483732/6287261696/00000000” in container “vmok_dumps_weekly_segments”
2017/05/30 18:03:08 DEBUG : inventory.ibd: Removing segment file “test_new/inventory.ibd/1496156279.630483732/6287261696/00000001” in container “vmok_dumps_weekly_segments”
2017/05/30 18:03:10 DEBUG : inventory.ibd: Removed empty container “vmok_dumps_weekly_segments”
2017/05/30 18:03:10 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 6287261696 vs 0
2017/05/30 18:03:10 Failed to copy: corrupted on transfer: sizes differ 6287261696 vs 0