B2 server side copy doesn't show cumulative progress

ncw · August 8, 2019, 9:09am

No worries! Rclone has a queue of objects which need to be copied. This queue can have up to 10,000 objects in by default (see --max-backlog). Rclone adds up the size of these objects to work out the size that needs to be copied.

Relatively recently rclone started accounting for server side copies too, so adding the amount copied by a server side copy into the stats (before this rclone only measured bytes actually read from the source).

In this bit of code

github.com

rclone/rclone/blob/d0c65b4c5ee2615d35a8727d9a80536cfbe3d0ed/fs/operations/operations.go#L279-L294


		// Try server side copy first - if has optional interface and
		// is same underlying remote
		actionTaken = "Copied (server side copy)"
		if doCopy := f.Features().Copy; doCopy != nil && (SameConfig(src.Fs(), f) || (SameRemoteType(src.Fs(), f) && f.Features().ServerSideAcrossConfigs)) {
			// Check transfer limit for server side copies
			if fs.Config.MaxTransfer >= 0 && accounting.Stats(ctx).GetBytes() >= int64(fs.Config.MaxTransfer) {
				return nil, accounting.ErrorMaxTransferLimitReached
			}
			newDst, err = doCopy(ctx, src, remote)
			if err == nil {
				dst = newDst
				accounting.Stats(ctx).Bytes(dst.Size()) // account the bytes for the server side transfer
			}
		} else {
			err = fs.ErrorCantCopy
		}

You can see rclone doing the server side copies and accounting for their transfers after the copy is complete (line 290). You'll note there is no mechanism for updating the transfer as it continues, it is either all done or not done. However rclone will turn this chunk of data copied into an estimate MB/s of the transfer rate and use that for estimating the remaining time to transfer.

It probably isn't terribly accurate for server side copies!