Rclone copy post-copy check logic with SMB

ptilmn · September 26, 2025, 12:43am

What is the problem you are having with rclone?

This is not necessarily a problem with rclone. I’d like to understand how post-copy check of checksums works with rclone copy when the destination is a mounted SMB folder.

I know that a checksum is generated during the reading of the source files, but I’m unsure how the checksum is calculated after writing to the destination completes. Since the destination is a mounted SMB folder does that mean the files are effectively downloaded, or read, from the destination through the SMB protocol just to compute the written checksum?

If so, I guess that the bandwidth is effectively halved since we’re doing both a read and write of the files at the destination.

As a bonus, I’d appreciate if someone with knowledge of the code can point me to where to find this logic.

Thanks!

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.1
- os/version: oracle 8.10 (64 bit)
- os/kernel: 4.18.0-553.16.1.el8_10.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.0
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Local storage but transferring from local computer folder to SMB mounted folder.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

sudo rclone copy /usr/local/myfolder/tmppp0/ /mnt/fire-smb/pete/tmppp0/ --no-check-dest --progress -vv --inplace

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

File not found - using defaults

A log from the command that you were trying to run with the `-vv` flag

2025/09/25 15:10:57 DEBUG : rclone: Version "v1.69.1" starting with parameters ["/bin/rclone" "copy" "/usr/local/myfolder/tmppp0/" "/mnt/fire-smb/pete/tmppp0/" "--no-check-dest" "--progress" "-vv" "--inplace"]
2025/09/25 15:10:57 DEBUG : Creating backend with remote "/usr/local/myfolder/tmppp0/"
2025/09/25 15:10:57 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
2025/09/25 15:10:57 DEBUG : fs cache: renaming cache item "/usr/local/myfolder/tmppp0/" to be canonical "/usr/local/myfolder/tmppp0"
2025/09/25 15:10:57 DEBUG : Creating backend with remote "/mnt/fire-smb/pete/tmppp0/"
2025/09/25 15:10:57 DEBUG : fs cache: renaming cache item "/mnt/fire-smb/pete/tmppp0/" to be canonical "/mnt/fire-smb/pete/tmppp0"
2025/09/25 15:10:57 DEBUG : 100355-0.bin: Need to transfer - File not found at Destination
2025/09/25 15:10:57 DEBUG : 100355-1.bin: Need to transfer - File not found at Destination
2025/09/25 15:10:57 DEBUG : 100355-2.bin: Need to transfer - File not found at Destination
[redacted due to size]
2025/09/25 15:12:02 DEBUG : 968093868-0.bin: md5 = eb0c8066235e35332fb720b4ca8768b8 OK
2025/09/25 15:12:02 INFO  : 968093868-0.bin: Copied (new)
Transferred:       46.851 GiB / 46.851 GiB, 100%, 740.329 MiB/s, ETA 0s
Transferred:          635 / 635, 100%
Elapsed time:       1m5.1s
2025/09/25 15:12:02 INFO  :
Transferred:       46.851 GiB / 46.851 GiB, 100%, 740.329 MiB/s, ETA 0s
Transferred:          635 / 635, 100%
Elapsed time:       1m5.1s

2025/09/25 15:12:02 DEBUG : 4 go routines active

nielash · September 27, 2025, 3:24am

Yes. Rclone still sees your smb dest as a local path, so it calculates hashes the same way it would for any other local path (i.e. manually). The SMB protocol part is handled by your OS, not rclone per se.

Note that this would be different if you were using the rclone smb backend, but it looks like you are not.

Not necessarily, because rclone reads and hashes the file while it is copying it, and then caches that hash internally if successful.

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          	// Calculate the hash of the object we are reading as we go along
          	if hasher != nil {
          		in = io.TeeReader(in, hasher)
          	}

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          	// All successful so update the hashes
          	if hasher != nil {
          		o.fs.objectMetaMu.Lock()
          		o.hashes = hasher.Sums()
          		o.fs.objectMetaMu.Unlock()
          	}

github.com/rclone/rclone

fs/operations/copy.go

8ed55c61e


      
          	// Verify hashes are the same after transfer - ignoring blank hashes
          	if c.hashType != hash.None {
          		// checkHashes has logs and counts errors
          		equal, _, srcSum, dstSum, _ := checkHashes(ctx, c.src, newDst, c.hashType)
          		if !equal {
          			return fmt.Errorf("corrupted on transfer: %v hashes differ src(%s) %q vs dst(%s) %q", c.hashType, c.src.Fs(), srcSum, newDst.Fs(), dstSum)
          		}
          	}
          	return nil

Which leads to here:

github.com/rclone/rclone

fs/operations/operations.go

8ed55c61e


      
          	g.Go(func() (err error) {
          		dstHash, dstErr = dst.Hash(ctx, ht)
          		if dstErr != nil {
          			return dstErr
          		}
          		if dstHash == "" {
          			fs.Debugf(dst, "Dst hash empty - aborting Src hash check")
          			return errNoHash
          		}
          		return nil
          	})

Which (for the local backend) leads to here:

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          // Hash returns the requested hash of a file as a lowercase hex string
          func (o *Object) Hash(ctx context.Context, r hash.Type) (string, error) {

As discussed earlier, if the internal object has cached hashes, those are returned:

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          		} else {
          			return "", fmt.Errorf("hash: failed to stat: %w", err)
          		}
          	} else {
          		o.fs.objectMetaMu.RLock()
          		changed = !o.modTime.Equal(oldtime) || oldsize != o.size
          		o.fs.objectMetaMu.RUnlock()
          	}
          
          	o.fs.objectMetaMu.RLock()
          	hashValue, hashFound := o.hashes[r]
          	o.fs.objectMetaMu.RUnlock()
          
          	if changed || !hashFound {
          		var in io.ReadCloser
          
          		if !o.translatedLink {
          			var fd *os.File
          			fd, err = file.Open(o.path)
          			if fd != nil {
          				in = newFadviseReadCloser(o, fd, 0, 0)

The dst hash we read while copying is then compared to the src hash, and if they are different, it will error:

github.com/rclone/rclone

fs/operations/copy.go

8ed55c61e


      
          		if !equal {
          			return fmt.Errorf("corrupted on transfer: %v hashes differ src(%s) %q vs dst(%s) %q", c.hashType, c.src.Fs(), srcSum, newDst.Fs(), dstSum)
          		}

ptilmn · September 27, 2025, 6:35pm

Thanks for the details!

So while a file is being read from src, to copy into the destination, the read bytes are hashed, and then that hash is stored in the dst object to be used later in the checkHashes function?

I didn’t see where the src object gets its hash from. Does it get computed from scratch by reading it again during this call?

github.com/rclone/rclone

fs/operations/operations.go

8ed55c61e


      
          
          var errNoHash = errors.New("no hash available")
          
          // checkHashes does the work of CheckHashes but takes a hash.Type and
          // returns the effective hash type used.
          func checkHashes(ctx context.Context, src fs.ObjectInfo, dst fs.Object, ht hash.Type) (equal bool, htOut hash.Type, srcHash, dstHash string, err error) {
          	// Calculate hashes in parallel
          	g, ctx := errgroup.WithContext(ctx)
          	var srcErr, dstErr error
          	g.Go(func() (err error) {
          		srcHash, srcErr = src.Hash(ctx, ht)
          		if srcErr != nil {
          			return srcErr
          		}
          		if srcHash == "" {
          			fs.Debugf(src, "Src hash empty - aborting Dst hash check")
          			return errNoHash
          		}
          		return nil
          	})
          	g.Go(func() (err error) {

nielash · September 27, 2025, 9:04pm

Yes that's basically it.

The src should also be hashed while we're reading it (but separately from the dst) and then cached on the object:

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          			return 0, fserrors.NoLowLevelRetryError(fmt.Errorf("can't copy - source file is being updated (size changed from %d to %d)", oldsize, fi.Size()))
          		}
          		if !oldtime.Equal(readTime(file.o.fs.opt.TimeType, fi)) {
          			return 0, fserrors.NoLowLevelRetryError(fmt.Errorf("can't copy - source file is being updated (mod time changed from %v to %v)", oldtime, fi.ModTime()))
          		}
          	}
          
          	n, err = file.in.Read(p)
          	if n > 0 {
          		// Hash routines never return an error
          		_, _ = file.hash.Write(p[:n])
          	}
          	return
          }
          
          // Close the object and update the hashes
          func (file *localOpenFile) Close() (err error) {
          	err = file.in.Close()
          	if err == nil {
          		if file.hash.Size() == file.o.Size() {
          			file.o.fs.objectMetaMu.Lock()

Note that the src.Hash method there ends up at the same Hash function as the dst I linked to earlier, just with a different object:

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          // Hash returns the requested hash of a file as a lowercase hex string
          func (o *Object) Hash(ctx context.Context, r hash.Type) (string, error) {

But that's essentially a coincidence of doing a local-to-local transfer (it could just as easily have been dropbox-to-local or something). So the src and dst are not really directly aware of each other.

For completeness' sake, I suppose I should mention that there's a slight exception to this on macOS, where there's an equivalent of server-side copying:

github.com/rclone/rclone

backend/local/clone_darwin.go

8ed55c61e


      
          
          // Copy src to this remote using server-side copy operations.
          //
          // # This is stored with the remote path given
          //
          // # It returns the destination Object and a possible error
          //
          // Will only be called if src.Fs().Name() == f.Name()
          //
          // If it isn't possible then return fs.ErrorCantCopy
          func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
          	if runtime.GOOS != "darwin" || f.opt.NoClone {
          		return nil, fs.ErrorCantCopy
          	}
          	srcObj, ok := src.(*Object)
          	if !ok {
          		fs.Debugf(src, "Can't clone - not same remote type")
          		return nil, fs.ErrorCantCopy
          	}
          	if f.opt.TranslateSymlinks && srcObj.translatedLink { // in --links mode, use cloning only for regular files
          		return nil, fs.ErrorCantCopy

ptilmn · September 28, 2025, 4:47pm

I think I get it now. Please correct me if I’m wrong.

The src file is opened with its own instance of a hasher. Whenever any bytes are read from it, they get passed to this hasher as well. When the src file is closed, the final hash is computed and cached in the src object to be used later in the Hash function.

The Update function is used to copy the src data to the dst. In this function, a new hasher instance is created. As the src bytes are read, they are also passed to this hasher instance. After the copy is complete, the final hash is computed and cached in the dst object to be used in the Hash function.

The checkHashes function then calls the Hash function on both src and dst, which returns the cached value, to compare them.

In this case of local to local copy, hashing is done twice on the same exact data that gets read from the src file. Would it be accurate to say then that for local to local copy the hashing is not providing any value since it gets computed from the same byte buffers for both src and dst?

nielash · September 29, 2025, 9:20pm

Yes, that's right.

I wouldn't say it's not providing any value. It is potentially valuable in the rare case when a file gets corrupted during transfer, or the source file is modified while it's being read. There's a test here that illustrates this (although we disabled it because it's racy. It's actually quite hard to cause corruption on purpose!)

github.com/rclone/rclone

backend/local/local_internal_test.go

8ed55c61e


      
          // Test corrupted on transfer
          // should error due to size/hash mismatch
          func TestVerifyCopy(t *testing.T) {
          	t.Skip("FIXME this test is unreliable")
          	r := fstest.NewRun(t)
          	filePath := "sub dir/local test"
          	r.WriteFile(filePath, "some content", time.Now())
          	src, err := r.Flocal.NewObject(context.Background(), filePath)
          	require.NoError(t, err)
          	src.(*Object).fs.opt.NoCheckUpdated = true
          
          	for i := range 100 {
          		go r.WriteFile(src.Remote(), fmt.Sprintf("some new content %d", i), src.ModTime(context.Background()))
          	}
          	_, err = operations.Copy(context.Background(), r.Fremote, nil, filePath+"2", src)
          	assert.Error(t, err)
          }

ptilmn · September 29, 2025, 9:47pm

Maybe I’m missing something, but how is this possible to detect? I thought the reading of the src happens once, and then gets passed to both instances of the hasher.

Doesn’t the p byte array here, also get passed through the io.TeeReader to the destination’s hasher instance?

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          		oldsize := file.o.size
          		file.o.fs.objectMetaMu.RUnlock()
          		if oldsize != fi.Size() {
          			return 0, fserrors.NoLowLevelRetryError(fmt.Errorf("can't copy - source file is being updated (size changed from %d to %d)", oldsize, fi.Size()))
          		}
          		if !oldtime.Equal(readTime(file.o.fs.opt.TimeType, fi)) {
          			return 0, fserrors.NoLowLevelRetryError(fmt.Errorf("can't copy - source file is being updated (mod time changed from %v to %v)", oldtime, fi.ModTime()))
          		}
          	}
          
          	n, err = file.in.Read(p)
          	if n > 0 {
          		// Hash routines never return an error
          		_, _ = file.hash.Write(p[:n])
          	}
          	return
          }
          
          // Close the object and update the hashes
          func (file *localOpenFile) Close() (err error) {
          	err = file.in.Close()

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          				}
          			}
          		}
          		out = f
          	} else {
          		out = nopWriterCloser{&symlinkData}
          	}
          
          	// Calculate the hash of the object we are reading as we go along
          	if hasher != nil {
          		in = io.TeeReader(in, hasher)
          	}
          
          	_, err = io.Copy(out, in)
          	closeErr := out.Close()
          	if err == nil {
          		err = closeErr
          	}
          
          	if o.translatedLink {
          		if err == nil {

I figure the io.Copy on line 1465 is the one and only time the reading happens of the src file resulting in the same byte array getting passed to both hasher instances, in addition to being copied to the dst file.

ptilmn · September 29, 2025, 11:06pm

I see that this gets detected due to the logic in the Hash function that looks for whether the file changed. In this case it looks like both the src and the dst file may get read again for the sake of hashing.

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          	} else {
          		o.fs.objectMetaMu.RLock()
          		changed = !o.modTime.Equal(oldtime) || oldsize != o.size
          		o.fs.objectMetaMu.RUnlock()
          	}
          
          	o.fs.objectMetaMu.RLock()
          	hashValue, hashFound := o.hashes[r]
          	o.fs.objectMetaMu.RUnlock()
          
          	if changed || !hashFound {
          		var in io.ReadCloser
          
          		if !o.translatedLink {
          			var fd *os.File
          			fd, err = file.Open(o.path)
          			if fd != nil {
          				in = newFadviseReadCloser(o, fd, 0, 0)
          			}
          		} else {
          			in, err = o.openTranslatedLink(0, -1)

However, when this is not the case then I don’t think the cached hash value will ever be different as I wrote in my previous post.

Is this “changed” case that you were thinking of that would still provide value?

nielash · September 30, 2025, 3:37am

I see what you mean. I think the key is here:

github.com/rclone/rclone

backend/local/local.go

8ed55c61e


      
          		if file.hash.Size() == file.o.Size() {
          			file.o.fs.objectMetaMu.Lock()
          			file.o.hashes = file.hash.Sums()
          			file.o.fs.objectMetaMu.Unlock()
          		}

So, on the src side, if we didn't read and hash exactly the expected number of bytes, we don't cache the hashes. That will cause o.Hash() to recalculate fresh ones, instead of using the cached ones (in the part of the code you cited).

I suppose that leaves open a small possibility that we read the right number of bytes but they are the wrong bytes somehow, and then we still cache the hash and don't recheck it... but maybe that's by design. Maybe @ncw can chime in if I'm missing something (I didn't write this part of the code!)

I think that would be true for the src but not the dst. I think the dst would always have a cached hash at this point, so hashFound should always be true (but you are right that changed could be true, in which case it would recheck.)

ptilmn · September 30, 2025, 5:39pm

Interesting, I hadn’t noticed that if statement before. I think I have a solid grasp of how these pieces fit together now. Thanks for you time and patience!

system · October 3, 2025, 5:39pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.

What is the problem you are having with rclone?

Run the command 'rclone version' and share the full output of the command.

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

A log from the command that you were trying to run with the -vv flag

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command that you were trying to run with the `-vv` flag