SFTP move bloated transfer estimate

Suspected Bug
1

I am transferring data with rclone, using SFTP between 2 servers. The remote server has 6.3TB of data, according to du and df, ran on the remote server. However, on the local server, the transfer progress estimate, is many, many times that:
Transferred: 28.995G / 258.498 TBytes, 0%, 109.889 MBytes/s, ETA 685h5m59s
Transferred: 45.320G / 130.535 TBytes, 0%, 110.444 MBytes/s, ETA 344h8m17s

I am not sure why this is happening, as I have never seen this issue on rclone. Command I am using is:
rclone copy andy10gbit:/home/user . --verbose --stats=15s --transfers=10, with rclone v1.44

2

Those numbers look accurate to me. What are you expecting them to be?

image
image.png1324×664 83.6 KB

3

Sorry, I mean rclone reporting remote size is 130 TBytes, but it’s only 6.3TiB.

I seem to have resolved this, using the exclude: --exclude=.wine/**

I think there was an odd 2 files that rclone couldn’t calculate size for (and I don’t need wine folder anyways). Now I have:
Transferred: 20.916G / 2.831 TBytes, 1%, 178.346 MBytes/s, ETA 4h35m25s

4

What happens if you run a du -ms in the .wine directory? I’m curious what the OS reports back.

5

This is all I get:

server:~/.wine$ du -ms
43
6

Is is possible to run the .wine directly only with verbose output and see what file is messing it up?

7 
Transferred:       21.063G / 128.005 TBytes, 0%, 179.454 MBytes/s, ETA 207h43m50s
Errors:              5486 (retrying may help)
Checks:                 0 / 0, -
Transferred:        19240 / 29268, 66%
Elapsed time:      2m0.1s
Transferring:
 *                        dosdevices/z:/dev/fd/3: 17482006% /6.318k, 9.109M/s, -
 *                        dosdevices/z:/dev/full:  0% /0, 14.272M/s, -
 *                        dosdevices/z:/dev/zero:  0% /0, 9.147M/s, -
 *                     dosdevices/z:/dev/urandom:  0% /0, 8.012M/s, -
 *                    dosdevices/z:/dev/char/1:5:  0% /0, 13.021M/s, -
 *                    dosdevices/z:/dev/char/1:7:  0% /0, 13.594M/s, -
 *                    dosdevices/z:/dev/char/1:9:  0% /0, 7.956M/s, -
 *                dosdevices/z:/proc/10298/fd/23:  0% /0, 8.072M/s, -
 *                dosdevices/z:/proc/10380/fd/29:  0% /0, 11.498M/s, -
 *                dosdevices/z:/proc/10439/fd/29:  0% /0, 8.094M/s, -
 *              dosdevices/z:/proc/10298/pagemap:  0% /0, 8.789M/s, -
 *              dosdevices/z:/proc/10380/pagemap:  0% /0, 14.013M/s, -
 *              dosdevices/z:/proc/10439/pagemap:  0% /0, 13.462M/s, -
 *              dosdevices/z:/proc/12026/pagemap:  0% /0, 8.248M/s, -
 *     dosdevices/z:/home/v…ices/z:/dev/char/1:5:  0% /0, 13.390M/s, -
 *     dosdevices/z:/home/v…ices/z:/dev/char/1:7:  0% /0, 8.763M/s, -
 *     dosdevices/z:/home/v…ices/z:/dev/char/1:9:  0% /0, 8.025M/s, -
 *     dosdevices/z:/home/v…sdevices/z:/dev/full:  0% /0, 8.738M/s, -
 *     dosdevices/z:/home/v…sdevices/z:/dev/zero:  0% /0, 8.731M/s, -
 *     dosdevices/z:/home/v…vices/z:/dev/urandom:  0% /0, 8.029M/s, -

Looks to me like this is causing the issue: dosdevices/z:/dev/fd/3

8

What is the output if you stat the file?

9

Stat said no file. So I did an ls of the parent:

user@server:~$ ls -la .wine/dosdevices/z:/dev/fd/
total 0
dr-x------ 2 user user  0 Nov  3 03:02 .
dr-xr-xr-x 9 user user  0 Nov  3 03:02 ..
lrwx------ 1 user user 64 Nov  3 03:02 0 -> /dev/pts/2
lrwx------ 1 user user 64 Nov  3 03:02 1 -> /dev/pts/2
lrwx------ 1 user user 64 Nov  3 03:02 2 -> /dev/pts/2
lr-x------ 1 user user 64 Nov  3 03:02 3 -> /proc/11205/fd
10

I would guess you either have files with holes (sparse files), hardlinked files or mount bind directories.

Eg to create a 1PB file on your laptop

$ dd if=/dev/zero of=1PB count=0 bs=1G seek=1k
0+0 records in
0+0 records out
0 bytes copied, 0.000249287 s, 0.0 kB/s
$ ls -l 1PB 
-rw-rw-r-- 1 ncw ncw 1099511627776 Nov  3 17:20 1PB
$ du 1PB
0	1PB
$

rclone will see that as a 1PB file but du and df will see it as a 0 byte file!

I suspect you’ve got some of those files in your .wine directory - it is quite common to store disk images as sparse files with holes in.