Rclone repeats a copy files of a more than 7 GB

Help and Support
1

What is the problem you are having with rclone?

When I upload files on ftp. Files larger than 7GB are copied and after copying, rclone will copy those files again.

The example below shows that the entire backup is 8.278 GB, but one of the 6.gz backups is approximately 8.00 GB.

I don't know where is the problem because the same rclone setup on another server works perfectly fine.

Run the command 'rclone version' and share the full output of the command.

rclone v1.55.1-DEV

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.15.5
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

FTP

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /media/data/ FTP:ftp/backups/`date +"%Y-%m-%d"`/ --log-file=/root/.config/rclone/logs/ftp/`date +"%Y-%m-%d"`.log --log-level=INFO --ftp-disable-epsv

The rclone config contents with secrets removed.

[FTP]
type = ftp
host = XXX.XX.XXX.XX
user = user
port = 21
pass = XXXXXXXXXXXXXXXXXX

A log from the command with the -vv flag

2023/05/14 09:14:53 INFO  : data/1.tar.gz: Copied (new)
2023/05/14 09:14:53 INFO  : databases/2: Copied (new)
2023/05/14 09:14:54 INFO  : databases/3.gz: Copied (new)
2023/05/14 09:14:58 INFO  : databases/4.gz: Copied (new)
2023/05/14 09:15:00 INFO  : data/5.gz: Copied (new)
2023/05/14 09:15:52 INFO  :
Transferred:        1.215G / 8.278 GBytes, 15%, 20.766 MBytes/s, ETA 5m48s
Transferred:            5 / 6, 83%
Elapsed time:       1m0.6s
Transferring:
 *                            data/6.gz: 13% /8.188G, 20.905M/s, 5m45s

2023/05/14 09:19:52 INFO  :
Transferred:        6.222G / 8.278 GBytes, 75%, 21.247 MBytes/s, ETA 1m39s
Transferred:            5 / 6, 83%
Elapsed time:       5m0.6s
Transferring:
 *                            data/6.gz: 74% /8.188G, 21.421M/s, 1m38s

2023/05/14 09:20:52 INFO  :
Transferred:        7.387G / 8.278 GBytes, 89%, 21.017 MBytes/s, ETA 43s
Transferred:            5 / 6, 83%
Elapsed time:       6m0.6s
Transferring:
 *                            data/6.gz: 89% /8.188G, 19.733M/s, 46s

2023/05/14 09:21:52 INFO  :
Transferred:        8.567G / 16.466 GBytes, 52%, 20.892 MBytes/s, ETA 6m27s
Transferred:            5 / 6, 83%
Elapsed time:       7m0.6s
Transferring:
 *                            data/6.gz:  3% /8.188G, 20.928M/s, 6m26s
2

Try running with -vv this might give you more of a clue.

I suspect rclone has a problem at the end of the upload and retries it. The -vv log should tell you what.

3

That's also a really, really old version. You'd want to update.

4

Below debug log.

2023/05/14 13:13:10 INFO  :
Transferred:        7.616G / 8.023 GBytes, 95%, 14.446 MBytes/s, ETA 28s
Transferred:            0 / 1, 0%
Elapsed time:       9m1.6s
Transferring:
 *                      data/6.gz: 94% /8.023G, 14.977M/s, 27s

2023/05/14 13:13:41 DEBUG : ftp://XXX.XX.XX.XX:21/ftp/backups/2023-05-14/dataspace: Connecting to FTP server
2023/05/14 13:13:43 DEBUG : data/6.gz: Removed after failed upload: read tcp XXX.XX.XX.XX:46394->XXX.XX.XX.XX:21: i/o timeout
2023/05/14 13:13:43 DEBUG : data/6.gz: Received error: update stor: read tcp XXX.XX.XX.XX:46394->XXX.XX.XX.XX:21: i/o timeout - low level retry 1/10
2023/05/14 13:14:10 INFO  :
Transferred:        8.393G / 16.047 GBytes, 52%, 14.328 MBytes/s, ETA 9m6s
Transferred:            0 / 1, 0%
Elapsed time:      10m1.6s
Transferring:
 *                      data/6.gz:  4% /8.023G, 14.363M/s, 9m5s
5

Is that from v1.62.2?

You can see there is a timeout and rclone retries.

What causes the timeout I don't know.

6

Debugging was on version 1.55.1. After the update, the problem does not exist.

1 Like
7

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.