Rclone still initiating transfers after provided max duration

Help and Support
1

What is the problem you are having with rclone?

--max-duration field seems to be being ignored. I have not been able to find a reference as to how to format the input for it, but "1d" returns an error while "24h" does not.

If it helps there's over 100k files that are in the directory I'm trying to sync.

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0-DEV

  • os/version: rocky 8.5 (64 bit)
  • os/kernel: 4.18.0-348.2.1.el8_5.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.12
  • go/linking: dynamic
  • go/tags: none

(This was the RPM in the repos when I set up the system. Don't know why they're using a "dev" version)

Which cloud storage system are you using? (eg Google Drive)

SFTP, self managed server at remote datacenter

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Being run from crontab:

30  0  *  * 1-5 root /usr/bin/rclone sync /nfs/repo/ remotenfs01:/Array/nfs/repo --log-file /Array/datasync/logs/remote_repo_"`date +"\%d-\%m-\%Y"`".log --log-level=INFO --max-duration=24h --transfers=8

The rclone config contents with secrets removed.

[remotenfs01]
type = sftp
host = non.routable.internal.ip
user = root
key_file = a key file
known_hosts_file = a known_hosts file
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command with the -vv flag

If you really want it I'll provide it but I have to redact some portions. Suffice it to say that transfers were still being initiated after 24 hours of running. In fact transfers were still being started at the 32 hour mark.

2

Update the latest version and just run a small test with 1 minute or something and post a log as that's what we need to see.

3

hi,
as per rclone https://rclone.org/docs/#options, d is not a valid time unit
Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

as per the rclone log,
unknown unit "d" in duration "1d"

4

Can't update it right now, got 4 hours left on a transfer that has to finish today, get it to you later.

Current package: rclone-1.57.0-1.el8.x86_64

I'll try installing the 1.59 RPM as soon as I can and get that test run.

5

You can just download the binary, unzip, run from that location for a small test and that won't impact your running job.

6

Sorry this took so long, got a lot of production stuff I'm dealing with. Looks like the latest binary actually does hit the --max-duration point and stop, when I run it with --max-duration=10s, the old RPM just keeps going, the new binary produces this:

2022/10/13 11:50:07 DEBUG : 21 go routines active
2022/10/13 11:50:07 DEBUG : sftp://root@10.5.17.60:22//Array/nfs/qaelasticACES: Closing 2 unused connections
2022/10/13 11:50:07 Failed to sync: max transfer duration reached as set by --max-duration

So I guess it's fixed, I just need to upgrade from the one in the Enterprise Linux repo...and I hope they update their repos soon.

7

I always avoid repos as you really don't know what you are getting as it's up to a maintainer to do something.

rclone can selfupdate as well so it's a single binary to manager, super easy.

8

Yeah, but this is a production setup, there's security/audit requirements here. ALso, looks like they HAVEN'T updated the package, that's the latest one they have, and I'll have to yank it to use the generic RPM because they conflict. hat's Red Hat/Rocky's fault though, not yours. :slight_smile:

9

Right - with security / audit, by using an old repo version though, you are more likely to have open/already fixed CVEs and generally stuff not work. The bug list on the RedHat site shows a number of old CVEs on that version.

If you already have clearance to use rclone, grabbing it from the source repo should be fine as well.

Just use the official RPM -> Rclone downloads but remove the other first and should be ok I'd think.(

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.