Rclone move running as cronjob - want to be able to access the progress/stats somehow

Help and Support
1

What is the problem you are having with rclone?

i have a cronjob running via the below command, and with slow upload speed, sometimes realize its still running. i'd like to be able to figure out what file its currently uploading, and how much time is left/current speed/current bytes remaining.

What is your rclone version (output from rclone version)

rclone v1.51.0-020-ga54210a2-beta

  • os/arch: linux/amd64
  • go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ubuntu 1804 lts x64

Which cloud storage system are you using? (eg Google Drive)

google drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

\\\/usr/bin/rclone move /media/transmission gd:/transmission --log-file /opt/rclone/logs/nightly.log -v --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs --fast-list --max-transfer 700G --progress --transfers 2

The rclone config contents with secrets removed.

not important for this purpose

i have looked through the other forum posts and have seen some people ask for something LIKE this, but some posts are pretty old, others don't seem to be too technically savvy.

i'm willing to not have the output be super cute, and don't care if i have to manually run something. i felt like --rc would be a good match for this, but as i kept trying different options and commands, i ended up getting what seems like VERY old transfers and stats. like last summer.

i don't know if the web gui would be well suited for this or not - it isn't something i need running all the time but i'm okay with it having to be if thats the case.

every night at 1:00AM i run the above command. it transfers the 20-50GB, and occasionally more, thats downloaded for the day. it can usually finish by 8-9AM...but in an instance where a season of a show has downloaded, or even worse yet - an entire show seasons 1-10 *or the worst was when star wars was released as 4k remuxes a few weeks ago, all at once :*

but in those cases, i'd like to query or at least gain some insight into....
oh man, its uploading an 80GB mkv, and its 94% done, only a little longer..
something like that.

i do occasionally check the nightly.log file that is created by the above command, but honestly most days i don't need to as the upload doesn't effect me, and my monitoring of everything gives mea pretty good idea of what needs uploaded that night/morning. but SOMETIMES i miss something, and realize a huge file is in process and it'll take a while..

thanks in advance

2

It'll only return stats for the current run. Rc is what you're looking for. Or just create a log file and look at what is doing that way.

3

am I correct in assuming that the output of the log file with -vv is the following:

2020/06/03 18:49:54 DEBUG : Movies/linuxISO.Remux-2160p.mkv: Sending chunk 79423340544 length 8388608
2020/06/03 18:50:00 DEBUG : Movies/linuxISO.Remux-2160p.mkv: Sending chunk 79431729152 length 8388608
2020/06/03 18:50:07 DEBUG : Movies/linuxISO.Remux-2160p.mkv: Sending chunk 79440117760 length 8388608

the "chunk 79440117760" is the byte count that the file is currently on - and the length "8388608" is the size of the individual piece its sending?

so i can parse this as ~79GB sent, in 8MiB pieces?

and if i wanted to, i can write a script to parse this output, and give me an approximate speed and transfer status of an individual file?

4

I highly recommend you create a log for transfers. For multiple reasons...

  1. Will let you see the current status with little delay.
    2)It allows you to go and check when things stop working and what is happening.

If it's stable enough INFO level should be enough, and you can configure how often you want the transfer status to be printed. But the biggest is being able to confirm if all files are transfered correctly and not experiencing any issues.

5

My command above as well as the output listed above is from the log file the job creates.

My issue is that I need a status of individual file, an approximate speed, and an approximate completion percentage.

If the log file is actually putting out bytes uploaded, I can parse that logfile and get a fairly decent output.

6

Yes you can but --rc might be better to get progress rather than scraping a log file.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.