Rclone different behaviour when run through cron

Help and Support
1

What is the problem you are having with rclone?

The result is different when use run it from the command line and when you invoke it from crontab or fcrontab..

I have a bash script that runs two Python scripts and uploads two png outputs to a website using curl and the entire contents of the directory to Google Drive using Rclone. The first Python script generates five files (one Excel, two CSV, one PNG and one SVG) and the second two files (one PNG and one SVG). Curl uploads the two PNG to the website.

Rclone uploads only what have changed. In both cases:

  • The two Python script are only uploaded if they have changed.
  • The CSV and Excel files are always uploaded (even if identical).
  • The PNG and SVG are always uploaded if they are different.

However:

  • If I run the bash file from the command line the image files are uploaded if they have been newly generated even if they are identical. I guess it is checking the modification date and time.

  • However, when run from crontab or fcrontab the image files are only uploaded if they are different. I guess it is doing a checksum...

Now the question is, who is governing this behaviour Rclone or Google Drive?

Is there a way to control it?

The files are few and small so I do not mind having them overwritten every time.

What is your rclone version (output from rclone version)

rclone v1.36

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Operating System: Ubuntu 18.04.4 LTS
Kernel: Linux 4.19.112
Architecture: x86-64

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /tmp/ remote:tmp/

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

/usr/bin/rclone --config /home/user/.config/rclone/rclone.conf copy --no-traverse "/tmp/" "remote:tmp/"

2

The rclone version you are using is over two years old so you should update that.

If you run the command and use -vv, we can see why something is being updated. Add that to your command share the log. If you are using through cron you can add --log-file /tmp/somelog.log as well to capture the output to a file.

3

Thanks a million for the hints.

I have updated rclone and produced a very verbose log as instructed.

It turns out that, when run from cron, Rclone does not report realistic time deltas:

Size and modification time the same (differ by -329.96µs, within tolerance 1ms)

What puzzles me is that it is only from PNG and SVG files. Maybe from non-text files?

For text files it seems to be accurate.

The other thing is that, when run from the Bash shell, there is no problem.

Maybe an environmental variable I should define?

4

Can you share the log and point out the example of the issue you are seeing?

5

It seems I cannot upload anything. I will try to paste it.

Normal behaviour:

2020/04/04 14:52:15 DEBUG : Galicia_in_Coronavirusland.png: Modification times differ by -2m43.595329505s: 2020-04-04 14:52:07.519329505 +0200 CEST, 2020-04-04 12:49:23.924 +0000 UTC

Cron behaviour (those microseconds make no sense, it should be minutes)

2020/04/04 14:55:00 DEBUG : Galicia_in_Coronavirusland.png: Size and modification time the same (differ by -329.505µs, within tolerance 1ms)

6

I'm confused as to what the issue is.

You are trying to upload the same file twice? Can you do an example with 1 file and explain what you want to happen and what is happening.

From what I can tell, you are trying to upload the same file twice and it is telling you it hasn't changed so it just updates the time.

7

I have updated my answer above. Where it says that "differ by -329.505µs", that is nonsense. It should be several minutes.

8

Can we use one file and in cron just lsl the file, you can see what the timestamp is on it.

rclone lsl filename

felix@gemini:/etc$ rclone lsl hosts
2020/04/04 09:31:17 NOTICE: os-release: Can't follow symlink without -L/--copy-links
2020/04/04 09:31:17 NOTICE: rmt: Can't follow symlink without -L/--copy-links
2020/04/04 09:31:17 NOTICE: localtime: Can't follow symlink without -L/--copy-links
2020/04/04 09:31:17 NOTICE: mtab: Can't follow symlink without -L/--copy-links
      266 2020-03-27 08:31:26.338269312 hosts

rclone is just reading what the file system is reporting.

You can also stat the file as well.

felix@gemini:/etc$ stat /etc/hosts
  File: /etc/hosts
  Size: 266       	Blocks: 8          IO Block: 4096   regular file
Device: 812h/2066d	Inode: 268641068   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-04 08:37:41.341827437 -0400
Modify: 2020-03-27 08:31:26.338269312 -0400
Change: 2020-03-27 08:31:26.338269312 -0400
 Birth: -
9

Yes, you are right. I think that the problem is that the files are not being created when calling the scripts from cron. That is the problem. It has nothing to do with Rclone. Sorry about that.

10

So you got it figured out? I'm glad for that.

The best way to figure things out is just talking through them sometimes. Happy to help!

11

Yes, thanks a million! It was a Python problem with MatPlotLib. When called directly, by default, it expects a display to be available. From crontab there is no display. So I needed to add an option to my Python scripts in order to account for that fact...

1 Like
12

I'll play around a little more and see if I have another idea.

13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.