Rclone uploads duplicates many times

Hi!

I'm having an issue for some time where rclone uploads the same file over and over to google drive. It creates copies of the files on the server, sometimes 20-40 copies.

I also caught my server spawning many rclone processes, sometimes a hundred.

I have no idea on how to go about error checking this as there is nothing in the logs.

I run it on linux Ubuntu 18.04, rclone v1.48.0.

My exact cronjob is:
/usr/bin/rclone copy /mnt/local/downloads/files/allfiles/transfer gdrive:Files/bmn -v --tpslimit 5 --transfers 2 >> /home/seed/logs/transferout.log 2>&1

I have tried to limit transfers as can be seen above as well as chaging the --tpslimit. No effect :frowning:

Any help appreciated. Its driving me crazy.

Can you pick one file and run a copy command and upload with -vv on it and share the log?

1 Like

Are you running rclone off a crontab? You need to make sure that one has stopped before the next one starts. A really easy way of doing this is with adding --rc to the command, but there are more sophisticated ways!

1 Like

I'll be sure to add -vv. Regretfully it doesn't happen all the time so I cannot easily replicate it. I'll paste the log when it happens again.

Thanks for chiming in!

Thanks for chiming in @ncw.

Yea i'm running it off crontab, but it only executes once every 24hrs.

I also had the problem when running off a script where I had several rclone jobs execute one after the other. I moved to crontab to see if it would help. The script also only ran once every 24 hrs.

How were you identifying hundreds running?

1 Like

Well normally I have some 240'ish processes on my machine. When it gets crazy with rclone it exceeds 500 and when I do pidof rclone it shows like alot.

Granted I haven't counted them all but it is alot, and the machine is also sluggish. I normally just kill them with killall.

What was your crontab entry? Were you able to validate it was the actual cron command going off all those times?

1 Like

The crontab entry is
* 03 * * * /usr/bin/rclone copy /mnt/local/downloads/files/allfiles/transfer gdrive:Files/bmn -v --tpslimit 5 --transfers 2 >> /home/seed/logs/transferout.log 2>&1

It works most of the time but then sometimes it just goes crazy.

I'll check the syslog to see if I can see multiple crontabs going off.

Thanks for coming up with suggestions @Animosity022.

That way that's written, you'd get one launched every minute at 3am which would explain the multiples.

So my upload runs at 3:12AM each night:


# Cloud Upload
12 3 * * * /opt/rclone/scripts/upload_cloud
1 Like

Right so it looks like I'm idiot. Thanks @Animosity022.

That would also explain why I only see the error intermittently - I have fiddled often with the crontab often to see if that makes any difference and I have on occasion changed the minute to an absolute number, which I will do again.

Thanks for the fast fault checking, its appreaciated!

1 Like