Rclone sync deletes all files on remote when scheduled on OMV

Help and Support
1

What is the problem you are having with rclone?

Have set 2 rclone sync commands to run daily by OpenMediaVault to back up two local directories to another local debian computer.

When run on the CLI OR using the 'run' button on the scheduled tasks page in OMV, both work properly.

When OMV runs them automatically as scheduled, both commands delete files in the remote share - even when the files are still present on the Source. eg: I have Omada running on a docker container on the OMV computer. It backs up nightly to the local drive and keeps 7 days of backups on the local drive. When Rsync runs it should delete the oldest file from the remote folder and replace it with the most current backup file - this will match the contents on the backup folder on the OMV computer.

Running Rclone sync from the CLI or 'run' button on the scheduled tasks page of OMV does exactly this but if I leave it overnight to run Rclone sync as a scheduled task, it removes all the backup files on the remote folder machine and does not write any new file - it empties the directory.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.1

  • os/version: debian 11.7 (64 bit)
  • os/kernel: 6.1.0-0.deb11.7-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.6
  • go/linking: static
  • go/tags: none

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
--> YES

Which cloud storage system are you using? (eg Google Drive)

SMB share on other (internal networked) debian system

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync /docker/omada debianbackup:debian_hdd_backup/omv_backup/docker/omada  
```    AND

rclone sync /docker/mariadb debianbackup:debian_hdd_backup/omv_backup/docker/mariadb




#### The rclone config contents with secrets removed.  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

[googledrive]
type = drive
scope = drive
token = {"access_token":"ya29.a0AfB_byB7-t-ixxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxS>
team_drive =

[debianbackup]
type = smb
host = 192.168.0.120
user = xxx
pass = xxxxxxxxxxxxxxxxxxxxxx




#### A log from the command with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

unable to post as run as a scheduled (CRON) task. When run manually does not return error's - see above

2

Add log file flags and post rclone.log when run from crontab:

rclone sync /docker/omada debianbackup:debian_hdd_backup/omv_backup/docker/omada --log-file /path/to/rclone.log --log-level DEBUG

3

Thanks - have added the --log-file........... part to the command and will post the logs tomorrow after it has run using the OMV scheduler

4

might want to add --dry-run.
with that, the log file will list what files would be deleted without actually deleting them.

5

Thanks. Left it without dry-run as it is only a backup of a backup.

Last nights run worked without issue! Not sure why as nothing was altered apart from the log-file part. With continue to monitor and run with the log files to see what happens over the next few days.

6

Didn't touch anything on OMV yesterday and last nights scheduled tasks reverted to deleting everything on the remote again. Strangely, the most recent omada backupfile was sync'd and left but everything else on both jobs was deleted!
Logs files are huge site won't let me upload files so have uploaded links: (Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.) and (Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.)

7 
$ rclone sync "/docker/mariadb" "debianbackup:debian_hdd_backup/omv_backup/docker/mariadb

Transferred:   	    2.936 GiB / 2.936 GiB, 100%, 24.059 MiB/s, ETA 0s
Checks:               137 / 137, 100%
Transferred:           16 / 16, 100%
Elapsed time:      1m21.1s

$ rclone sync "/docker/omada" "debianbackup:debian_hdd_backup/omv_backup/docker/omada"

Transferred:   	  151.441 MiB / 151.441 MiB, 100%, 13.768 MiB/s, ETA 0s
Checks:              1242 / 1242, 100%
Deleted:               63 (files), 0 (dirs)
Transferred:          307 / 307, 100%
Elapsed time:        11.7s

None of the jobs show deleting everything. Some files are deleted, some new files are transferred.

Do you mean that everything is deleted and nothing is left?

8

Hi, its a bit inconsitent.
Omada: The source directory holds 7 backup files for the last 7 days. Yesterday the 7 files were there, today, the only file remaining is yesterday's backup file. The files in the other Omada directory all look OK and haven't been deleted.
MariaDB: This is a home assistant database. Yesterday, all the files were there. Today they have all been deleted.
I'm running another Scheduled task to backup the OMV backup files to the same remote Debian share. That is running perfectly. The only difference between this and the Omada/MariaDB set up is Omada and MariaDB also backup to a google drive by the same Scheduling service on OMV (The OMV backup is too big for my google drive so I only have a local backup). I can't see how that could be affecting the local backup though.

9

There is no known bug in rclone sync causing files existing in source to be deleted from destination.

Your explanation that when you run sync manually all is ok and only when you run it scheduled at night things go wrong tells me that it is not rclone sync issue.

It is not this scheduled rclone sync deleting your files but something else you run after.

Of course a new bug is always possibility. But for this you have to provide some solid evidence - ideally something reproducible.

10

after rclone sync 137 files were at destination - total 2.94GB

If now they are all gone it is not that rclone sync responsible IMO

11

OK, thanks for taking the time to look at it. doesn't really make sense as there is nothing else running to interfere with that remote directory (it was set up only for rsync). I'll fiddle around a bit more and see if it becomes more stable.

12

Maybe try to log as much as possible? As now it is a lot of description but not much solid data.

It is not clear at least for me which Omada directory has problem? Files are deleted from source one? Or from destination? What is other Omada directory?

It would help if e.g. your scheduled rclone sync records:

ls -l src_directory
rclone sync -vv --log file
ls -l  src_directory
ls -l  dst_directory

then we would know if something was really deleted. And it would be clear where.

13

At the end of the log you get a summary which shows what happened.

Transferred:      151.441 MiB / 151.441 MiB, 100%, 13.768 MiB/s, ETA 0s
Checks:              1242 / 1242, 100%
Deleted:               63 (files), 0 (dirs)
Transferred:          307 / 307, 100%
Elapsed time:        11.7s

Here you can see that there were 1242 checks which are files that rclone saw were in the right place and didn't need touching. 63 files were deleted and 307 new files were transferred.

So I'd say that looks OK.