Howto keep only last 10 backups on a Google Drive storage

bairog · September 15, 2024, 2:47pm

What is the problem you are having with rclone?

I'm trying to use rclone for backuping my database file. I need to copy it (only one file) to my Google Drive once a day (I've added a cron task for it). And I also add copy datetime to file name in a storage. But I want to keep only last 10 backups. Howto setup this behavior?

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.0

os/version: ubuntu 24.04 (64 bit)
os/kernel: 6.8.0-44-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.23.1
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy /var/www/MySite/data.db gdrive:/MySite/`date "+%Y.%m.%d_%H:%M:%S"`_data.db -v

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive.file
token = XXX

A log from the command that you were trying to run with the `-vv` flag

2024/09/15 17:40:19 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "/var/www/MySite/data.db" "gdrive:/MySite/2024.09.15_17:40:19_data.db" "-vv"]
2024/09/15 17:40:19 DEBUG : Creating backend with remote "/var/www/MySite/data.db"
2024/09/15 17:40:19 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/15 17:40:19 DEBUG : fs cache: adding new entry for parent of "/var/www/MySite/data.db", "/var/www/MySite"
2024/09/15 17:40:19 DEBUG : Creating backend with remote "gdrive:/MySite/2024.09.15_17:40:19_data.db"
2024/09/15 17:40:19 DEBUG : Google drive root 'MySite/2024.09.15_17:40:19_data.db': 'root_folder_id = root' - save this in the config to speed up startup
2024/09/15 17:40:20 DEBUG : fs cache: renaming cache item "gdrive:/MySite/2024.09.15_17:40:19_data.db" to be canonical "gdrive:MySite/2024.09.15_17:40:19_data.db"
2024/09/15 17:40:20 DEBUG : data.db: Need to transfer - File not found at Destination
2024/09/15 17:40:22 DEBUG : data.db: md5 = c83c9d6d2124ac33256a969b8f6b1cb1 OK
2024/09/15 17:40:22 INFO  : data.db: Copied (new)
2024/09/15 17:40:22 INFO  :
Transferred:        1.633 MiB / 1.633 MiB, 100%, 836 KiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         3.7s

2024/09/15 17:40:22 DEBUG : 8 go routines active

asdffdsa · September 15, 2024, 3:09pm

welcome to the forum,

tho, not a perfect solution as it deletes by date, not by count, assumes the cron job never fails.
rclone delete gdrive:MySite --min-age=10d --dry-run -vv

a safer version might be this. then manually prune as needed.
rclone move gdrive:MySite gdrive:MySite_archive --min-age=10d --dry-run -vv

a more complete solution to would be something like.

rclone lsf gdrive:MySite > all_files.txt to list all the files in the folder.
process all_files.txt, to create a subset of files to be deleted called delete_files.txt
rclone delete gdrive:MySite --include-from=delete-files.txt --dry-run -vv

bairog · September 16, 2024, 5:25am

Thank you, but that is not working for me.
In my storage have 1 file that was created early this morning (rclone lsf gdrive:MySite outputs 2024.09.16_03:00:01_data.db/). I've tried to delete files that were created more that 1 hour ago but command rclone delete gdrive:MySite --min-age=1h -vv doesn't delete any file. Isn't a problem that I rename my file when create a backup copy (data.db -> 2024.09.16_03:00:01_data.db)?

asdffdsa · September 16, 2024, 12:34pm

the answer is in the debug output.

--min-age uses modification time of the file, not the name of the file.
renaming the file does not change the modtime of the file.

to list filenames and the modtime, can use
rclone lsl gdrive:MySite --min-age=1h -vv

bairog · September 16, 2024, 2:14pm

Even rclone lsl gdrive:MySite -vv (without --min-age=1h) displays no files, while rclone lsf gdrive:MySite displays my file (2024.09.16_03:00:01_data.db/).
As you see every day I copy my file with a new name (via rclone copy) and I don't use rclone sync.
Maybe thats why rclone lsl is empty, maybe there is no modtime in this case?

asdffdsa · September 16, 2024, 2:20pm

no idea, cannot see into your machine.
need to post the debug output for each command??

bairog · September 16, 2024, 2:25pm

Sorry, of course you need logs

2024/09/16 17:22:19 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "lsl" "gdrive:MySite" "--min-age=1h" "-vv"]
2024/09/16 17:22:19 DEBUG : Creating backend with remote "gdrive:MySite"
2024/09/16 17:22:19 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/16 17:22:19 DEBUG : Google drive root 'MySite': 'root_folder_id = root' - save this in the config to speed up startup
2024/09/16 17:22:20 DEBUG : 8 go routines active

2024/09/16 17:24:19 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "lsf" "gdrive:MySite" "-vv"]
2024/09/16 17:24:19 DEBUG : Creating backend with remote "gdrive:MySite"
2024/09/16 17:24:19 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/16 17:24:19 DEBUG : Google drive root 'MySite': 'root_folder_id = root' - save this in the config to speed up startup
2024.09.16_03:00:01_data.db/
2024/09/16 17:24:20 DEBUG : 6 go routines active

asdffdsa · September 16, 2024, 2:34pm

tl;dr - if you want to rename a file whilst copying it, must use rclone copyto, then would see something like
INFO : data.db: Copied (new) to: 2024.09.16_03:00:01_data.db

sorry, not correct. check the debug output of your original copy command.
INFO : data.db: Copied (new)

2024.09.16_03:00:01_data.db/
that is a not a file, it is a folder, notice the trailing slash.

note: the destination of rclone copy is always a folder, never a file.
in effect, you asked rclone to create a new folder named 2024.09.16_03:00:01_data.db

bairog · September 16, 2024, 2:45pm

Oh, my mistake. I'm newbie to Linux so didn't notice that. Thank you, now after rclone copyto I can see my file with rclone lsl gdrive:MySite. Will check tomorrow (after cron task executes) if --min-age flag is working.

bairog · September 17, 2024, 5:51am

So now rclone lsl gdrive:MySite displays info about my file (created by cron at 03:00):
1712128 2024-07-03 17:55:38.000000000 2024.09.17_03:00:01_data.db.
As I can see --min-age flag is working by local file LastWriteTime (2024-07-03 17:55:38), not LastWriteTime of a file on a remote storage (2024.09.17 03:00:01). Looks like I will need to use that your recommendation:

Is that correct and I cannot make rclone with some flag work by remote file LastWriteTime or name?
And BTW what is the first number (1712128) in rclone lsl command result output?
Many thanks for your help.

asdffdsa · September 17, 2024, 12:40pm

can check the rclone docs.

filters, use as --min-age work on the source, not the dest.
the source can be local, can be remote, that is up to you and the command you run.

yes, can create filters based on name. check the rclone docs

system · October 17, 2024, 12:40pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.