Rclone sync max-age and delete-before issue

msalhab · February 24, 2025, 10:01am

What is the problem you are having with rclone?

I'm currently using rclone sync to backup files from local mount to a union of remote gdrives (gdrive1 and gdrive2), each having a capacity of 15GB, so total is 30GB with union.

on the /mnt/external/dump/dump/ I currently have the following files:

total 10297096
-rw-r--r-- 1 nobody nogroup 3500685348 Feb  9 01:02 vzdump-lxc-106-2025_02_09-02_00_06.tar.zst
-rw-r--r-- 1 nobody nogroup         31 Feb  9 01:02 vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes
-rw-r--r-- 1 nobody nogroup       1242 Feb  9 01:02 vzdump-lxc-106-2025_02_09-02_00_06.log
-rw-r--r-- 1 nobody nogroup 3505944304 Feb 16 01:02 vzdump-lxc-106-2025_02_16-02_00_02.tar.zst
-rw-r--r-- 1 nobody nogroup         31 Feb 16 01:02 vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes
-rw-r--r-- 1 nobody nogroup       1242 Feb 16 01:02 vzdump-lxc-106-2025_02_16-02_00_02.log
-rw-r--r-- 1 nobody nogroup 3537552849 Feb 23 01:02 vzdump-lxc-106-2025_02_23-02_00_06.tar.zst
-rw-r--r-- 1 nobody nogroup         31 Feb 23 01:02 vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes
-rw-r--r-- 1 nobody nogroup       1386 Feb 23 01:02 vzdump-lxc-106-2025_02_23-02_00_06.log

on the remote I have files created on the 9th and 16th of February.

As far as I understand, I expect the sync command to perform the following actions based on the current parameters:
1- fetch files under /mnt/external/dump/dump/ newer than 7 days -> so practically fetch files created on the 23rd of February
2- compare with the files in remote union
3- delete files in remote union and do not use the trash, to free up space
4- copy the files (fetched in step 1) to remote union

Currently, it doesn't delete the remote files, and stops the sync due to disk full issue

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.1

os/version: debian 12.9 (64 bit)
os/kernel: 6.8.12-7-pve (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.24.0
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync /mnt/external/dump/dump/ gdrive_union:/pvebackup/PVE2 --modify-window 2s --log-level DEBUG --delete-before --max-age 7d

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive_backup1]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 
use_trash = false

[gdrive_backup2]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
team_drive = 
use_trash = false

[gdrive_union]
type = union
action_policy = all
create_policy = mfs
search_policy = ff
upstreams = gdrive_backup1: gdrive_backup2:

A log from the command that you were trying to run with the `-vv` flag

2025/02/24 09:53:49 DEBUG : --max-age 1w to 2025-02-17 09:53:49.463014442 +0000 UTC m=-604799.940718260
2025/02/24 09:53:49 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "sync" "/mnt/external/dump/dump/" "gdrive_union:/pvebackup/PVE2" "--modify-window" "2s" "--log-level" "DEBUG" "--delete-before" "--max-age" "7d"]
2025/02/24 09:53:49 DEBUG : Creating backend with remote "/mnt/external/dump/dump/"
2025/02/24 09:53:49 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2025/02/24 09:53:49 DEBUG : fs cache: renaming cache item "/mnt/external/dump/dump/" to be canonical "/mnt/external/dump/dump"
2025/02/24 09:53:49 DEBUG : Creating backend with remote "gdrive_union:/pvebackup/PVE2"
2025/02/24 09:53:49 DEBUG : Creating backend with remote "gdrive_backup2:"
2025/02/24 09:53:49 DEBUG : Creating backend with remote "gdrive_backup1:"
2025/02/24 09:53:49 DEBUG : Google drive root '': 'root_folder_id = 0AF-5GsMSCqlJUk9PVA' - save this in the config to speed up startup
2025/02/24 09:53:49 DEBUG : Creating backend with remote "gdrive_backup1:pvebackup/PVE2"
2025/02/24 09:53:49 DEBUG : Google drive root '': 'root_folder_id = 0AD7jBVVU9YHWUk9PVA' - save this in the config to speed up startup
2025/02/24 09:53:49 DEBUG : Creating backend with remote "gdrive_backup2:pvebackup/PVE2"
2025/02/24 09:53:49 DEBUG : Google drive root 'pvebackup/PVE2': 'root_folder_id = 0AF-5GsMSCqlJUk9PVA' - save this in the config to speed up startup
2025/02/24 09:53:49 DEBUG : Google drive root 'pvebackup/PVE2': 'root_folder_id = 0AD7jBVVU9YHWUk9PVA' - save this in the config to speed up startup
2025/02/24 09:53:50 DEBUG : union root 'pvebackup/PVE2': actionPolicy = *policy.All, createPolicy = *policy.Mfs, searchPolicy = *policy.FF
2025/02/24 09:53:50 DEBUG : fs cache: renaming cache item "gdrive_union:/pvebackup/PVE2" to be canonical "gdrive_union:pvebackup/PVE2"
2025/02/24 09:53:50 DEBUG : Waiting for deletions to finish
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:50 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 09:53:50 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:50 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Need to transfer - File not found at Destination
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Size and modification time the same (differ by -696.899µs, within tolerance 2s)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Unchanged skipping
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Size and modification time the same (differ by -666.523µs, within tolerance 2s)
2025/02/24 09:53:51 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Unchanged skipping
2025/02/24 09:53:51 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 09:53:51 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 09:53:52 ERROR : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 09:53:52 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 09:53:52 DEBUG : Waiting for deletions to finish
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 09:53:52 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Need to transfer - File not found at Destination
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Size and modification time the same (differ by -696.899µs, within tolerance 2s)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Size and modification time the same (differ by -666.523µs, within tolerance 2s)
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Unchanged skipping
2025/02/24 09:53:52 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 09:53:52 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Unchanged skipping
2025/02/24 09:53:52 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 09:53:53 ERROR : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 09:53:53 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 09:53:53 DEBUG : Waiting for deletions to finish
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:53 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 09:53:53 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:53 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Need to transfer - File not found at Destination
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Size and modification time the same (differ by -696.899µs, within tolerance 2s)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Unchanged skipping
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Size and modification time the same (differ by -666.523µs, within tolerance 2s)
2025/02/24 09:53:54 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Unchanged skipping
2025/02/24 09:53:54 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 09:53:54 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 09:53:54 ERROR : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 09:53:54 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 09:53:54 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 6 / 6, 100%
Elapsed time:         4.3s

2025/02/24 09:53:54 DEBUG : 12 go routines active
2025/02/24 09:53:54 NOTICE: Failed to sync: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded

kapitainsky · February 24, 2025, 10:09am

You might have to add --delete-excluded but be careful. When things are gone they are gone sometimes forever:)

Read docs, use --dry-run when testing.

msalhab · February 24, 2025, 10:19am

Thanks @kapitainsky

I added the flag, it looks a little better, but not final yet. The files are now getting deleted, however when rclone attempted to upload, it reported that disk is full. When I ran it again, it uploaded succesfully. I suspect it's a matter of couple of seconds until google api reports that it has enough free space to upload. What do you think? If so, is there a retry mechanism to increase? or a wait parameter before trying to upload?

Here are the logs:

2025/02/24 10:12:07 DEBUG : --max-age 1w to 2025-02-17 10:12:07.723052175 +0000 UTC m=-604799.940244824
2025/02/24 10:12:07 DEBUG : rclone: Version "v1.69.1" starting with parameters ["rclone" "sync" "/mnt/external/dump/dump/" "gdrive_union:/pvebackup/PVE2" "--modify-window" "2s" "--log-level" "DEBUG" "--delete-before" "--max-age" "7d" "--delete-excluded"]
2025/02/24 10:12:07 DEBUG : Creating backend with remote "/mnt/external/dump/dump/"
2025/02/24 10:12:07 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2025/02/24 10:12:07 DEBUG : fs cache: renaming cache item "/mnt/external/dump/dump/" to be canonical "/mnt/external/dump/dump"
2025/02/24 10:12:07 DEBUG : Creating backend with remote "gdrive_union:/pvebackup/PVE2"
2025/02/24 10:12:07 DEBUG : Creating backend with remote "gdrive_backup2:"
2025/02/24 10:12:07 DEBUG : Creating backend with remote "gdrive_backup1:"
2025/02/24 10:12:07 DEBUG : Google drive root '': 'root_folder_id = 0AF-5GsMSCqlJUk9PVA' - save this in the config to speed up startup
2025/02/24 10:12:07 DEBUG : Creating backend with remote "gdrive_backup1:pvebackup/PVE2"
2025/02/24 10:12:07 DEBUG : Google drive root '': 'root_folder_id = 0AD7jBVVU9YHWUk9PVA' - save this in the config to speed up startup
2025/02/24 10:12:07 DEBUG : Creating backend with remote "gdrive_backup2:pvebackup/PVE2"
2025/02/24 10:12:08 DEBUG : Google drive root 'pvebackup/PVE2': 'root_folder_id = 0AF-5GsMSCqlJUk9PVA' - save this in the config to speed up startup
2025/02/24 10:12:08 DEBUG : Google drive root 'pvebackup/PVE2': 'root_folder_id = 0AD7jBVVU9YHWUk9PVA' - save this in the config to speed up startup
2025/02/24 10:12:08 DEBUG : union root 'pvebackup/PVE2': actionPolicy = *policy.All, createPolicy = *policy.Mfs, searchPolicy = *policy.FF
2025/02/24 10:12:08 DEBUG : fs cache: renaming cache item "gdrive_union:/pvebackup/PVE2" to be canonical "gdrive_union:pvebackup/PVE2"
2025/02/24 10:12:08 DEBUG : Waiting for deletions to finish
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:08 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 10:12:09 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 10:12:09 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 10:12:09 INFO  : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Deleted
2025/02/24 10:12:09 INFO  : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Deleted
2025/02/24 10:12:09 INFO  : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Deleted
2025/02/24 10:12:09 INFO  : vzdump-lxc-106-2025_02_16-02_00_02.log: Deleted
2025/02/24 10:12:09 INFO  : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Deleted
2025/02/24 10:12:10 INFO  : vzdump-lxc-106-2025_02_09-02_00_06.log: Deleted
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Need to transfer - File not found at Destination
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Size and modification time the same (differ by -696.899µs, within tolerance 2s)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Size and modification time the same (differ by -666.523µs, within tolerance 2s)
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Unchanged skipping
2025/02/24 10:12:10 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Unchanged skipping
2025/02/24 10:12:10 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 10:12:10 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 10:12:11 ERROR : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 10:12:11 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 10:12:11 DEBUG : Waiting for deletions to finish
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:11 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 10:12:12 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 10:12:12 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Need to transfer - File not found at Destination
2025/02/24 10:12:12 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Size and modification time the same (differ by -696.899µs, within tolerance 2s)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Size and modification time the same (differ by -666.523µs, within tolerance 2s)
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Unchanged skipping
2025/02/24 10:12:12 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Unchanged skipping
2025/02/24 10:12:12 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 10:12:13 ERROR : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 10:12:13 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 10:12:13 DEBUG : Waiting for deletions to finish
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 10:12:13 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 10:12:13 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.log: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.log: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst.notes: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_16-02_00_02.tar.zst.notes: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded (ModTime Filter)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_09-02_00_06.tar.zst: Excluded
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Need to transfer - File not found at Destination
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Size and modification time the same (differ by -696.899µs, within tolerance 2s)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.log: Unchanged skipping
2025/02/24 10:12:13 DEBUG : union root 'pvebackup/PVE2': Waiting for checks to finish
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Size and modification time the same (differ by -666.523µs, within tolerance 2s)
2025/02/24 10:12:13 DEBUG : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst.notes: Unchanged skipping
2025/02/24 10:12:13 DEBUG : union root 'pvebackup/PVE2': Waiting for transfers to finish
2025/02/24 10:12:14 ERROR : vzdump-lxc-106-2025_02_23-02_00_06.tar.zst: Failed to copy: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 10:12:14 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded
2025/02/24 10:12:14 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                12 / 12, 100%
Deleted:                6 (files), 0 (dirs), 6.525 GiB (freed)
Elapsed time:         5.7s

2025/02/24 10:12:14 DEBUG : 20 go routines active
2025/02/24 10:12:14 NOTICE: Failed to sync: googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded

kapitainsky · February 24, 2025, 10:24am

Free space values are valid or if not than forget it with such tight setup (space wise).

Union assumes that you have enough free space in time period set by --union-cache-time. But do not set it to 1s Google Drive will probably ban you shortly after.

Run in 2 steps with some delay in between - like in your tests. You can wrap all solution in shell script and add some free space validation before step 2.

msalhab · February 24, 2025, 1:41pm

Thanks @kapitainsky. So I've added cache_time = 30 to rclone.conf , so I force the free space refresh every 30 seconds instead of 120 seconds. I also added --retries-sleep=31s to retry every 31 seconds.

So far it's working. Thank you for your help.