What is the problem you are having with rclone?
I'm working on using rclone for backups and wondering can I give rclone permission to read and write from the remote, but not modify/delete. If this were possible it gives strong immutable storage guarantees, since data can never be deleted by the agent. Restricting permissions like this would achieve the same thing as using Google Cloud Storage's Retention Periods to guarantee that no data is modified/deleted for a certain amount of time (lifecycle operations also allow me to clean up older items too, so rclone doesn't need to worry about that either). However whether these guarantees are enforced by limiting the permissions that rclone has or by using a Rentention Period, rclone copy appears to always attempt to delete/modify a file which has changed on local, failing when it doesn't have permissions to do this.
I'm using incremental backups, since the total size of the backup is very large, and the daily increment much much smaller. rclone copy --no-traverse --max-age
does the trick as it only copies recent files and it does it efficiently. It works great even if the rclone credentials don't have modify/delete permissions, new files are created on the remote as they appear in the source. However if a file does change on the source then rclone fails with no permissions because it tries to do a delete and then a re-upload, this is true when I use --suffix
as well.
Does rclone support taking backups in this situation, where it doesn't have permission to modify or delete from the remote, only read and write? Maybe by always appending a timestamp to the filename on the remote and saving multiple copies when things change? Or else something like --backup-dir
but allowing for incremental backups?
What is your rclone version (output from rclone version
)
rclone --version
rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none
-->
Which OS you are using and how many bits (eg Windows 7, 64 bit)
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
Which cloud storage system are you using? (eg Google Drive)
Google Cloud Storage
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy --no-traverse --suffix=`date "+-%Y-%m-%d-%H:%M"` --suffix-keep-extension /tmp/test-backup incremental-encrypted: --max-age 48h
The rclone config contents with secrets removed.
[offsite-backups]
type = google cloud storage
project_number = <redacted>
service_account_file = /root/.rclone/backup-agent-credentials.json
anonymous = false
object_acl = private
bucket_acl = private
bucket_policy_only = true
location = europe-west2
storage_class = COLDLINE
[incremental-encrypted]
type = crypt
remote = offsite-backups:incremental
filename_encryption = off
directory_name_encryption = false
password = <redacted>
password2 = <redacted>
A log from the command with the -vv
flag
# rclone copy --no-traverse --suffix=`date "+-%Y-%m-%d-%H:%M"` --suffix-keep-extension test-backup incremental-encrypted: -vv --max-age 48h
2021/04/27 14:50:04 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/04/27 14:50:04 DEBUG : --max-age 2d to 2021-04-25 14:50:04.975345942 +0100 BST m=-172799.984532566
2021/04/27 14:50:04 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "--no-traverse" "--suffix=-2021-04-27-14:50" "--suffix-keep-extension" "test-backup" "incremental-encrypted:" "-vv" "--max-age" "48h"]
2021/04/27 14:50:04 DEBUG : Creating backend with remote "test-backup"
2021/04/27 14:50:04 DEBUG : fs cache: renaming cache item "test-backup" to be canonical "/root/test-backup"
2021/04/27 14:50:04 DEBUG : Creating backend with remote "incremental-encrypted:"
2021/04/27 14:50:05 DEBUG : Creating backend with remote "offsite-backups:incremental"
2021/04/27 14:50:05 DEBUG : pacer: Reducing sleep to 6.522509ms
2021/04/27 14:50:05 DEBUG : hello.txt: Sizes differ (src 19 vs dst 15)
2021/04/27 14:50:05 DEBUG : hello2.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/04/27 14:50:05 DEBUG : hello2.txt: Unchanged skipping
2021/04/27 14:50:05 DEBUG : Encrypted drive 'incremental-encrypted:': Waiting for checks to finish
2021/04/27 14:50:05 DEBUG : pacer: Reducing sleep to 0s
2021/04/27 14:50:05 INFO : hello.txt: Copied (server-side copy) to: hello-2021-04-27-14:50.txt
2021/04/27 14:50:05 ERROR : hello.txt: Couldn't delete: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object., forbidden
2021/04/27 14:50:05 DEBUG : Encrypted drive 'incremental-encrypted:': Waiting for transfers to finish
2021/04/27 14:50:05 ERROR : Attempt 1/3 failed with 1 errors and: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to the Google Cloud Storage object., forbidden
2021/04/27 14:50:05 DEBUG : hello.txt: Sizes differ (src 19 vs dst 15)
2021/04/27 14:50:05 DEBUG : hello2.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/04/27 14:50:05 DEBUG : hello2.txt: Unchanged skipping
2021/04/27 14:50:05 DEBUG : Encrypted drive 'incremental-encrypted:': Waiting for checks to finish
2021/04/27 14:50:05 ERROR : hello.txt: Failed to copy: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to incremental/hello-2021-04-27-14:50.txt.bin., forbidden
2021/04/27 14:50:05 ERROR : hello.txt: Not deleting source as copy failed: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to incremental/hello-2021-04-27-14:50.txt.bin., forbidden
2021/04/27 14:50:05 DEBUG : Encrypted drive 'incremental-encrypted:': Waiting for transfers to finish
2021/04/27 14:50:05 ERROR : Attempt 2/3 failed with 1 errors and: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to <redacted>-incremental/hello-2021-04-27-14:50.txt.bin., forbidden
2021/04/27 14:50:05 DEBUG : pacer: Reducing sleep to 9.051459ms
2021/04/27 14:50:05 DEBUG : Encrypted drive 'incremental-encrypted:': Waiting for checks to finish
2021/04/27 14:50:05 DEBUG : hello.txt: Sizes differ (src 19 vs dst 15)
2021/04/27 14:50:05 DEBUG : hello2.txt: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/04/27 14:50:05 DEBUG : hello2.txt: Unchanged skipping
2021/04/27 14:50:05 DEBUG : pacer: Reducing sleep to 0s
2021/04/27 14:50:05 ERROR : hello.txt: Failed to copy: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to incremental/hello-2021-04-27-14:50.txt.bin., forbidden
2021/04/27 14:50:05 ERROR : hello.txt: Not deleting source as copy failed: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to incremental/hello-2021-04-27-14:50.txt.bin., forbidden
2021/04/27 14:50:05 DEBUG : Encrypted drive 'incremental-encrypted:': Waiting for transfers to finish
2021/04/27 14:50:05 ERROR : Attempt 3/3 failed with 1 errors and: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to incremental/hello-2021-04-27-14:50.txt.bin., forbidden
2021/04/27 14:50:05 INFO :
Transferred: 15 / 15 Bytes, 100%, 30 Bytes/s, ETA 0s
Errors: 1 (retrying may help)
Checks: 10 / 10, 100%
Deleted: 1 (files), 0 (dirs)
Transferred: 1 / 1, 100%
Elapsed time: 0.9s
2021/04/27 14:50:05 DEBUG : 4 go routines active
2021/04/27 14:50:05 Failed to copy: googleapi: Error 403: offsite-backup-agent@<redacted>.iam.gserviceaccount.com does not have storage.objects.delete access to incremental/hello-2021-04-27-14:50.txt.bin., forbidden