Backup changing files from a live server

janosch · January 11, 2024, 9:01am

What is the problem you are having with rclone?

I would like to backup a server while in use. It has some busy files that keep changing all the time, like sqlite databases or logfiles. rclone sync keeps failing on those with corrupted on transfer: sizes differ. I am looking for a solution only for log files at the moment, which are typically only appended to or rotated.

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1-DEV
- os/version: debian 11.7 (64 bit)
- os/kernel: 4.19.0-17-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none

It is the version that ships with debian bullseye. I also built the current git version and the behavior is still the same:

rclone v1.66.0-beta.7604.1045f5412
- os/version: debian 11.7 (64 bit)
- os/kernel: 4.19.0-17-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Hidrive + encrypt

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync -vv --links --inplace --stats-log-level NOTICE --progress /var/log/auth.log hidrive-enc:root/var/log/auth.log

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[hidrive]
type = hidrive
disable_fetching_member_count = true
token = XXX

[hidrive-enc]
type = crypt
remote = hidrive:some/redacted/path/here
password = XXX
password2 = XXX

A log from the command that you were trying to run with the `-vv` flag

2024/01/11 09:48:25 DEBUG : rclone: Version "v1.66.0-beta.7604.1045f5412" starting with parameters ["/home/jan/src/git/rclone/rclone" "sync" "-vv" "--links" "--inplace" "--stats-log-level" "NOTICE" "--progress" "/var/log/auth.log" "hidrive-enc:root/var/log/auth.log"]
2024/01/11 09:48:25 DEBUG : Creating backend with remote "/var/log/auth.log"
2024/01/11 09:48:25 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/01/11 09:48:25 DEBUG : local: detected overridden config - adding "{b6816}" suffix to name
2024/01/11 09:48:25 DEBUG : fs cache: adding new entry for parent of "/var/log/auth.log", "local{b6816}:/var/log"
2024/01/11 09:48:25 DEBUG : Creating backend with remote "hidrive-enc:root/var/log/auth.log"
2024/01/11 09:48:25 DEBUG : Creating backend with remote "hidrive:users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k"
2024/01/11 09:48:25 DEBUG : auth.log: Sizes differ (src 115111614 vs dst 115108976)
2024/01/11 09:48:25 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Uploading with chunks of size 50331648 and 4 transfers in parallel at path '/users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k/c8t8p90qhvi87hd32evbvgkv2k'.
2024/01/11 09:48:37 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Done uploading chunk of size 50331648 at offset 0.
2024/01/11 09:48:39 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Done uploading chunk of size 14476690 at offset 100663296.
2024/01/11 09:48:44 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Done uploading chunk of size 50331648 at offset 50331648.
2024/01/11 09:48:44 DEBUG : auth.log: hidrive = 8f60d43fd37e7c833b736bc391d8d03dbd000216 OK
2024/01/11 09:48:44 ERROR : auth.log: corrupted on transfer: sizes differ 115111614 vs 115111842
2024/01/11 09:48:44 INFO  : auth.log: Removing failed copy
2024/01/11 09:48:44 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 115111614 vs 115111842
2024/01/11 09:48:44 DEBUG : auth.log: Need to transfer - File not found at Destination
2024/01/11 09:49:22 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Uploading with chunks of size 50331648 and 4 transfers in parallel at path '/users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k/c8t8p90qhvi87hd32evbvgkv2k'.
2024/01/11 09:49:24 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Done uploading chunk of size 14478580 at offset 100663296.
2024/01/11 09:49:24 DEBUG : auth.log: hidrive = b8303f09d3049f817ebdd4a77d7a41cc5e639546 OK
2024/01/11 09:49:25 ERROR : auth.log: corrupted on transfer: sizes differ 115112317 vs 115113732
2024/01/11 09:49:25 INFO  : auth.log: Removing failed copy
2024/01/11 09:49:25 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 115112317 vs 115113732
2024/01/11 09:49:25 DEBUG : auth.log: Need to transfer - File not found at Destination
2024/01/11 09:50:24 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Uploading with chunks of size 50331648 and 4 transfers in parallel at path '/users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k/c8t8p90qhvi87hd32evbvgkv2k'.
2024/01/11 09:50:27 DEBUG : HiDrive root 'users/jan/backup/brockman/qk12vima9q4bt0nqjp9ieu6ol4/n2ubd34d5gku0nkhjkqjv07ivg/pv8897k8oj7v7318te2j9mur88/c8t8p90qhvi87hd32evbvgkv2k': Done uploading chunk of size 14481264 at offset 100663296.
2024/01/11 09:50:27 DEBUG : auth.log: hidrive = cc9c33ecbabaadfd20985572437a35b736dc1dd8 OK
2024/01/11 09:50:27 ERROR : auth.log: corrupted on transfer: sizes differ 115114415 vs 115116416
2024/01/11 09:50:27 INFO  : auth.log: Removing failed copy
2024/01/11 09:50:27 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 115114415 vs 115116416
Transferred:      329.424 MiB / 329.424 MiB, 100%, 1.183 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       2m2.8s
2024/01/11 09:50:27 NOTICE: 
Transferred:      329.424 MiB / 329.424 MiB, 100%, 1.183 MiB/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:       2m2.8s

2024/01/11 09:50:27 DEBUG : 7 go routines active
2024/01/11 09:50:27 Failed to sync: corrupted on transfer: sizes differ 115114415 vs 115116416

As you can see, the file size keeps changing. Excerpt from above:

2024/01/11 09:48:44 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: sizes differ 115111614 vs 115111842
2024/01/11 09:49:25 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: sizes differ 115112317 vs 115113732
2024/01/11 09:50:27 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: sizes differ 115114415 vs 115116416

I just picked one example here to illustrate, it affects many more log files on the server to backup.

I can see the reasoning behind "rather have no file but a corrupted one". However, there is no actual corruption going on here. rclone should be able to track the data it actually uploaded and compare that to the checksum provided by the backend, which would match. Instead it seems as if rclone will stat the file again after the upload and compare that new size to the remote size, which doesn't match.

When it comes to log files, I'd rather have some version of it in my backup than none.

kapitainsky · January 11, 2024, 9:29am

Firstly you are using fairly old version of rclone. You should consider updating.

Back to your issue. Transferring changing files is always a bit challenging.

To do this right it requires filesystem snapshots which are immutable during transfer/backup duration.

Please note that this is not rclone limitation but general problem which applies to any similar program. Many other programs simply transfer what you throw at them when rclone at least informs you that your transfer is corrupted (file at the remote is not the same as file from the source).

For database and similar I would recommend:

Stop database
Take filesystem snapshot
Start database
Run your backup/transfer against snapshot and not live filesystem

Taking snapshots is pretty much instantaneous so expected application downtime is measured in seconds.

janosch · January 11, 2024, 12:01pm

You are right that file system snapshots would help with the issue. However there is no chance to change the existing machine and the task is to backup it.

I think that it is reasonable to make backups of log files and with rclone I currently cannot, though if it spared the (suspected) second stat call, it could theoretically detect that the file got indeed transferred correctly and there was no need to report an error in the first place.

kapitainsky · January 11, 2024, 12:17pm

And here I disagree. What if we are talking about "large" enough file? And it changes when rclone is ploughing through it slowly, transferring it to the destination. What you end up with is not something what it was at the beginning, neither what it was at the end. You transferred some hybrid which never existed at the source. In my world it is called corrupted file and I prefer rclone to throw error here.

Could rclone have "no safety" switch to allow what you attempt? Why not at the end as long as it is not default. But I would not vote for implementing it:)

janosch · January 11, 2024, 12:38pm

I absolutely do see your point, that is why I wanted to restrict the discussion to append-only log files. These have no intermediate version which never existed, they only get bigger and every log record backed up is worth having.

It should be verifiable if a file was only appended to during transfer. So I am not asking for an "unsafe" switch, but rather for a safe implementation of append-only files (In fact I was asking whether there already was a solution, but it seems that there is not.) I can make a patch if somebody provides me with a little guidance where to start.

kapitainsky · January 11, 2024, 12:45pm

Agree. Log files are specific case. I am not aware of existing rclone solution to achieve it though. Let's see maybe others have some clever "ways".

This is the right and welcomed attitude:) @ncw could you please comment here? Is it worth pursuing?

asdffdsa · January 11, 2024, 3:16pm

welcome to the forum,

--local-no-check-updated
"If the file is only having things appended to it (e.g. a log) then rclone will transfer the log file with the size it had the first time rclone saw it."

kapitainsky · January 11, 2024, 3:27pm

Brilliant! rclone keeps surprising me again and again. Sometimes I feel like only --magic flag is missing:)

janosch · January 11, 2024, 4:07pm

Exactly what I was looking for, thank you very much! Not really easy to find in the docs. Any idea why some flags like --local-no-check-updated or --ignore-errors are missing from Documentation?

asdffdsa · January 11, 2024, 4:10pm

most welcome....

those flags are documented, sometimes in multiple locations.

on the docs page, in the upper right corner, is a search box

https://www.google.com/search?words=--ignore-errors&q=--ignore-errors+-site%3Aforum.rclone.org&as_sitesearch=https%3A%2F%2Frclone.org%2F

https://www.google.com/search?words=--local-no-check-updated&q=--local-no-check-updated+-site%3Aforum.rclone.org&as_sitesearch=https%3A%2F%2Frclone.org%2F

system · January 14, 2024, 4:10pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.