Did my rclone setup wipe my LO data or was it an unrelated bug?

Help and Support
1

What is the problem you are having with rclone?

I created a backup folder that is backedup to OneDrive using rclone. I recently had an issue where a LibreOffice (LO) Calc file was completely wiped after working on it and rebooting my machine. I do not know if this was because of rclone or if this is an unrelated issue with LO Calc. I still wanted to ask you guys if there is maybe something wrong with the way I am using rclone.

FYI since this happened on my Linux Mint machine, I asked the same question in the forum other there: https://forums.linuxmint.com/viewtopic.php?t=450137

Repro Steps from a user perspective

  • I opened a LibreOffice Calc file that is backuped to OneDrive using rclone
  • I changed something
  • I saved
  • I turned off the machine
  • Later I rebooted
  • I tried to open the file again

Expected Result:

  • The file opens with the contents I had last entered

Actual Result:

  • The file opens as an empty text, it did not even open in LibreOffice Calc anymore
  • Any contents in the file were wiped, any, like the file was 0KB afterwards

Run the command 'rclone version' and share the full output of the command.

rclone v1.69.3
- os/version: linuxmint 22.1 (64 bit)
- os/kernel: 6.8.0-64-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.24.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

MS OneDrive, personal cloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I am running this service on my machine:

[Unit]
Description=Rclone Mount for OneDrive
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/rclone mount onedrive: /home/user/Documents/onedrive --vfs-cache-mode full
ExecStop=/bin/fusermount -u /home/user/Documents/onedrive
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[onedrive]
type = onedrive
token = XXX
drive_id = XXX
drive_type = personal
### Double check the config for sensitive info before posting publicly
2

welcome to the forum,

i believe it is recommned to use Type=notify

without a rclone debug log, no way to know.

3

First of all thank you for your fast reply.

Thanks for the tip, if you got the time I would be happy to know more on why that option is recommended. If not I will try to read up on it in the documentation.

The thing is that this is not one command that failed but maybe an operation that failed while running the service. Do you know if there is any way for me to generate a debug log now?

4

--log-level=DEBUG --log-file=/path/to/rclone.log

5

These flags will not enable me to retroactively capture past actions.

6

that is correct

1 Like
7

I mean I could try to turn on the logs for the service now. But then I'll have to wait for the next time that a complete LO Calc file is wiped, which is honestly such a critical event that I'd like to proactively avoid it happening again.

I guess there is no other way. I find it very interesting that this seemingly only happened to me and to this guy. But except us two nobody has their LO Calc files getting wiped.

I guess if you work in Software QA you become a natural magnet for bugs at some point :laughing:

8

where is the LO file located?, local or onedrive?

imo, rclone mount is not a backup.

9

I accessed it locally. But because of the mount it exists both locally and in the cloud.

You're right, bad choice of words. I guess what I'm talking about might be called "Live Remote Access". But no matter how we call it my LO Calc file got wiped.

10

ok, we dealt with that. next time it happens, check the rclone debug.
and check the LO Calc log.

anything else you need with?

11

I will do that. And I do not want to sound unappreciative for the great help I'm receiving and for the great tool that rclone is. But I do want to say that getting your spreadsheet wiped completely out of nowhere is a huge bug. I do not know what software caused it, if it's a Linux Mint issue, a rclone issue or a LO Calc issue or maybe an issue with these software interacting with each other but it's a huge bug and the fact that I don't have the log ready won't make this bug disappear.

I mean since you are asking, I wonder if I could maybe improve on my rclone service somehow? There are dozens of flags in rclone and I haven't had the time to really read through it to see if I might need anything else for my use case. I was e.g. wondering if reducing the max-cache-age would make my setup a little more stable.

12

maybe, maybe not.

it is just one file in all the time you have been using rclone mount
so, just pull the last backup copy.

fwiw, i try to keep files local and rclone sync to cloud as needed.
that does not work for all use-cases but important files i keep local.

most good software will save backups on the file you are working on.
maybe Calc does that and you can retrieve the file.

for example, microsoft word;
there are many options, including to save a backup copy of a local file to the cloud.

image
image785×271 7.12 KB

imo, i keep important files on local and rclone sync to cloud using --backup-dir

rclone sync /home/user/source remote:latest --backup-dir="remote:versions/`date +%Y%m%d.%I%M%S` -vv --dry-run"

other software such as keepass, will run a script, and that is a way to backup local files on-demand.

  • on program start, but before the database is accessed for the first time.
  • each time the database is saved.
  • on program exit, after the databased has been closed.

image
image700×236 6.85 KB

another option, create a simple script

  1. run rclone sync --backup-dir= ...
  2. start Calc
  3. exit Calc
  4. run rclone sync --max-age=1m --backup-dir= ...
1 Like
13

Thanks for sharing your setup. I will consider it. But one big question I have now: Am I not using rclone mount exactly the way it is intended? I thought rclone mount is exactly there to give users Live Remote Access to the cloud. Or is rclone sync just the better tested operation and rclone mount is still in "beta" or something like that?

There are three options here:

  1. I'm misrepresenting what happened -> And I am not. I guarantee it.
  2. The behavior is intended -> And I don't see how that could be.
  3. This is a huge bug in one of these pieces of software.

Not saying that you or anyone else here can do anything about this or that this is necessarily even rclone's fault. I can retrieve the last backup copy. It is just one file. But this is still a significant critical bug, these are not some random pieces of software these are Linux Mint, rclone and LO Calc which are three of the most famouse pieces of open source software. Stuff like this happening is what makes me hesitant to fully commit to the Linux and Open Source universe.

14

The challenge you have is you can't go back in time and document with log files or other ways to show exactly what was done.

You list out some options but maybe you didn't even save it where you thought you saved it and it was /tmp or something and on reboot, that got wiped. It's a pure guessing game.

Rclone losing data shouldn't happen and there a number of ways that is ensured. OneDrive some funky things with Office files as documented here:

Microsoft OneDrive

Your best bet as you 'are sure' on your steps would be to enable debug logging, repeat them and share the log files when you have recreated the issue.

15

You won't believe how often I have talks like this at work. While I generally agree with you, you have to see that there are "flaky bugs" that happen for some obscure reason or only in very specific constellations. I'm sure if spreadsheets in LO were wiped every other day people would talk about it more. This just happening once is still something to look at in my opinion. You know if like 100.000 new electric cars of some kind of model are sold and just one of them randomly explodes then that's not good. Also I seem to not be the only one experiencing this, see this guy.

While it is indeed a guessing game since I can neither reproduce the issue nor do I have any logs of any kind; what you describe cannot have happened here because the file was completely wiped, like any data was wiped, it was 0KB afterwards and LO Calc didn't even recognize it as a spreadsheet anymore. So the data I had originally was gone as well. I also know that there was data for sure because I can still retrieve that from the OneDrive version history.

That is indeed true. Maybe something like a rough log that is opt out could make sense to avoid this.

16

One guy from 2 years ago that posted no details as well? That guy doesn't mention rclone either.

Depends on how the car exploded as if you've been in the software business, users do the wackiest of things as well and if they dump gasoline on the car and toss a match, who knows what happened?

Data loss is a very serious thing for any tool and without anyway to recreate or reproduce it, it's just a one off and it'll fall into the ether as your linked comment also have zero replies minus the bot which leads one to think it was some user error possibly and not a software bug.

Over the years, we have seen issues and once they have something to look at, folks can dig into it.

OneDrive is notorious for modifying files after upload and for me, there isn't a chance I'd ever use that provider for my data in that fashion.

1 Like
17

Yes, exactly. Not sure what your point here is. Do you think that guy and me just don't know how to use spreadsheets? Just because something isn't easy to debug because you have nothing to go off does not mean that it did not happen. A lot of very bad persistent software errors stem from developers looking the other way because there is no easy way to debug.

Exactly.

Just my personal opinion but this is a bad way to look at things. And a mindset that can lead to significant bugs/accidents. Not every user goes through the effort of reporting stuff like this, many users just go back to Excel/Windows after that happens once.

Never had issues with file contents changing in OneDrive when working on a Windows or MacOS machine with the native client.

18

Of course not as this is an API thing and how OneDrive's API works. The native client doesn't use the same methods that other 3rd party apps use.

I just use the information provided and go through a logical problem solving process based on the data and percentages and try to keep my feelings out of problem solving. Appreciate not juding my mindset as I'm not juding yours. Thanks.

We're cirlcing a bit here. People makes mistakes. You make mistakes. I make mistakes. We all make mistakes. Knowing or not knowing how to use something doesn't change that. If you've been in IT a long time, it's not "if" someone/something breaks, it's "when" will it break as it always will.

We want to help you. So rather than going back and forth, you made the effort to post. I've asked to try to capture some details and share a log. If you'd like to give that a try, by all maens, that will be the best way to help progress the issue rather tha discuss our personal philophies.