Rclone copy deleting data in destination

Suspected Bug
1

What is the problem you are having with rclone?

Running an rclone copy command resulted in data being deleted in the destination similar to what would be expected from an rclone sync command. The command was being ran via cron on a 30 minute timer. We have since halted the cron and are not running it again until we have a better understanding of what caused the deletions.

Run the command 'rclone version' and share the full output of the command.

rclone --version
2023/01/25 12:29:37 NOTICE: Config file "/home/aaadmin/.config/rclone/rclone.conf" not found - using defaults
rclone v1.55.1-DEV
- os/type: linux
- os/arch: amd64
- go/version: go1.15.5
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Wasabi S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy cdn-download-set-data:/az-cdn-setdata/ /mnt/aa-prod/CDN_VFX/zShoot/_Set_Data/ --log-level INFO --log-file /var/log/rclone.log --update --transfers 10 --multi-thread-streams 12

The rclone config contents with secrets removed.

[cdn-download-set-data]
type = s3
provider = Wasabi
access_key_id =
secret_access_key =
endpoint = s3.us-central-1.wasabisys.com

A log from the command with the -vv flag

The relevant bits are lines 868 to 1943. As this command was running in production, and it seems to be related to an error of some sort, we are not able to run it again at the requested level of verbosity.

2

What is that? Local? Network mount?

That's a very old, custom compiled version. I'd use the latest stable and only from the download site:

or installed

3

Good to know. I just spoke with the engineer that set it up and he said that it's what was installed when running yum install rclone on CentOS 7. We've also created a script to easily restore data if it gets deleted again, so we're going to update to the latest version and turn it back on. I'll update this thread if we run into anymore deletions.

4

Can you let me know what that is?

5

That is a cifs network mount

6

So if that's the case my best guess would be the mount went away / had an issue as that would cause this:

2023/01/25 08:22:48 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to close object after copy: close /mnt/aa-prod/CDN_VFX/zShoot/_Set_Data/Int_North_Pole_Hangar/witcam/20230105/120AN/witcamB/witcamb_120AN_take04.mov: input/output error

I'd check for any network interruptions or something unmount/remounting or something along those lines.

7

Yes, there was most likely a blip in the network storage connectivity at that time, but that doesn't explain why rclone would start deleting things when the mount came back up.

8

I'd imagine it's going to delete the directories that worked properly and not the ones that failed.

2023/01/25 08:22:48 ERROR : Local file system at /mnt/aa-prod/CDN_VFX/zShoot/_Set_Data/: not deleting files as there were IO errors
2023/01/25 08:22:48 ERROR : Local file system at /mnt/aa-prod/CDN_VFX/zShoot/_Set_Data/: not deleting directories as there were IO errors
2023/01/25 08:22:48 ERROR : Attempt 1/3 failed with 1 errors and: multi-thread copy: failed to close object after copy: close /mnt/aa-prod/CDN_VFX/zShoot/_Set_Data/Int_North_Pole_Hangar/witcam/20230105/120AN/witcamB/witcamb_120AN_take04.mov: input/output error

That's not deleting.

2023/01/25 08:28:13 INFO  : _gsdata_/2022-1104-020506-Mac-02-OnSetSync.log: Deleted
2023/01/25 08:28:13 INFO  : _gsdata_/2022-1104-173151-Mac-02-OnSetSync.log: Deleted
2023/01/25 08:28:13 INFO  : Daily_Reports/1U/D32_20221128/.DS_Store: Deleted

That does delete but without a lot more detail, I wouldn't be able to tell you if is right or wrong.

9

hopefully, just a one-time glitch

i use wasabi s3, have over 45,000 rclone debug logs.
could not find that exact error.

tho, i have had multi-thread upload issues over the years.
might reduce --s3-upload-concurrency

might run the command with --dry-run.
if the error only occurs during file transfer, then that will not help diagnose.

10

That's nothing to do with the S3 remote as it's a local disk spouting that error.

11

If the deleted items were items that were copied over as part of the rclone copy process, then I would chalk it up to some sort of rollback and move one, but the items that were deleted were exactly the items that were not in the Wasabi bucket. Which leads me to believe that there's something else happening here.

12

The log you sent is from an rclone sync command - only rclone sync prints

  1. 2023/01/25 08:22:48 ERROR : Local file system at /mnt/aa-prod/CDN_VFX/zShoot/_Set_Data/: not deleting files as there were IO errors

We'd know for sure with a debug log which prints the command line arguments.

13

Unfortunately, we don't have access to a debug log on this one. The version of rclone that we were using does sound like the culprit though. We've updated to the latest version provided via website and will no longer be using the version installed via yum on any of our deployments. I'll create another thread if we somehow run into this issue again. Thank you all for the help!

1 Like
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.