Greetings - I managed to get rclone-v1.65.2 working on my QNAP NAS.
I am using Mega for storage and have issued the command rclone copy /local_share remote:backup
I see that data is being copied. It's been running for a few hours and about 13GB has been copied so far.
I'm backing up my music collection which is a few TB, so I understand that it will take days/weeks to finish.
My question is: Once the initial copy has completed, what's the best method/command(s) to keep the remote share on Mega up to date as I continue to add more music to my local share?
I don't want stuff to disappear from Mega if they are somehow lost locally. That doesn't make sense to me as the whole point of copying everything to Mega is to have a cloud backup in case of a catastrophe at home.
I know this might seem like a silly question, but it seems like there are lots of different options and I was wondering what would be best/most simple?
Any suggestions will be greatly appreciated! Thanks!!
I'm guessing that you will want to continue to use the copy command rather than the sync command.
Sync will delete files from the destination (if they are not on the source) which is your mega storage, and you don't want anything deleted from there.
The copy command will not, at least, it should not recopy anything that exists in the destination.
You can always stop and start and Rclone will pick up where it left off nothing will be duplicated.
Check out the -P flag when copying. It will give you interactive statistics about the transfer that is underway, how much has been done how much there is to go, how fast it is going etc.
You can also use --backup-dir flag to create the closest to "backup like" solution using rclone. It will protect you to some extend from unrecoverable propagated deletions and changes. Depends on your usage restoring such backup to specific date in the past might be very cumbersome though.
In general rclone is not a substitute for proper backup software. It is great tool to transfer files to/from almost any cloud. And real backup requires much more than that.
For proper solution use something designed for the job. There are many free and paid options available. For open source I would recommend to look at restic/rustic or kopia.
I write shell scripts for backup then link the crontab to that. Easier to make changes and also shell script useful to add custom dates in file and do other interesting things at the same time.
I use rclone extensively for backup. It's great to have a historic log file to work from too.
So, I was able to move the directory with the problem file out of the path being copied by rclone to Mega and I am no longer seeing errors relating to that file.
But there's still a problem - It appears that rclone thinks it has copied my entire Music directory up to Mega, because when I run the copy command again it, here is what it returns:
Nobody can guess without you providing all info. rclone copies everything you ask it to. Most likely there is some problem with your share/access rights etc. Maybe instead of using Windows and shares install rclone directly on your NAS and run from there.
On your pictures you definitely compare different things - as your Music Property contains 103039 files and directory you run rclone copy from 116366 files.
Also when possible do not post pictures - they are hard to read + not easy to use any data from them to even give you some suggestions. Copy/paste text in the future.
I discovered a few more interesting things (and btw, I am running rclone directly from the NAS via ssh)
The output from the rclone command does show 116399 files, however when checking folder info from the Mega website it shows only 53735 files. That's why I believe that rclone thinks it copied everything, but it didn't.
Another interesting thing is that as a test I decided to copy a random folder containing a few albums. Total size 1.71GB.
The rclone output appears to indicate that it was successful. However, it completed much too quickly - 6.8 s.
Then when I look at my drive on Mega, the folder is not shown. However, when I search for the folder name on Mega, it does find the subdirectories but the sizes are all 0B. These folders are only found when a search is done, they do not show up when I browse the parent folder.
Hopefully the above explanation is sufficent. Including some screen shots below as well to help illustrate.
Ok, I added crypt ( I think I did it correctly) and then copied another folder but the behavior appears the same. It completed in 0.4s, not showing on Mega when browsing, but can be found using search, and shows as 0B
I think you have may have been correct before when you suggested that there may be an issue with my share/permissions, unfortunately.
I probably need to get that sorted before going further down the rclone rabbit hole.
rclone was working perfectly for a while...
I first started having issues with Roon, having to restore the database from backup several times, etc.
Then things were working again but rclone seemed to terminate when a particular file was scanned. And that same file is the one that I cannot delete from my NAS no matter what I try. I was able to move it to a different folder which isn't in the path being copied to rclone though. But since then, things have not been the same.
The file shows as follows from ssh session to the NAS
? d????????? ? ? ? ? ? .@__thumb/
When I try to delete it I get errors like "directory not empty" and "structure needs cleaning", etc.
So, there seems to be some file/db corruption on the NAS? I am just wondering if this happened because I started using rclone? Maybe a roon db backup and a rclone copy happening simultaneously? Maybe I could've/should've done things differently?
Stop posting pictures - it is not Instagram - use copy/paste for text output
Post all details including your remotes setup and logs
Otherwise it is just a story maybe interesting to read but with no value to help to troubleshoot.
rclone copy NAS remote: does not change anything on your system. So it is close to impossible. Most likely by being very strict and double checking everything it just reveals problems with you storage (NAS).
Sorry that the pictures are annoying you, kaptin. I'll refrain from posting more.
I wasn't trying to imply that rclone changed anything on my system. I was just thinking that maybe something could have gotten corrupted if it was being accessed by another program while rclone was doing a copy or a scan? Not trying to blame rclone.
Anyway, whatever happened is now preventing me from copying the rest of my collection to Mega using rclone, which is a bummer.
I think one clue is that when I try to copy my collection again it shows 116386 / 116386 files checked but Mega only shows 46832 files in my backup directory..
The number of files on Mega is lower than before because I deleted a bunch of @__thumb files which I though might be causing problems.
