before i fill out the template, could someone tell me what a "MD5 Hash Difference" is? i was doing a random check on previously synced files between two servers and out of around 1 million files a few of them come up with "MD5 Hash Difference". does that mean the file is corrupt and unusable? or does it just mean there might be some arbitrary difference between them? as i understand file transfers, there's no such thing is "arbitrary difference", but come someone shed a little light on this? doing a sync of the two directories doesn't find anything to transfer.
alright. so it's that simple. finding a MD5 has difference is BAD. and if that is found, something happened to the file during transfer and it is to be deleted and replaced? without question?
what was the rclone comamnd you used?
what was the source, local, gdrive or what?
what was the dest, local, gdrive or what?
the file could get corrupted during transfer, that does happen.
as part of the copy process, in most cases, rclone computes the checksum of the source and compares it to the dest after the copy process has finished and is there is a mismatch rclone will output that.
however, in your case, you were checking files, not copying files.
i was checking files that had been transferred in the past with the 'sync' command. does the sync command do a checksum?
the real reason i was doing the check is because i was trying to load the server with SMB traffic. and a hash comparison over SMB was the biggest load i could think of
sure. two local servers, connected over SMB. the source was a Windows Server 2019, and the destination was TrueNAS 12.
the files had been transferred with the sync command in the past, and then i did a sync again yesterday to transfer any new files, and then did a check. the checksum error was found on a file that had been transferred in the past.
what program did the original copy? do you have the original log file?
what kind of raid is in use on ws2019 and truenas?
hardware, software or what?
i use the free ws2019-hyperv edition, using REFS in a raid5 configuration.
for what it is worth, i have never used rclone on local, never would trust all my copying/backups to just a single program. rclone for cloud, fastcopy for local.
There are lots of possible causes. If this was my computer I'd check its memory with memtest86 - RAM problems are suprisingly common and they can cause bit errors like this.
i used rclone sync to transfer the original file. and no, i do not have the log from then.
the WS2019 is running on hardware raid, with a dell H730 mini doing a RAID1 for the OS and RAID6 for the strorage, using Samsung 870 Pro's for the OS and Samsung 870 Evo's for the storage.
the TrueNAS is doing everything with softraid, which is what they're all about. there is an HBA330 that's passing the hard drives through to TrueNAS.
no, i bought both the servers refurbished from ebay. and yes, both of them have ECC RAM.
admittedly, i hate TrueNAS. it doesn't let me do local file managment with a GUI like windows server. but i didn't want to sell my wife and children into slavery for windows server, and TrueNAS is supposed to be all the rave. it has a really good reputation. since i'm still in the testing phase and learning truenas, i have not been depending on it for backups yet, and am still relying on cloud backups from rclone. this back and fourth is part of the vetting process. i am really paranoid about losing files and file corruption, so i do overtest it.
I told the TrueNAS forum about the file transfer issues i've been having, and they released an update with some SMB fixes and it went away. however, it seems there have been some losses in the past.
still. does rclone do a checksum when running a local sync?
i use the free windows server 2019 hyper-v edition, in 8+ bunch of locations.
like truenas, has softraid REFS file system.
and when combined with free veeam backup and replication.
rock stable and free.
yes.
best to use and keep a debug log, the answer would be in the log.
i always keep the logs and scan them for errors.
this is an example of a command i use with rclone combined with VSS snapshots C:\data\rclone\scripts\rclone.exe sync "b:\mount\rcloner\data_wasabi_sync+check_20210408.094522\data" "wasabicrypten07data:en07data/data/rclone/backup" --stats=0 --fast-list --exclude-from=c:\data\rclone\scripts\rr\data\exclude.txt --backup-dir=wasabicrypten07data:en07data/data/rclone/archive/20210408.094522 --log-level=DEBUG --log-file=C:\data\rclone\logs\data_wasabi_sync+check\20210408.094522\rclone.log
Windows Server 2019 hyper-v is a hypervisor. how do you run veeam, which is a windows application, on a hypervisor, without installing windows? and then, where are the files stored? which OS actually houses the SMB share for you to copy files to it?
it is confusing as the name implies it is just for hyper-v.
the hyper-v edition, which is free, is almost a complete version of windows server.
the main difference is it cannot be a domain server and does not have a full GUI,
no windows explorer which i hate and have not used in over 10+ years
no control panel.
the files are stored on the window server 2019 hyper-v.
i use them as backup servers, REFS softriad 5, which has checksums for metadata and checksums for the files. REFS is windows version of ZFS.
on that server i run a python script that coordinates the following software for backups
uses VSS snapshots
full version veeam backup and replication, free edition
ok that is nuts. they make it sound like there is absolutely nothing on there but a CLI for you to add VM's. but, is it complicated to set up folders and network shares for users wince there's no GUI? like, no GUI at all? i mean, TrueNAS has a gui for setting up the datasets at least.
also, i must insist that at some point in the past 10 years you DID use windows explorer. even if it was just to see how full the C: drive is. actually, opening a file from the desktop of your home computer constitutes as "having used windows explorer"
well, on my personal computers, my desktop is empty, no icons at all.
not sure if you know about it, but there was a famous lawsuit against microsoft. they wanted to kill netscape they claimed they could not remove internet explorer from windows as it was tied to windows explorer and the desktop was involved in the trial, also i never use a mouse, i use a trackpoint. so minimizing apps and clicking on the desktop is very annoying to me...
about free space, i get that info from double commander and if i need more detail, wiztree.
as for hyper-v edition, i run it headless
RDP works fine
openvpn clients gui works fine.
wiztree runs fine
double commander runs fine
7zip gui runs fine
veeam runs fine.
lots of software runs fine.
as for managing the server, it is exactly the same as managing any windows server remotely.
there is server manager.
as for network shares, compmgmt.msc, as with many, .msc, can manage a remote server.
there is the shiny new windows admin center which can create shares. going forward, this is the how microsoft supports remote management of servers and desktops. very powerful. i created the refs raid5 dataset using it.
really, very little, almost nothing, needs to be done of the command line.
another nice advantage.
if you wanted to rent a windows server VM from a cloud provider, they charge for the windows license. if you run the hyper-v edition and upload the iso yourself, there is no windows fee
for example, vultr.com charges $10.00/month.
also, hyper-v runs fine with 2GB ram and limited cpu
which OS actually houses the SMB share for you to copy files to it?