Colons :, new lines \n, etc.
Yes, I did, this hasn't happened in any of the previous logs, I am trying to figure out what to make of it, what does it mean?
to be clear, both rclone and rsync fail with the same weird symbols ?
and why do you need encoding in the remote config?
1 Like
I see, but still I prefer the multithreading capabilities of rclone.
For some reason both rclone and rsync were failing to sync files with those symbols in their name, and I couldn't make rsync sync them, however with the encoding capabilities of rclone it was able to successfully sync them.
Its a brand new SSD and I did a SMART test on the SSD and all seems good:
sudo smartctl -H /dev/sdc
[sudo] password for usertemp:
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.8.0-40-generic] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
I researched how to debug Ubuntu Server the best I can, and here is what I came up with so far:
Ubuntu Server DEBUG information:
sudo cat /var/log/kern.log | grep error
sudo cat /var/log/syslog | grep error
Still those things seem to be unrelated according to what I found on the internet.
I will try a direct smb mount to rclone, like kapitainsky suggested, and report back.
In the mean time any help on how to debug this and find what exactly is failing will really mean a lot to me...
More DEBUG information:
Here are some screenshots of when the crash happens on the VM, the CPU just flatlines:
And also screenshots of the TrueNas Scale VM, it doesn't freeze up and is constantly accessible:
There is nothing yet pointing into rclone I am afraid. What is does is just reading a lot of data (all files) from both src and dst as you run check with --checksum flag.
And it does kill your VM....
Do you have another SSD to test? Can you connect it via SATA? and not USB?
Also if you have enough disk space on your Proxmos server you could create ad-hoc 2TB disk and sync/check. If it works it would be very strong argument against rclone issues.
You could also try some other OS.
PS. Also try Proxmos forum - sometimes things do not work as they should when VMs are involved, e.g. Backup to Samba Share freezes | Proxmox Support Forum
1 Like
Thanks for the reply, I appreciate it because I am not able to find anything else yet.
I figured it might be a good idea to post to ProxMox forums since you suggested it might not be an rclone, but a VM issue, and I posted yesterday, however noone has replied and by now my topic in the ProxMox forum is far behind new topics an has no visibility and will probably get no replies...
It is connected via SATA, and I took out the SSD and connected it to a USB v3.0 enclosure - same problem.
It does not seem to be the connection...
Unfortunately I don't have another SSD to test with, nor sufficient space on ProxMox to create a disk there.
I am trying to debug the ubuntu server vm and the ProxMox PVE, but so far I am not seeing anything in the logs, I am trying to find out what is pushed to the limit so it kills the VM... no luck so far...
Yeah it might be very pesky issue indeed.
What about to try the same rclone check without any VM? Directly from Proxmos OS level? If it works than you would know that issue is VMs.
1 Like
It would be good if rclone has some features for throttling local syncs, such as from one local dir to another local dir.
It seems in the case of local sync the --bwlimit flag does not take effect.
Also it would be good to have the same throttling flags to both the sync and check command, for such scenarios, where the sync command succeeds but the check command goes blazing fast and overloads something.
As it is Linux you could use cgroups to throttle all aspects of rclone process including I/O bandwidth.
But in this case I think you will only try to mask existing issue with throttling - whatever the issue is. OS like Ubuntu should not die because some process is reading a lot of data....
1 Like
Well said.
I have posted on AskUbuntu in StackExchange and on the Ubuntu forum... let's hope that the threads get some replies as well as the one on the ProxMox forum...
It really messes with me, if I can't find a problem and fix it, but rather patch it up with a "temporary" work around.
Hello again,
So, I made some time to do more debugging:
I passed the SMB shares directly to rclone as suggested in the comments above.
There was no change, it still crashed.
However it seems that now the --bwlimit flag is actually taking effect as I played around with throttling it.
Previously when the SMBs were locally mounted and I was passing them to rclone as locals it seemed that the --bwlimit had no effect at all.
I also re-wrote the script to use rsync and it did not crash.
When it ran with rsync it did not reach any high bandwidth throughput, which is I guess to the non-multithreaded nature of rsync, it also ran slower, so it did not seem to push something to the limit, something related to I/O reading/writing large amounts of data at a very high speed/rate.
I do want to be able to use rclone because of it's more advanced features, not just multithreading but encoding, SMB support etc...
With rsync I also got the following errors in the log file:
rsync: [receiver] mkstemp "/mnt/sd25-4tb-bkp4-mnt4/tns-maindatads-smb/tns-maindatads-smb/main-data-subds-1/GoogleDrive-alk8915-bkp-2024-07-09-1800/Programirane neshta failove alk19890105/SoftUni/SoftUni AI 2021 Materials/SoftUni - AI - Data Science June 2021/softuni-ai-datascience-2021-project-FINAL-SUBMISSION/softuni-ai-datascience-2021-project-atkuzmanov-FINAL/resources/images/.Icon\#015.pu2lTm" failed: Invalid argument (22)
I don't get these problems with the rclone encoding config I have posted in the beginning of the thread, this is another reason why I want to use rclone, I was not able to find out how to get around those errors in rsync.
I also got this weird output in the log:
2024/08/17 22:33:46 [9667] calling match_sums /mnt/tns-maindatads-smb/main-data-subds-1/Prog hranilishte alk8915 arhiv bkp-2024-07-10-1118/Rabotilnitsa-programirane-alk19890105-2022-08-02-1455.zip
calling match_sums /mnt/tns-maindatads-smb/main-data-subds-1/Prog hranilishte alk8915 arhiv bkp-2024-07-10-1118/Rabotilnitsa-programirane-alk19890105-2022-08-02-1455.zip
tns-maindatads-smb/main-data-subds-1/Prog hranilishte alk8915 arhiv bkp-2024-07-10-1118/Rabotilnitsa-programirane-alk19890105-2022-08-02-1455.zip
^M 32.77K 0% 97.86kB/s 38:59:03 rsync: [receiver] mkstemp "/mnt/sd25-4tb-bkp4-mnt4/tns-maindatads-smb/tns-maindatads-smb/main-data-subds-1/Prog hranilishte alk8915 arhiv bkp-2024-07-10-1118/.Icon\#015.0JwIpt" failed: Invalid argument (22)
^M 46.92M 0% 44.75MB/s 0:04:58 ^M 114.26M 0% 54.51MB/s 0:04:03 ^M 183.11M 1% 58.25MB/s 0:03:47 ^M 251.33M 1% 59.97MB/s 0:03:39 ^M 319.68M 2% 65.06MB/s 0:03:21 ^M 388.10M 2% 65.32MB/s 0:03:19 ^M 456.59M 3% 65.24MB/s 0:03:18 ^M 525.73M 3% 65.47MB/s 0:03:17 ^M 593.82M 4% 65.41MB/s 0:03:16 ^M 662.41M 4% 65.45MB/s 0:03:15 ^M 730.60M 5% 65.38MB/s 0:03:14 ^M 799.15M 5% 65.24MB/s 0:03:13 ^M 867.40M 6% 65.28MB/s 0:03:12 ^M 935.40M 6% 65.12MB/s 0:03:11 ^M 1.00G 7% 65.20MB/s 0:03:10 ^M 1.07G 7% 65.20MB/s 0:03:09 ^M 1.14G 8% 65.20MB/s 0:03:08 ^M 1.21G 8% 65.29MB/s 0:03:07 ^M 1.28G 9% 65.22MB/s 0:03:06 ^M 1.35G 9% 65.20MB/s 0:03:05 ^M 1.41G 10% 65.21MB/s 0:03:04 ^M 1.48G 10% 65.31MB/s 0:03:03 ^M 1.55G 11% 65.34MB/s 0:03:02 ^M 1.62G 11% 65.31MB/s 0:03:01 ^M 1.69G 12% 65.36MB/s 0:02:59 ^M 1.76G 12% 65.27MB/s 0:02:59 ^M 1.81G 13% 62.51MB/s 0:03:06 ^M 1.88G 13% 62.44MB/s 0:03:05 ^M 1.95G 14% 62.34MB/s 0:03:04 ^M 2.02G 14% 62.43MB/s 0:03:03 ^M 2.09G 15% 65.31MB/s 0:02:54 ^M 2.16G 15% 65.38MB/s 0:02:52 ^M 2.22G 16% 65.48MB/s 0:02:51 ^M 2.29G 16% 65.42MB/s 0:02:50 ^M 2.36G 17% 65.32MB/s 0:02:50 ^M 2.43G 17% 65.35MB/s 0:02:48 ^M 2.50G 18% 65.42MB/s 0:02:47 ^M 2.57G 18% 65.49MB/s 0:02:46 ^M 2.63G 19% 65.43MB/s 0:02:45 ^M 2.70G 19% 65.39MB/s 0:02:44 ^M 2.77G 20% 65.31MB/s 0:02:43 ^M 2.84G 20% 65.17MB/s 0:02:43 ^M 2.91G 21% 65.13MB/s 0:02:42 ^M 2.98G 21% 65.06MB/s 0:02:41 ^M 3.04G 22% 64.94MB/s 0:02:40 ^M 3.11G 22% 64.84MB/s 0:02:39 ^M 3.18G 23% 64.74MB/s 0:02:39 ^M 3.25G 23% 64.78MB/s 0:02:38 ^M 3.32G 24% 64.68MB/s 0:02:37 ^M 3.38G 24% 64.66MB/s 0:02:36 ^M 3.45G 25% 64.73MB/s 0:02:35 ^M 3.52G 25% 64.69MB/s 0:02:34 ^M 3.59G 26% 64.77MB/s 0:02:33 ^M 3.65G 26% 64.82MB/s 0:02:31 ^M 3.72G 27% 64.56MB/s 0:02:31 ^M 3.79G 27% 64.16MB/s 0:02:31 ^M 3.86G 28% 64.06MB/s 0:02:30 ^M 3.92G 28% 64.12MB/s 0:02:29 ^M 3.99G 29% 64.06MB/s 0:02:28 ^M 4.06G 29% 64.38MB/s 0:02:26 ^M 4.13G 30% 64.45MB/s 0:02:25 ^M 4.19G 30% 64.45MB/s 0:02:24 ^M 4.26G 31% 64.73MB/s 0:02:22 ^M 4.33G 31% 64.75MB/s 0:02:21 ^M 4.40G 32% 64.88MB/s 0:02:20 ^M 4.46G 32% 64.57MB/s 0:02:20 ^M 4.53G 32% 64.56MB/s 0:02:19 ^M 4.60G 33% 64.67MB/s 0:02:17 ^M 4.67G 33% 64.58MB/s 0:02:17 ^M 4.74G 34% 64.91MB/s 0:02:15 ^M 4.80G 34% 64.98MB/s 0:02:14 ^M 4.87G 35% 64.89MB/s 0:02:13 ^M 4.94G 35% 64.85MB/s 0:02:12 ^M 5.01G 36% 64.78MB/s 0:02:11 ^M 5.07G 36% 64.48MB/s 0:02:11 ^M 5.14G 37% 64.48MB/s 0:02:10 ^M 5.21G 37% 64.08MB/s 0:02:09 ^M 5.28G 38% 64.03MB/s 0:02:08 ^M 5.34G 38% 63.98MB/s 0:02:08 ^M 5.41G 39% 63.96MB/s 0:02:07 ^M 5.48G 39% 64.31MB/s 0:02:05 ^M 5.54G 40% 64.17MB/s 0:02:04 ^M 5.61G 40% 64.35MB/s 0:02:03 ^M 5.68G 41% 64.10MB/s 0:02:02 ^M 5.75G 41% 63.95MB/s 0:02:01 ^M 5.81G 42% 64.01MB/s 0:02:00 ^M 5.88G 42% 64.06MB/s 0:01:59 ^M 5.95G 43% 63.84MB/s 0:01:59 ^M 6.01G 43% 63.81MB/s 0:01:58 ^M 6.08G 44% 63.53MB/s 0:01:57 ^M 6.15G 44% 63.70MB/s 0:01:56 ^M 6.22G 45% 64.25MB/s 0:01:54 ^M 6.28G 45% 64.48MB/s 0:01:52 ^M 6.35G 46% 65.02MB/s 0:01:50 ^M 6.42G 46% 64.88MB/s 0:01:50 ^M 6.49G 47% 64.57MB/s 0:01:49 ^M 6.55G 47% 64.70MB/s 0:01:48 ^M 6.62G 48% 64.27MB/s 0:01:48 ^M 6.
After which it carried on fine.
Any help in debugging and fixing this is appreciated!
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.