I use rclone sync to back up a drive by cloning it to an external drive plugged directly into the machine. However, this external drive occasionally disconnects during the backup process for unknown reasons. What I would like to do is automatically pause rclone when this happens and then resume it when the drive reconnects. Is this possible?
This is clearly problem with hardware you are using. Strongly advisable to fix it if you care about your data.. Can be as trivial as replacing USB cable.
Not by using rclone only but you can add few liner script to achieve what you want, e.g.:
while : ; do
rclone sync ...
RESULT=$?
if [ $RESULT -eq 0 ]; then
break
else
sleep 60
done
That might work under other circumstances. The problem is that when the hard drive disconnects, it doesn't interrupt rclone. Rather, rclone continues to copy the data to the folder where the drive was mounted, but now that folder is local, essentially duplicating the data on the same machine.
I'm pretty sure the hardware issue is that the external drive is connected through a 2-port USB switch, since the drive is shared between two computers. I don't want to lose this setup and was hoping I could find a temporary software fix until I figured out how to manage the hardware better.
Only one computer can own a drive at any time. If the USB device is visible to two machines then you must ensure only one machine "owns" the device.
(You might be able to kludge it with partitions; machine A owns partition 1, machine B owns partition 2).
If this is a manual "KVM" type switch then you must ensure the disk is dismounted and idle before you switch. Disk drives aren't keyboard/mouse; they can't be hotswapped this way.
A drive is "randomly" (either 'cos you pressed a button or something else) dropping off the "owning" machine is a problem and it could lead to data corruption (eg cached data not be written out).
If this is a manual "KVM" type switch then you must ensure the disk is dismounted and idle before you switch. Disk drives aren't keyboard/mouse; they can't be hotswapped this way.
It is a KVM-type switch (specifically, this one here), and I'm aware they can't be hot-swapped. The drive is dismounted from one computer before I hit the button that switches it to the other computer. It basically saves me the step of having to unplug the drive from one machine and plug it in the other one.
I think the hardware problem is because during large data transfers, the switch can't provide enough power, and the connection gets dropped. I've read about similar things happening when external drives are connected to computers through USB hubs (particularly unpowered ones). I'm investigating alternative switches to see if there's a better option.
Hmm, Startech is generally a reputable brand (I have some of their stuff).
What type of drive is this? If it's spinning rust that's USB powered then definitely power may be an issue; this USB switch is self-powered and it's common for self-powered devices to struggle with powering high demand peripherals. It's why many hubs have external power.
What might also work is if you put a powered hub between your drive and the USB switch.
(To your original question; I very much doubt rclone, itself, will handle this; there's just far too many OS specific issues to deal with; Linux and Windows and MacOS will deal with devices dropping off the bus differently. Fix the real problem).
The external drive is one of these. Yeah, I know I need a hardware solution. I just wanted to see if I could do something stop-gap with rclone until I got the real problem fixed.
Oh, they're externally powered (at least the 8Tb versions I have are!) with a power plug, so USB power shouldn't be an issue. If you're not using a wall-wart to power this then you should.
ditto. the stuff i have purchased is not much different or not at all different from no-name companies.
but i am willing to pay a bit extra for peach of mind with startech QA and tech support.
fwiw, rclone runs fine on the wd devices and can access usb devices.
root@MyCloudEX2Ultra rclone # uname -a
Linux MyCloudEX2Ultra 4.14.22-armada-18.09.3 #1 SMP Thu Oct 27 08:25:39 UTC 2022 ga-18.09.3 Build-30 armv7l GNU/Linux
root@MyCloudEX2Ultra rclone # ./rclone version
rclone v1.68.1
- os/version: unknown
- os/kernel: 4.14.22-armada-18.09.3 (armv7l)
- os/type: linux
- os/arch: arm (ARMv7 compatible)
- go/version: go1.23.1
- go/linking: static
- go/tags: none
root@MyCloudEX2Ultra rclone # ./rclone ls /mnt/USB/USB1_c1
0 file.ext