Ok, so you have done the right thing and set up your linux server (or desktop) with software RAID1 on all partitions, but when a hard drive dies, how do you actually recover the system with minimal downtime? Elementary my dear Watson, read on and learn!

First off, let us define our hypothetical server. It has 2×250GB SATA HDDs, /dev/sda1 and /dev/sda2, with the following partitions:

• /dev/md0: /boot (200MB - sda1/sdb1)
• /dev/md1: /var (20GB - sda2/sdb2)
• /dev/md2: / (206GB - sda3/sdb3)
• /dev/md3: swap (4GB - sda5/sdb5)

Bam, a drive fails, but which drive was it?

# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdb2[1] sda2[0]
20482752 blocks [1/2] [_U]

md2 : active raid1 sdb3[1] sda3[0]
4192896 blocks [1/2] [_U]

md3 : active raid1 sdb5[1] sda5[0]
219319232 blocks [1/2] [_U]

md0 : active raid1 sdb1[1] sda1[0]
200704 blocks [1/2] [_U]

unused devices: <none>

Lets break down the output a little more to be able to understand it better:

md1 : active raid1 sdb2[1] sda2[0]
20482752 blocks [1/2] [_U]

And here is what that means:

• md1: The raid device is named md1
• active: The raid is currently active
• raid1: The raid device is set to RAID1
• sdb2[1] sda2[0]: This is a list of all the partitions in the raid device, the [x] refers to the order (ie: sda2 is the first partition, sdb2 is the second)
• 20482752 blocks: The size of the raid device
• [1/2]: Number of active devices/Number of total devices
• [_U]: Status of each member of the array, uses the list order (ie: sdb2 sda2) rather than the order specified by [x]

The confusing part here is that sda is the first drive and sdb is the second drive (as shown by sdax[0] sdbx[1]) yet mdadm lists the drives in reverse order.

So what we know this far, is that the 2nd drive in the server has died. This is good news, because if the first drive died, you would need to re-install grub into the MBR of the second drive, and make it the bootable drive, or else the system would refuse to boot.

1. First thing to do is shut down the machine and replace the dead drive with a new one. Make sure the replacement drive as at least the same number of cylinders as the previous drive or you will complicate things.

2. Next we need to know exactly what partitions are on the current good drive.

# /sbin/fdisk -l /dev/sda

Disk /dev/sda: 250.0 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 25 200781 fd Linux raid autodetect
/dev/sda2 26 2575 20482875 fd Linux raid autodetect
/dev/sda3 2576 3097 4192965 fd Linux raid autodetect
/dev/sda4 3098 30401 219319380 5 Extended
/dev/sda5 3098 30401 219319348+ fd Linux raid autodetect

Write that down (or copy/paste into notepad if you have the luxury of doing this from SSH rather than console). Now create the exact same partitions on the new drive. Make sure each partition has the same number of blocks as the respective partition on the good drive. Set the partition type to “Linux raid autodetect”. Also try not to get tripped up by the Extended partition, if you dont know why it is there or what it is for, you probably should not be rebuilding a raid, either get someone to do it for you or find out exactly what an extended partition is and why/when it is needed.

3. Add each partition back into the raid device.

mdadm --manage -add /dev/md0 /dev/sdb1
or
mdadm /dev/md0 -a /dev/sdb1

This will tell mdadm to add that partition into the raid device, and it will start an automatic rebuild of the data. It is probably not wise to do this in a peak usage time if you can help it as it will have a fair impact on system performance. Of course if the safety of your data is more important than a bit of system lag then go for it.

If you screwed up the partition sizes in step 2 then you will get an error when you try to add the partition to the array. Go back and fix it.

And there you have it, when mdadm finishes syncing the drives (this will take a while), you are back online with a fully functional raid, and the only downtime was to replace the dead hdd with a new one.