This week I had to replace a drive in a linux based voip system, the drive was a member of a RAID 1 array holding the OS, boot partition, and swap partition. Replacing the drive was pretty simple, just a one for one swap, and then rebuilding the array was a little more involved then what most of us are used to with hardware RAID systems.
Basically I had to manually copy the partition table from the remaining original drive to the new one, and then tell MD (the linux software raid driver) that I wanted to to add the new drive to the array and to mirror the three partitions to the new drive.
The problem is that because this is a software RAID we are going to burn most of our CPU power and Disk I/O to rebuild the array. Because this system handles voice communications we cannot have all those systems resources being used to rebuild the array because that will severely affect voice services.
To counteract these problems there are two variables that we can modify to slow down the rebuild process so that critical voice services are not affected. These variables are:
These variables are located in ‘/proc/sys/dev/raid/’ and do exactly what you might expect. The ‘speed_limit_max’ variable limits the rebuilt rate to a certain number of KBps. And the ‘speed_limit_min’ sets the minimum rebuilding rate in KBps. By default the minimum is set to 1,000 KBps and the maximum is set to 200,000 KBps, which leaves a alot of room for variation.
You could use these variables two different ways, the first would be to issue the following command to turn up the minimum rebuild rate, which would increase the priority of rebuilding and get the drives rebuilt faster:
'echo -n 10000 > /proc/sys/dev/raid/speed_limit_min'
But if you are in my situation you can decrease the speed_limit_max so that the rebuilding priority is forced to slow down and free up resources for the rest of the system to use. You can do this by running the following command:
'echo -n 1000 > /proc/sys/dev/raid/speed_limit_max'
To check to see how fast your arrays are rebuilding you can run:
With these commands I was able to control the rebuild rate and allow for normal system operation to keep running while the drive was rebuilding.