Replace Linux RAID disk the ‘right’ way

Well before you power to replace the disk - can you be sure it will power back on?
Some distros have the bad habit of only install GRUB to one physical disk and if that disk dies…RAID wont save you (but a boot CD will…)

Firstly I like to confirm if /boot is configured in RAID 1. I then usually also install GRUB to all the physical disks via:

grub-install /dev/sda

and sdb and sdc etc. Then I power off the server and replace the disk.

After powering on the server, sometimes it won’t boot due to the new disk getting booted first, so make sure to select another disk in the BIOS boot-up menu.

After the operating system returns it’s a matter of recreating the partitions on the new disk, before trying to add it back into the RAID array. My servers all have the same disk sizes in the array and the same partition layouts, so to recreate them on the old disk I just perform:

sfdisk -d /dev/existing-disk | sfdisk /dev/new-disk

Confirm the new disks get the correct layout via:

cat /proc/partitions

and then add the partition back to the RAID array:

mdadm --add /dev/mdX /dev/sdXX

And then finally confirm its rebuilding via:

mdadm --detail /dev/mdX

OR

cat /proc/mdstat

How to mount dd images under Linux

For a raw filesystem try:

fdisk -l harddrive.img
mount -o ro,loop,offset=xxxxxxxxx harddrive.img /mnt/loop

or for filesystems with volume groups etc try:

losetup /dev/loop0 disk.img
kpartx -a /dev/loop0

Then to mount the first partition:

mount /dev/mapper/loop0p1 /mnt

Or to activate the volume group then mount the logical volume:

vgscan
vgchange -ay vg
mount /dev/vg/lv /mnt

Hope that helps.

How to measure IOPS with linux

So many times I need to measure the amount of IOPS on a Linux disk/storage system. While there are many tools for the jobs they just don’t seem to give you a ‘number’. For example Splunk indexers require 1200+ IOPS according to hardware recommendation guides but how do you find out if your any where close to that number? Use ‘bonnie++’, ‘iozone’ or perhaps ‘fio’? Well use any of those tools will create the type of read / write sequence you would like to replicate - but where the damn magic number???

Easiest two ways are:
Method #1:
run iozone -a (or bonnie++) in one screen then in another session / terminal use nmon, pressing D (capital D) to get disk stats and get the number from the Xfers column. This is your magic number (or IOPS reading)

┌nmon─14i─────────────────────Hostname=reddragon─────Refresh= 2secs ───19:51.57─
│ Disk I/O ──/proc/diskstats────mostly in KB/s─────Warning:contains duplicates─
│DiskName Busy    Read    Write       Xfers   Size  Peak%  Peak-RW    InFlight
│sda       99%    699.9     14.0KB/s  178.0   4.0KB  493%    3658.8KB/s   1   
│sda1       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0   
│sda2      99%    699.9     14.0KB/s  178.0   4.0KB  493%    3658.8KB/s   1   
│dm-0       0%      0.0      0.0KB/s    0.0   0.0KB    0%       0.0KB/s   0   
│dm-1      99%    699.9     14.0KB/s  178.5   4.0KB  494%    3658.8KB/s   1   
│dm-2       0%      0.0      0.0KB/s    0.0   0.0KB   76%    2553.5KB/s   0   
│Totals Read-MB/s=2.1      Writes-MB/s=0.0      Transfers/sec=534.4 

In the above example I’m getting about 178 IOPS for my disk ’sda’

Method #2:
run fio with the correct workload (google how to use fio) and while it’s running it will actually tell you the IOPS.

[root@reddragon ~]# fio random-read-test.fio 
random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
Jobs: 1 (f=1): [r] [85.1% done] [736K/0K/0K /s] [184 /0 /0  iops] [eta 00m:28s]

In this example I am getting 184 IOPS. Also if you wait until fio finishes it run - you can the IOPS reading from there. Eg.

random-read: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.0.13
Starting 1 process
Jobs: 1 (f=1): [r] [98.9% done] [2224K/0K/0K /s] [556 /0 /0  iops] [eta 00m:02s]
random-read: (groupid=0, jobs=1): err= 0: pid=7239: Tue Feb 25 16:49:16 2014
  read : io=131072KB, bw=747406 B/s, iops=182 , runt=179578msec
    clat (usec): min=107 , max=117530 , avg=5473.62, stdev=4112.08
     lat (usec): min=107 , max=117531 , avg=5473.93, stdev=4112.08
    clat percentiles (usec):
     |  1.00th=[  245],  5.00th=[  302], 10.00th=[  370], 20.00th=[ 2480],
<SNIP>

As you can see: iops=182 - pretty consistent with the other results!