Skip to content

DellRAID

Jump to bottom
robnagler edited this page Mar 13, 2018 · 15 revisions

Configuring Dell RAID Controller (PERC)

Our RAID disks are really Just a Bunch of Disks (JBOD) as it is easier to manage disks from the file system. ZFS and Linux RAID want to talk to the disks directly, and it's easier to move such disks around.

We've encapsulated all the m620 setup code in idrac-m620-setup.sh, which is a sequence of functions you can selectively execute, including idrac.

RAID And Storage Configuration using RACADM Commands in iDRAC7 is a simple manual documenting the raid commands. RACADM Command Line Reference Guide for iDRAC7 1.50.50 and CMC 4.5 (p106) is the complete reference.

Configure JBOD

We first have to reset the RAID config:

$ idrac 1 storage resetconfig:RAID.Integrated.1-1
$ idrac 1 jobqueue create RAID.Integrated.1-1 -r pwrcycle -s TIME_NOW -e TIME_NA

After this completes, we'll follow ThornLabs instructions:

$ for p in $(idrac 1 raid get pdisks); do
    idrac 1 raid createvd:RAID.Integrated.1-1 -rl r0 -wp wt -rp nra -ss 64k "-pdkey:$p"a
done
$ idrac 1 job RAID.Integrated.1-1

This sets up all the physical disks (pdisks) as independent virtual disks with RAID0 (r0), write through (wt), no read ahead (nra), and 64k strip size (seems what people recommend for SSDs).

Wait a bit and verify that the vdisks have been created:

$ idrac 1 raid get vdisks -o
Disk.Virtual.0:RAID.Integrated.1-1
   Status                           = Ok
   DeviceDescription                = Virtual Disk 0 on Integrated RAID Controller 1
   Name                             = Virtual Disk 0
   RollupStatus                     = Ok
   State                            = Online
   OperationalState                 = Not applicable
   Layout                           = Raid-0
   Size                             = 931.00 GB
   SpanDepth                        = 1
   AvailableProtocols               = SATA
   MediaType                        = SSD
   ReadPolicy                       = No Read Ahead
   WritePolicy                      = Write Through
   StripeSize                       = 64K
   DiskCachePolicy                  = Enabled
   BadBlocksFound                   = NO
   Secured                          = NO
   RemainingRedundancy              = 0
   EnhancedCache                    = Not Applicable
   T10PIStatus                      = Disabled
   BlockSizeInBytes                 = 512

Performance

Testing performance of PERC H310 (non-RAID) vs motherboard S110:

lvcreate -y --wipesignatures y -L 20G -n test centos
mkfs.xfs /dev/mapper/centos-test
mount /dev/mapper/centos-test /mnt
cd /mnt
# sync on close (every 10G with enough RAM (64G))
i=$(date +%s); dd bs=1M count=10240 if=/dev/zero of=test conv=fdatasync; echo $(( $(date +%s) - $i ))
rm -f test
sync
# sync on write (every 1M)
i=$(date +%s); dd bs=1M count=10240 if=/dev/zero of=test oflag=dsync; echo $(( $(date +%s) - $i ))
cd
umount /mnt
lvremove /dev/mapper/centos-test

Write through, no read ahead on SSD. LUKS-encryptedfdatasync, dsync 10GB/test (1024^3=1073741824)

  • H310 E2650 64g 850 256g: 384MBps (26s), 156MBps (64s)
  • S110 E2660 128g 860 256g: 222MBps (45s), 131MBps (76s)
  • S110 E2660 64g 850 256g: 222MBps (45s), 129MBps (77s)
  • f1 S110 E2660 64g 850 2TB luks mdadm: 232MBps (43s), 85MBps (117s)
  • AHCI E2660 64g 850 256g: 217MBps (46s), 75MBps (133s)
  • f8 E2660 64g 850 88s
  • f13 2660 128g 1TB S110 no-mdadm no-luks: dsync 125s 129s
  • f13 2660 128g luks 1TB 119s
  • f14 H310 megaraid sas 2008 850 256g 128g 2650 dsync 63s 67s
  • f2 E2660 H310 64g 1TB (newer) luks no-mdadm: fdatasync 91s, 94s; dsync: 113s, 124s
  • f11 E2650 AHCI 192g 860 256g no-mdadm: fdatasync 44s; dsync 77s
  • f4 E2660 S110 64g 2x1TB (newer) no-luks mdadm: fdatasync 44s; dsync: 101s, 102s
  • f2 E2660 H310 MegaRAID SAS 2008 64g 2x1TB (newer) luks mdadm: fdatasync 98s,89s,63s; dsync: 137s,131s,126s
  • f3 E2660 H310 MegaRAID SAS 2008 64g 2x1TB (newer) luks mdadm: fdatasync 40s, 33s; dsync: 84s,89s
  • f4 E2660 AHCI 64g 2x1TB (newer) luks mdadm: fdatasync 43s; dsync: 111s, 112s

Some configuration:

  • mdadm: raid1
  • luks: aes xts-plain64 sha256
  • Intel e5-2650 v2 & e5-2660 v2
  • Samsung EVO 850/860, 64g/128g
  • Some 1TB 850s bought two years ago
  • f2 & f3: identical config (compared in bios and hdparm, lshw)
  • f4 same as f2 & f3, but AHCI or S110

References on performance of hardware RAID vs madam:

Clone this wiki locally