Using storcli to Manage MegaRaid Disk Arrays


The tool to handle the LSI (now Broadcom) MegaRaid raid controllers, LSI 3108 in my case, is called storcli and may be downloaded from Broadcom's website .

When installed, the package resides at /opt/MegaRAID/storcli/ and there is a 64bit binary called storcli64.
To have it in my execution path I created a symbolic link in /usr/local/sbin.

Replacing a Good Disk by a Bigger One

To replace a disk still good and working in the array, e.g. by a bigger one to increase the capacity of the array, you begin with querying the system on it's hardware configuration:

[root@martes ~]# storcli show

This command returns something like:

CLI Version = 007.1316.0000.0000 Mar 12, 2020
...
Number of Controllers = 1
Host Name = ...
Operating System  = Linux ...

System Overview :
===============

--------------------------------------------------------------------------------
Ctl Model               Ports PDs DGs DNOpt VDs VNOpt BBU sPR DS  EHS ASOs Hlth 
--------------------------------------------------------------------------------
  0 LSI3108MegaRAIDROMB     8   8   1     0   1     0 Opt On  1&2 Y      3 Opt  
--------------------------------------------------------------------------------
...

The output above shows that in the system there is Controller (Ctl) 0 with 8 Physical Disks (PD) in one Device Group (DG) making one Virtual Drive (VD).
The array is in Optimal Health (Hlth - Opt) aswell as the Battery Backup Unit (BBU - Opt).

Next step would be to query the details especially about the physical disks:

[root@martes ~]# storcli /c0 show

This produces a detailed output:

Generating detailed summary of the adapter, it may take a while to complete.

CLI Version = 007.1316.0000.0000 Mar 12, 2020
...
TOPOLOGY :
========

----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type  State BT      Size PDC  PI SED DS3  FSpace TR 
----------------------------------------------------------------------------
 0 -   -   -        -   RAID6 Optl  N  32.745 TB dflt N  N   dflt N      N  
 0 0   -   -        -   RAID6 Optl  N  32.745 TB dflt N  N   dflt N      N  
 0 0   0   252:0    9   DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   1   252:1    10  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   2   252:2    12  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   3   252:3    11  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   4   252:4    13  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   5   252:5    14  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   6   252:6    16  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
 0 0   7   252:7    15  DRIVE Onln  N   5.457 TB dflt N  N   dflt -      N  
----------------------------------------------------------------------------
...
VD LIST :
=======

-----------------------------------------------------------------
DG/VD TYPE  State Access Consist Cache Cac sCC      Size Name    
-----------------------------------------------------------------
0/0   RAID6 Optl  RW     Yes     RAWBD -   ON  32.745 TB rd6Disk 
-----------------------------------------------------------------
...

In the output there are serveral paragraphs of which the TOPOLOGY and VD LIST are shown above. From that output may be seen that there is one Virtual Drive of eight disks configured as RAID6.

The PD LIST from the same output shows the details on all installed physical disks.

PD LIST :
=======

-------------------------------------------------------------------------------
EID:Slt DID State DG     Size Intf Med SED PI SeSz Model               Sp Type 
-------------------------------------------------------------------------------
252:0     9 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:1    10 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:2    12 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:3    11 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:4    13 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:5    14 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:6    16 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
252:7    15 Onln   0 5.457 TB SATA HDD N   N  512B ST6000NM0024-1HT17Z U  -    
-------------------------------------------------------------------------------

To remove disk from slot 0 on enclosure (EID) 252 we begin with identifying the physical drive in it's box.

[root@martes ~]# storcli /c0/e252/s0 start locate

This should lit up the identification LED of the disk disk drawer in question to make shure not to unplug the wrong disk.a If the enclosure/backplane does not support a dedicated identification LED the identification by the disk activity LED should be enabled:

[root@martes ~]# storcli /c0 set activityforlocate=on

Next step is to set the disk offline using the command:

[root@martes ~]# storcli /c0/e252/s0 set offline

To silence an eventually triggered acustic alarm the following command may be used:

storcli /c0 set alarm=silence

Some sources in the internet recommend the sequence:

[root@martes ~]# storcli /c0/e252/s0 set offline
[root@martes ~]# storcli /c0/e252/s0 set missing
[root@martes ~]# storcli /c0/e252/s0 spindown

and then remove the drive physically.

This didn't work in my case and the drive went automatically into rebuild state when issuing set missing and spindown returned an error saying that spindown is not supported in that state of the drive.
I simply issued the set offline command, then removed the drive and installed the new one.
After installing the new drive, it went automatically into rebuild state.

With the following command the state of the rebuilding process may be observed:

[root@martes ~]# storcli /c0/e252/s0 show rebuild
CLI Version = 007.1316.0000.0000 Mar 12, 2020
...
------------------------------------------------------
Drive-ID    Progress% Status      Estimated Time Left 
------------------------------------------------------
/c0/e252/s0        13 In progress 9 Hours 58 Minutes
------------------------------------------------------

The default rebuild rate is set to 30% but can be increased by using the command:

[root@martes ~]# storcli /c0 set rebuildrate=80%

Increasing the rebuild rate reduces the I/O performance, of course.

Expand the Size of the Array

After having replaced the three disks in the RAID6 array with bigger ones, the RAID controller must be made aware of that.
The following command should do the trick:

storcli /c0/v0 expand Size=14TB expandarray

This didn't work at first, I don't know why.
After several tries over several day, the RAID controller finally reported the new disk and the new array sizes. Sigh....

Resizing of the partition and resizing of the file system are done by:

parted /dev/sda resizepart 1 -1s
resize2fs /dev/sda1

These commands grow the partition 1 to full size of the disk and resize the ext4 file system to the size of the partition.

Add a Disk to the RAID Array

When adding an additinal disk to the RAID controller it initially gets listed as "UGood" (Unused Good).

PD LIST :
=======

-----------------------------------------------------------------------------
EID:Slt DID State DG      Size Intf Med SED PI SeSz Model            Sp Type 
-----------------------------------------------------------------------------
252:0    16 Onln   0 14.551 TB SAS  HDD N   N  512B ST16000NM002G    U  -    
252:1    17 Onln   0 14.551 TB SAS  HDD N   N  512B ST16000NM002G    U  -    
252:2    18 Onln   0 14.551 TB SAS  HDD N   N  512B ST16000NM002G    U  -    
252:3    21 UGood  - 14.551 TB SAS  HDD N   N  512B ST16000NM002G    U  -    
-----------------------------------------------------------------------------

To add the additional disk to the virtual drive:

storcli /c0/v0 start migrate type=RAID6 option=add drives=252:3

The migration will take quite some time.
The following command shows the status of the migration:

storcli /c0/v0 show migrate
--------------------------------------------------------------
VD Operation Progress% Status      Estimated Time Left        
--------------------------------------------------------------
 0 Migrate           0 In progress 3 Days 10 Hours 48 Minutes 
--------------------------------------------------------------

storcli Manual Flaws

The manual about storcli which can be downloaded from Broadcom (January 2013 53419-00, Rev. F) is not everwhere correct.
E. g. the start migrate command including example is wrong:
storcli /c0/v0 start migrate type=r6 option=add disk=e252:s3
is not an accepted syntax.
It should be:
storcli /c0/v0 start migrate type=RAID6 option=add drives=252:3

There were more command syntax I encountered, which didn't work, but I can't remember the details.
Usually an internet search reveals the correct syntax.

Links to some interesting articles on this topic

line
linux logo Powered by Apache
line
This site maintained by:
lukas.zimmermann@unibas.ch
My public PGP key
last updated: 2020-06-04 Valid CSS! Valid XHTML 1.0 Strict