Sunday, 5 May 2013

Centos 6/RHEL Mount Logical Volumes using Live CD

Occasionally, you might have a problem develop which stops you gaining access to your system either as user or root, even to do a rescue operation. This could be for a variety of reasons, such as you change a crucial script or an external script you use causes a system lock up. 

It means you have to mount the Logical Volumes on which your Centos 6 system is based, find the offending file or script and fix, delete or replace it using an alternative method.

So you need to perform a rescue operation whilst using a Live CD to gain access, because in the case of user/root login failure this may probably be the best or only way to do it.

As a recent example a maven shell script in the /etc/profile.d directory was incorrect and caused all system commands to disappear, ie no ls, cat, rm, su, sudo. Additionally, it prevented logging on as anyone, so the steps below were taken in order to fix the problem.

Use a Centos or Debian Live CD because when you are attempting a rescue you don't want the additional stress of hunting down needed but potentially obscure packages, you just want to be able to get them easily, do the job and finish. The package we need here is lvm2 which is in both Centos and Debian repos.

So first download Centos 6.4 Live CD iso or Debian Live CD iso and burn it to a disk or put it on a USB stick. (Not described in this post). 

Once done, boot from it, whether disk or USB and once you are in the live environment you can bring up the terminal emulator or use the console, run fdisk and list the available drives. Use
Ctrl+Alt+F4 to stop X server and go to the console. The live disk uses sudo for root access.

$ sudo fdisk -l

  Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3891    31254426    7  HPFS/NTFS
/dev/sda2            3892        3955      512000   83  Linux
/dev/sda3            3955       38914   280803328   8e  Linux LVM
 

The sda3 entry is not suitable to mount in the normal manner because you do not have the correct path, so you need to do a little more probing to find this.

First install the tools package with apt

$ sudo apt-get install lvm2

And issue the command to find the LV Group which is under VG

$ sudo pvs                  

 PV         VG        Fmt  Attr PSize   PFree
 /dev/sda3  vg_centos lvm2 a--  267.79g    0 

'vg_centos' is fine to start with, but we need a little more.

Issue the command to display the directories and path.

$ sudo lvdisplay            

 --- Logical volume ---
  LV Path                /dev/vg_centos/lv_root
  LV Name                lv_root
  VG Name                vg_centos
  LV UUID                tiDVyK-ct0t-8GOJ-UuRn-uDWI-95jX-bqQWfy
  LV Write Access        read/write
  LV Creation host, time Centos, 2013-04-21 20:13:14 +0100
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
  
  --- Logical volume ---
  LV Path                /dev/vg_centos/lv_home
  LV Name                lv_home
  VG Name                vg_centos
  LV UUID                yUzkjk-RMk0-0k77-NCkh-l2WB-y0So-gMzjDT
  LV Write Access        read/write
  LV Creation host, time Centos, 2013-04-21 20:13:29 +0100
  LV Status              available
  # open                 1
  LV Size                211.84 GiB
  Current LE             54231
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2


You will probably have a swap also but above just shows the directories we are interested in.  

The LV Path is what we use to mount the Logical Volumes, and now
we have the full and correct path, they can be mounted as normal, we just need a directory to mount them into, so create a newroot directory. (As unprivileged user or root)

$ sudo mkdir newroot                      

$ sudo mount /dev/vg_centos/lv_root /newroot

$ sudo mount /dev/vg_centos/lv_home /newroot

Depending which directory you want to access. 

Use vgchange to make them active 

$ sudo vgchange -a y
  3 logical volume(s) in volume group "vg_centos" now active


Now you can navigate to the /newroot directory either in the terminal or using the file manager and access the file you need to fix, replace or delete.

Straighforward mounting of the Volume Group in Centos 6 using
a Live CD.

Using chroot

Use chroot if necessary as in the image below, doing this and then running the mount command shows the newly mounted volumes for the 'damaged' system as they appear in the newroot directory


Test already mounted volumes

You can test what is mounted by using one of below as root

# df -H

Filesystem             Size   Used  Avail Use% Mounted on
/dev/mapper/vg_centos-lv_root
                        53G   5.5G    45G  11% /
tmpfs                  3.2G      0   3.2G   0% /dev/shm
/dev/sdb2              508M    33M   450M   7% /boot
/dev/mapper/vg_centos-lv_home
                       224G   9.0G   204G   5% /home


# mount

/dev/mapper/vg_centos-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sdb2 on /boot type ext4 (rw)
/dev/mapper/vg_centos-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)