Raw Device Mapping of local SATA disks on ESXi

       680 words, 4 minutes

My brand new ESXi machine is a “standard” Intel Z68 powered PC. It has three disks inside: two SSD and a SATA disk. My idea is to use the first SSD to host some VM (production VM, virtual ESXi, virtual Hyper-V…) and to connect the two other disks to a FreeNAS virtual instance. This enables to present a ZFS storage to the virtual hypervisors.

Here’s the way to attach two of the SATA drives to a single virtual machine using RDM technics. BTW, I’m using ESXi v5.0.0.

First of all, you’ll need an SSH access to the ESXi. SSH is enabled from the VI client, in the Configuration/Security Profile section. When “SSH” and “ESXi Shell” services are enabled, connect to the ESXi using your preferred SSH client.

  # uname -a
  VMkernel esxi.tumfatig.net 5.0.0 #1 SMP Release build-469512 Aug 18 2011 18:32:24 x86_64 unknown

The first thing to do is to get the identifiers for your local storage. What you will need is the path to the devices:

  # esxcfg-scsidevs -l | egrep '^[^ ]|Display Name|Vendor|Devfs Path'
  mpx.vmhba32:C0:T0:L0
     Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
     Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
     Vendor: Generic-  Model: SD/MMC            Revis: 1.00 
  t10.ATA_____M42DCT064M4SSD2__________________________00000000111703068787
     Display Name: Local ATA Disk (t10.ATA_____M42DCT064M4SSD2__________________________00000000111703068787)
     Devfs Path: /vmfs/devices/disks/t10.ATA_____M42DCT064M4SSD2__________________________00000000111703068787
     Vendor: ATA       Model: M4-CT064M4SSD2    Revis: 0309 
  t10.ATA_____SAMSUNG_MMDOE28G5MPP2D0VA________________SE853A9829__________
     Display Name: Local ATA Disk (t10.ATA_____SAMSUNG_MMDOE28G5MPP2D0VA________________SE853A9829__________)
     Devfs Path: /vmfs/devices/disks/t10.ATA_____SAMSUNG_MMDOE28G5MPP2D0VA________________SE853A9829__________
     Vendor: ATA       Model: SAMSUNG MMDOE28G  Revis: VAM0 
  t10.ATA_____WDC_WD2500BEVT2D75ZCT2________________________WD2DWX90EC9JDT19
     Display Name: Local ATA Disk (t10.ATA_____WDC_WD2500BEVT2D75ZCT2________________________WD2DWX90EC9JDT19)
     Devfs Path: /vmfs/devices/disks/t10.ATA_____WDC_WD2500BEVT2D75ZCT2________________________WD2DWX90EC9JDT19
     Vendor: ATA       Model: WDC WD2500BEVT-7  Revis: 11.0

In this example, I did a quite complex `grep`. You may want to use “esxcfg-scsidevs -l | grep 'Devfs Path'” to only get the device paths.

So in my configuration, I want to attach the Crucial M4 SSD and the Western Digital SATA disks to a virtual instance. The next thing to do is to create the RDM passthrough devices. What’s to be done is create the VMDK file that links to the physical device. That link is stored in some ESXi datastore and will be attached to the virtual machine:

  # vmkfstools -z /vmfs/devices/disks/t10.ATA_____M42DCT064M4SSD2__________________________00000000111703068787 /vmfs/volumes/SSD_datastore/rdmssd.vmdk -a lsilogic
  
  # vmkfstools -z /vmfs/devices/disks/t10.ATA_____WDC_WD2500BEVT2D75ZCT2________________________WD2DWX90EC9JDT19 /vmfs/volumes/SSD_datastore/rdmsata.vmdk -a lsilogic

Let’s have a look at what’s been created:

  # cd /vmfs/volumes/SSD_datastore
  /vmfs/volumes/4f46bbfc-4a0314f1-2663-00012e3d309f # ls -alh rdm*
  -rw-------    1 root     root       232.9G Feb 25 12:16 rdmsata-rdmp.vmdk
  -rw-------    1 root     root          496 Feb 26 12:15 rdmsata.vmdk
  -rw-------    1 root     root        59.6G Feb 25 12:15 rdmssd-rdmp.vmdk
  -rw-------    1 root     root          494 Feb 26 12:15 rdmssd.vmdk
  /vmfs/volumes/4f46bbfc-4a0314f1-2663-00012e3d309f #

You will see the links and the files used for direct attaching. The “rdmp” files claim to use some of the space on the datastore but they don’t actually do.

Now, you can attach the RDM to the virtual machine. Use the vSphere Client to edit the virtual machine settings. In the “Hardware” section, add a new “Hard Disk”. Choose the “Use an existing virtual disk” option and select one of the vmdk you just created. I attached “rdmssd.vmdk” and “rdmsata.vmdk” in that particular order and use the “SCSI(1:0)” and “SCSI (1:1)” Virtual Device Node ; just to be able to refer to them easily in the virtual machine.

Once this is done, boot the virtual machine and check for the disks. In mine, what I see is:

  # dmesg | grep "^da"
  da0 at mpt0 bus 0 scbus2 target 0 lun 0
  da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device 
  da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit)
  da0: Command Queueing enabled
  da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)
  da1 at mpt1 bus 0 scbus3 target 0 lun 0
  da1: <ATA M4-CT064M4SSD2 0309> Fixed Direct Access SCSI-5 device 
  da1: 6.600MB/s transfers (16bit)
  da1: 61057MB (125045424 512 byte sectors: 255H 63S/T 7783C)
  da2 at mpt1 bus 0 scbus3 target 1 lun 0
  da2: <ATA WDC WD2500BEVT-7 11.0> Fixed Direct Access SCSI-5 device 
  da2: 6.600MB/s transfers (16bit)
  da2: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C)

As you can see, the first disk in a “standard” virtual disk image. The two other disks appears as “real” disks thanks to the Raw Device Mapping. Now the storage can be configured as wished in the virtual machine.

Happy storing!

Related posts: