Creating Oracle ASM disks on EMC DSSD flash storage

Posted on by • Posted in ASM, DSSD, Linux, Oracle, Uncategorized • Tagged , • Leave a comment

EMC recently announced the availability of its DSSD rack scale flash storage appliance.

DSSD’s specs are impressive:

  • 10 Million IOPs
  • 100GB/S bandwidth
  • 144TB capacity
  • 100uS latency

The DSSD D5 takes a different approach to flash storage, by eliminating the latency prone networking layer typically handled by Fibre Channel, Infiniband or iSCSI. DSSD replaces this with a PCI extension through a proprietary PCIe Isley Card that connects servers to the D5.

DSSD also replaces conventional SSDs that package NAND technology and makes them look like fast spinning disks, with Flash Modules that eliminate much of the latency of mimicking spinning disks.

There’s a lot more to the D5 that just that, but I am taking a DBA centric approach, and all I want to know is, how do I consume this fast storage in my database?

The D5 natively supports an object store API, and a block interface. To use fast D5 storage on Oracle, we need to use the block interface, and present block devices to ASM.

To do this, first connect as root and make sure the block device driver is running:

[root@dssdnj04 ~]# ps -ef | grep blk
root       5030      1  0 Mar16 ?        00:28:57 /etc/opt/dssd/libexec/dssd-blkdev blkdev -v nj04
root       5032   5030  0 Mar16 ?        00:00:00 /etc/opt/dssd/libexec/dssd-blkdev blkdev -v nj04

Or you can check the contents of /etc/sysconfig/dssd-blkdev

[root@dssdnj04 ~]# cat /etc/sysconfig/dssd-blkdev

DSSD_BLKDEV_VOLUME_NAME="nj04"
DSSD_BLKDEV_LOGFILE_NAME="/var/log/dssd-blkdev.log"

This tells us we are consuming storage from the nj04 volume on the D5. We need to know this to create some ASM disks for ASM to use.

We should also check that the minimum IO size is 512 bytes:

[root@dssdnj04 ~]# cat /etc/modprobe.d/vpci.conf
options vpci vpci_blk_io_min=512

Now let’s create four ASM disks of 512GB each. Just like with regular flash storage, we should observe Oracle best practice of using multiple ASM disks in each disk group. This allows ASM to balance IO across multiple IO queues and avoid IO bottlenecks.

[root@dssdnj04 ~]# /opt/dssd/bin/flood create -F 4096 -l 512G -t block -V nj04 sidata01
[root@dssdnj04 ~]# /opt/dssd/bin/flood create -F 4096 -l 512G -t block -V nj04 sidata02
[root@dssdnj04 ~]# /opt/dssd/bin/flood create -F 4096 -l 512G -t block -V nj04 sidata03
[root@dssdnj04 ~]# /opt/dssd/bin/flood create -F 4096 -l 512G -t block -V nj04 sidata04

In the above example, -F is the fragment size used by DSSD. The -l switch sets the device size to 512G. The -t switch tells DSSD this is a block device, and the -V tells DSSD to create this block device from the nj04 volume.

To verify what we have created, we can use the flood ls command, with the volume name we used earlier; nj04

[root@dssdnj04 ~]# /opt/dssd/bin/flood ls -li -V nj04
b 549755813888  4096 2016-06-15T20:05:41 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-000000000000002d sidata01
b 549755813888  4096 2016-06-15T20:05:49 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-000000000000002e sidata02
b 549755813888  4096 2016-06-15T20:06:02 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-000000000000002f sidata03
b 549755813888  4096 2016-06-15T20:06:14 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000030 sidata04
b 1099511627776  4096 2016-06-15T20:07:35 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000031 sifra01
b 1099511627776  4096 2016-06-15T20:07:41 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000032 sifra02
b 1099511627776  4096 2016-06-15T20:07:46 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000033 sifra03
b 1099511627776  4096 2016-06-15T20:07:51 81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000034 sifra04

Note – if you want to destroy a DSSD volume, use the flood destroy command:

[root@dssdnj04 ~]# flood destroy -V nj04 sidata05

Having created the volumes on the D5, we now need to use UDEV rules in Linux to ensure the devices and readable and writable by the Oracle user.

The document Deploying Oracle Databases on EMC DSSD: Best Known Methods contains a short script to generate these rules, after which they need to be added to a suitable rules files such as /dev/udev/rules.d/99-oracleasm-rules.

# cr_udev.sh - create corresponding udev configuration for /etc/sysconfig/dssd-blkdev
#
# James Morle - DSSD EMC - morlej@dssd.com
#

source /etc/sysconfig/dssd-blkdev

# list of volumes is in $DSSD_BLKDEV_VOLUME_NAME

echo -e "# This file is autogenerated by sol-arch/scripts/cr_udev.sh\n"
for i in `echo $DSSD_BLKDEV_VOLUME_NAME |tr "," " "`
do
echo -e "\n# Volume: $i"
flood ls -lhi -V $i | awk '
{ printf "SUBSYSTEM==\"block\" KERNEL==\"dssd????\" ENV{DSSD_OID}==\"%s\", OWNER+=\"oracle\", GROUP+=\"dba\", SYMLINK+=\"asmdisks/%s\"\n",$5,$NF }
'
done

Executing the above script generates the rules we need for UDEV:

SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-000000000000002d", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sidata01"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-000000000000002e", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sidata02"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-000000000000002f", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sidata03"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000030", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sidata04"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000031", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sifra01"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000032", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sifra02"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000033", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sifra03"
SUBSYSTEM=="block" KERNEL=="dssd????" ENV{DSSD_OID}=="81a1aaa9-a3d2-4ae1-af7d-dedecdfbc1da-0000000000000034", OWNER+="oracle", GROUP+="dba", SYMLINK+="asmdisks/sifra04"

Verify the rules look correct (we are not including and DSSD devices that are not ASM disks), and then add them to the /dev/udev/rules.d/99-oracleasm-rules file.

Having updated the rules file, restart UDEV as follows:

[root@dssdnj04 ~]# start_udev
Starting udev: [  OK  ]

And now we can check the rules file generated the ASM disks by looking in /dev/asmdisks

[root@dssdnj04 ~]# ls /dev/asm*
sidata01  sidata03  sifra01  sifra03
sidata02  sidata04  sifra02  sifra04

Next, connect to the account that owns the ASM instance, and create the new disk group:

CREATE DISKGROUP DATA EXTERNAL REDUNDANCY
  DISK
'/dev/asmdisks/sidata01' NAME DATA_0001, 
'/dev/asmdisks/sidata02' NAME DATA_0002,
'/dev/asmdisks/sidata03' NAME DATA_0003, 
'/dev/asmdisks/sidata04' NAME DATA_0004
ATTRIBUTE 
  'AU_SIZE' = '4M',
  'COMPATIBLE.RDBMS' = '12.1.0.1',
  'COMPATIBLE.ASM' = '12.1.0.1'
;

Leave a comment