In this post I am going to explore deleting a storage snapshot of an Oracle 19c RAC database created on a Dell PowerMax storage array.

This post is a companion to the video Dell PowerMax – Refresh a storage snapshot of Oracle RAC database. and a follow on from the blog post Create a storage snapshot of an Oracle 19c RAC database using ASM diskgroups with Dell PowerMax.

This blog post assumes some basic knowledge of PowerMax storage concepts. If you need a basic introduction please check out my PowerMax basics video.

These examples will use the Solutions Enabler command line method of managing the PowerMax, unlike the video which uses the Unisphere graphical interface.

In the previous blog posts we used Dell PowerMax to create and then refresh a storage snapshot of an Oracle 19c RAC database. The process included creating the storage snapshot, creating link devices, linking the snapshot to the desired generation, creating the link storage group masking view, and then exposing the snapshot to the target hosts.

In the previous post I showed how to use Solutions Enabler to query which snapshots are linked and to which generation. From that approach we were able to determine that gctsnap_swingdata_sg and gctsnap_swingredo_sg were the linked snapshots.

As with the snapshot refresh process we will need to shut down any cloned database, stop the snapshot ASM diskgroups and drop them from the target server. We should also unlabel any ASM FD disks to avoid conflicts if we decide to recreate this snapshot again at a later time.

With the cloned database shutdown and ASM diskgroups dropped, the next step is to remove the link devices from the masking view. Once the link devices are removed, a refresh of the SCSI bus will cause the Powerpath managed devices to disappear.

I use the following simple script to query Solutions Enabler to determine which devices are linked to a snapshot, and which SGs those devices are in.

export SNAP_NAME=gctsnap_swingdata_sg

# enumerate all vols in the named SG
for sd in `symsnapvx list -linked | grep $SNAP_NAME | awk '{print $4}'`
do 
  # enumerate all SGs
  for sg in `symsg list -detail | awk '{print $1}' | grep -v "\----"`
  do
   # see if this is an SG
   chk=`symsg show $sg 2>/dev/null`
   status=$?
   # this is an SG
   if [ $status -eq 0 ]
    then
      # see if the vol exists in this SG
      out=`symsg show $sg 2>/dev/null | grep $sd`
      length=${#out}
      if [ $length -ne 0 ]
      then 
        echo 'vol '$sd' found in sg '$sg
      fi
    fi
  done
done

vol 00142 found in sg swing_sg_lnk
vol 00142 found in sg swingdata_sg_lnk
vol 00143 found in sg swing_sg_lnk
vol 00143 found in sg swingdata_sg_lnk
vol 00144 found in sg swing_sg_lnk
vol 00144 found in sg swingdata_sg_lnk
vol 00145 found in sg swing_sg_lnk
vol 00145 found in sg swingdata_sg_lnk

So for snapshot gctsnap_swingdata_sg we can see there are four link devices – 00142 through 00145 – and that each link device is a member of two SGs – swingdata_sg_lnk and swing_sg_lnk

Repeating this operation for gctsnap_swingredo_sg yields the following:

vol 00146 found in sg swing_sg_lnk
vol 00146 found in sg swingredo_sg_lnk
vol 00147 found in sg swing_sg_lnk
vol 00147 found in sg swingredo_sg_lnk
vol 00148 found in sg swing_sg_lnk
vol 00148 found in sg swingredo_sg_lnk

Next we need to see which masking view these three SGs are in:

[root@dsib0252 ~]# symaccess list -detail -v | grep -B1 -A8 -e swingdata_sg_lnk -e swingredo_sg_lnk -e swing_sg_lnk       

Storage Group Name    : swing_sg_lnk (IsParent)
Device Count          : 7
Storage Group Count   : 2
Masking View Count    : 0
Last update time      : 02:42:21 PM on Thu May 12,2022
Group last update time: 02:42:21 PM on Thu May 12,2022
Masking View Names    : snap_mv

Storage Group Name    : swingdata_sg (IsChild)
--

Storage Group Name    : swingdata_sg_lnk (IsChild)
Device Count          : 4
Storage Group Count   : 1
Masking View Count    : 0
Last update time      : 02:42:21 PM on Thu May 12,2022
Group last update time: 02:42:21 PM on Thu May 12,2022
Masking View Names    : snap_mv *

Storage Group Name    : swingfra_sg
--

Storage Group Name    : swingredo_sg_lnk (IsChild)
Device Count          : 3
Storage Group Count   : 1
Masking View Count    : 0
Last update time      : 02:42:21 PM on Thu May 12,2022
Group last update time: 02:42:21 PM on Thu May 12,2022
Masking View Names    : snap_mv *


* Denotes Masking Views through a cascaded group

So we can now see that all three of our link SGs are presented through the snap_mv masking. Removing them form the masking view, or simply deleting the masking view, will remove the snapshot volumes from the target hosts:

[root@dsib0252 ~]# symaccess delete view -name snap_mv

Now that the masking view is deleted, we need to rescan the SCSI bus on each of the target servers, and in each case also instruct Powerpath to clean up dead paths:

[root@dsib0251]# /usr/bin/rescan-scsi-bus.sh

[root@dsib0251]# echo "a" | powermt check dev=all

If you are using UDEV rules for device management in preference to ASM Filter Drivers or ASMLib, now would be a good time to delete those rules.

With the disks removed from the target hosts we can now delete the link devices. Before we do anything else, let’s remind ourselves which devices we are talking about:

[root@dsib0252 ~]# symsnapvx list -linked

Symmetrix ID              : 000120200304    (Microcode Version: 6079)

--------------------------------------------------------------------------------
Sym                                         Link  Flgs
Dev   Snapshot Name                    Gen  Dev   FCMDS Snapshot Timestamp
----- -------------------------------- ---- ----- ----- ------------------------
0014B gctsnap_swingdata_sg                1 00142 .D.X. Wed May  4 13:49:34 2022
0014C gctsnap_swingdata_sg                1 00143 .D.X. Wed May  4 13:49:34 2022
0014D gctsnap_swingdata_sg                1 00144 .D.X. Wed May  4 13:49:34 2022
0014E gctsnap_swingdata_sg                1 00145 .D.X. Wed May  4 13:49:34 2022
00153 gctsnap_swingredo_sg                1 00146 .D.X. Wed May  4 13:49:45 2022
00154 gctsnap_swingredo_sg                1 00147 .D.X. Wed May  4 13:49:45 2022
00155 gctsnap_swingredo_sg                1 00148 .D.X. Wed May  4 13:49:45 2022

We can see that our snapshot gctsnap_swingdata_sg which is based on Sym devices 0014B through 0014E is linked to link devices 00142 through 00145, and snapshot gctsnap_swingredo_sg, based on Sym devices 00153 through 00155 is linked to link devices 00146 through 00148.

We saw from the earlier query that our link devices are members of SGs swing_sg_lnk, which is a parent SG to swingdata_sg_lnk and swingredo_sg_lnk. Below we can see that the parent SG encompasses link devices 00142 through 00148.

[root@dsib0251 ~]# symsg show swing_sg_lnk 
Name: swing_sg_lnk
   Symmetrix ID               : 000120200304
   Last updated at            : Thu May 12 14:42:21 2022
   Masking Views              : No
   FAST Managed               : No
   Service Level Name         : 
   Workload                   : 
   SRP Name                   : 
   VP Saved (%)               : N/A
   Compression Enabled        : N/A
   Data Reduction Ratio       : N/A
   Host I/O Limit             : None
   Host I/O Limit MB/Sec      : N/A
   Host I/O Limit IO/Sec      : N/A
   Dynamic Distribution       : N/A
   Number of Gatekeepers      :    0
   Child Storage Groups (2):
    {
    ----------------------------------------------------------------
    Storage Group Name
    ----------------------------------------------------------------
    swingdata_sg_lnk
    swingredo_sg_lnk
    }
   Devices (7):
    {
    --------------------------------------------------------------------
    Sym                             Device                   Provisioned
    Dev    Pdev Name                Config         Attr  Sts        (GB)
    --------------------------------------------------------------------
    00142  N/A                      TDEV                  RW      1024.0
    00143  N/A                      TDEV                  RW      1024.0
    00144  N/A                      TDEV                  RW      1024.0
    00145  N/A                      TDEV                  RW      1024.0
    00146  N/A                      TDEV                  RW        50.0
    00147  N/A                      TDEV                  RW        50.0
    00148  N/A                      TDEV                  RW        50.0
    }

We cannot delete an SG that has child SGs. And we cannot delete an SG that has devices contained within it. Furthermore we cannot delete devices that are part of a SG. Therefore we need to dismantle the SGs and remove the link devices from it.

First, let’s remove the child SGs:

symsg -sg swing_sg_lnk remove sg swingdata_sg_lnk,swingredo_sg_lnk

The SG swing_sg_lnk is now empty. SGs swingdata_sg_lnk and swingredo_sg_lnk still have devices. Let’s remove those:

symsg rmall -sg swingdata_sg_lnk -devs 00142:00145
symsg rmall -sg swingredo_sg_lnk -devs 00146:00148

Now we can delete the link Storage Groups:

symsg delete swingdata_sg_lnk 
symsg delete swingredo_sg_lnk 
symsg delete swing_sg_lnk 

Next we can unlink the snapshots from the link devices:.

[root@dsib0251 ~]# symsnapvx  unlink -snapshot_name gctsnap_swingdata_sg -symforce -devs 0014B:0014E -lndevs 00142:00145 -generation 1

Unlink operation execution is in progress for the device range(s). Please wait...

    Polling for Unlink................................................Started.
    Polling for Unlink................................................Done.

Unlink operation successfully executed for the device range(s)

[root@dsib0251 ~]# symsnapvx  unlink -snapshot_name gctsnap_swingredo_sg -symforce -devs 00153:00155 -lndevs 00146:00148 -generation 1

Unlink operation execution is in progress for the device range(s). Please wait...

    Polling for Unlink................................................Started.
    Polling for Unlink................................................Done.

Unlink operation successfully executed for the device range(s)

With the snapshots unlinked, we can now delete the snapshots:.

[root@dsib0251 ~]# symsnapvx -sg swingdata_sg terminate -snapshot_name gctsnap_swingdata_sg -generation 1

Terminate operation execution is in progress for the storage group swingdata_sg. Please wait...

    Polling for Terminate.............................................Started.
    Polling for Terminate.............................................Done.

Terminate operation successfully executed for the storage group swingdata_sg

[root@dsib0251 ~]# symsnapvx -sg swingredo_sg terminate -snapshot_name gctsnap_swingredo_sg  -generation 1

Terminate operation execution is in progress for the storage group swingredo_sg. Please wait...

    Polling for Terminate.............................................Started.
    Polling for Terminate.............................................Done.

Terminate operation successfully executed for the storage group swingredo_sg

Lastly we still need to delete the link devices if we are not planning to use them again.

[root@dsib0251 ~]# symdev delete -devs 00142:00148

Delete devices operation succeeded.

Acknowledgements.

Sincere thanks to Yaron Dar at Dell for his generous help and allowing me time on his PowerMax lab system.