I recently added two new nodes to an existing 2-node RAC cluster, bringing the total nodes to 4.

After the installer completed, with more than a few errors along the way I might add, I had ASM running on just 3 nodes….

I’ve edited the crsctl output below to try to make it more readable.

The installer had added both of the new nodes – dsib0253 and dsib0254 – to my existing cluster, and CRS was running on all 4 nodes. However ASM was not started on dsib0254 and would not start.

[oracle@dsib0254 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       dsib0251                 STABLE
               ONLINE  ONLINE       dsib0252                 STABLE
               ONLINE  ONLINE       dsib0253                 STABLE
               ONLINE  ONLINE       dsib0254                 STABLE

--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 STABLE
      2        ONLINE  ONLINE       dsib0252                 STABLE
      3        ONLINE  ONLINE       dsib0253                 STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 STABLE
      2        ONLINE  ONLINE       dsib0252                 STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.FRA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 STABLE
      2        ONLINE  ONLINE       dsib0252                 STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.GRID.dg(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 STABLE
      2        ONLINE  ONLINE       dsib0252                 STABLE
      3        ONLINE  ONLINE       dsib0253                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       dsib0252                 STABLE
ora.REDO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 STABLE
      2        ONLINE  ONLINE       dsib0252                 STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 Started,STABLE
      2        ONLINE  ONLINE       dsib0252                 Started,STABLE
      3        ONLINE  ONLINE       dsib0253                 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       dsib0251                 STABLE
      2        ONLINE  ONLINE       dsib0252                 STABLE
      3        ONLINE  ONLINE       dsib0253                 STABLE

Trying to manually start ASM on dsib0254 yielded an error telling me the resource was already running.

[oracle@dsib0254 ~]$ srvctl start asm
PRCC-1014 : asm was already running
PRCR-1004 : Resource ora.asm is already running
PRCR-1079 : Failed to start resource ora.asm
CRS-5702: Resource 'ora.asm' is already running on 'dsib0251'

Trying to manually start ASM on dsib0254 explicitly yielded a different error.

[oracle@dsib0254 ~]$ srvctl start asm -n dsib0254
PRCR-1013 : Failed to start resource ora.asm
PRCR-1064 : Failed to start resource ora.asm on node dsib0254
CRS-2552: There are no available instances of resource 'ora.asm' to start.

Trying to manually add ASM on dsib0254 told me that the resource already existed.

[oracle@dsib0254 ~]$ srvctl add asm
PRCA-1095 : Unable to create ASM resource because it already exists.

After some digging, MOS Note 2341248.1 provided a solution:

[oracle@dsib0253 ~]$ srvctl modify asm -count ALL

I executed the command from node 3, but I think it would work from any node.