Install Oracle Grid Infrastructure 12.1.0.1++ on Fedora 17 – 20


In my previous post i’vh described how to set up the Oracle 12c Database Software to get working on Fedora 17 up to 20. Today i’m going to to show how to setup a RAC Cluster (12.1.0.1) on Fedora 17++ (I’vh testet on Fedora 20). My enviroment: 2x VirtualBox, Fedora 20 x86_64, with 4 CPUs, 4092 MB RAM, 50GB local, 10GB shared disks, 2x NIC Paravirtualized

1. Prerequirements

First start follow the Oracle Database 12c Quick Installation Guide to setup the base for the Grid Infrastructure (see here)
During the setup of the inital stuff you should mention something like that:

Loaded plugins: langpacks, refresh-packagekit
No package libcap1 available.
Error: Nothing to do

This problem doesn’t matter, because there is a newer working version:

[root@fed01 ~]# yum list libcap
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
libcap.x86_64                                             2.22-7.fc20                                             @anaconda
Available Packages
libcap.i686                                               2.22-7.fc20                                             fedora

To avoid an exception like this:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/oracle/12.1.0.1/grid/crs/install/crsconfig_params
/opt/oracle/12.1.0.1/grid/bin/crsctl query crs releaseversion ... failed rc=127 with message:
 /opt/oracle/12.1.0.1/grid/bin/crsctl.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory 

Failed to create keys in the OLR, rc = 127, Message:
  /opt/oracle/12.1.0.1/grid/bin/clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory 

2014/02/28 00:13:19 CLSRSC-188: Failed to create keys in Oracle Local Registry

Died at /opt/oracle/12.1.0.1/grid/crs/install/crsutils.pm line 6479.
The command '/opt/oracle/12.1.0.1/grid/perl/bin/perl -I/opt/oracle/12.1.0.1/grid/perl/lib -I/opt/oracle/12.1.0.1/grid/crs/install /opt/oracle/12.1.0.1/grid/crs/install/rootcrs.pl ' execution failed

… you have to relink the existing library …

[root@fed01 install]# cd /usr/lib64/
[root@fed01 lib64]# ln -s /usr/lib64/libcap.so.2 /usr/lib64/libcap.so.1

Futher i’vh disabled the firewall to avoid network problems during my tests

systemctl disable firewalld.service

2. Unpack and install the software

There will be nothing special. During linking process you will get the following error:

fedora20_12c_rac

To workaround this errors please follow my previous post. (see here)

You should be able to complete the installation up to the root.sh successfully.

3. root.sh

If you are going to execute the root.sh right now, you will get the following problem:

[root@fed01 lib64]# /opt/oracle/12.1.0.1/grid/root.sh 
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/oracle/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:  
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/oracle/12.1.0.1/grid/crs/install/crsconfig_params
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2014/02/28 00:16:34 CLSRSC-330: Adding Clusterware entries to file '/etc/inittab'

2014/02/28 00:18:40 CLSRSC-214: Failed to start 'ohasd'

Failed to start the Clusterware. Last 20 lines of the alert log follow: 
2014-02-28 00:13:19.635: 
[client(8796)]CRS-2101:The OLR was formatted using version 4.
2014-02-28 00:15:11.647: 
[client(9189)]CRS-2101:The OLR was formatted using version 4.

CRS-4133: Oracle High Availability Services has been stopped.
2014/02/28 00:31:53 CLSRSC-117: Failed to start Oracle Clusterware stack

Died at /opt/oracle/12.1.0.1/grid/crs/install/crsutils.pm line 8813.
The command '/opt/oracle/12.1.0.1/grid/perl/bin/perl -I/opt/oracle/12.1.0.1/grid/perl/lib -I/opt/oracle/12.1.0.1/grid/crs/install /opt/oracle/12.1.0.1/grid/crs/install/rootcrs.pl ' execution failed

Oracle’s Grid Infrastructure starts with the intitialization script:

[root@fed01 lib64]# cat /etc/inittab
# inittab is no longer used when using systemd.
#
# ADDING CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM.
#
# Ctrl-Alt-Delete is handled by /etc/systemd/system/ctrl-alt-del.target
#
# systemd uses 'targets' instead of runlevels. By default, there are two main targets:
#
# multi-user.target: analogous to runlevel 3
# graphical.target: analogous to runlevel 5
#
# To set a default target, run:
#
# ln -sf /lib/systemd/system/.target /etc/systemd/system/default.target
#
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1

Fedora doesn’t use the old style and switched to systemd, therefore you have to setup the daemon with the new enviroment. Here my implementation

[root@fed01 lib64]# cat /etc/systemd/system/ohasd.service 
[Unit]
Description=ohasd daemon

[Service]
ExecStart=/etc/init.d/init.ohasd run >/dev/null 2>&1

Reload the dameon unit to inform systemd about my changes

systemctl daemon-reload

Now enable and start the daemon

systemctl enable ohasd.service
systemctl start ohasd.service

Check how it’s work?

[root@fed01 ~]# ps -ef | grep oha                                                                                                                     
root     13069     1  0 00:56 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1

Now re execute the root.sh script and it should finalize successfully:

...
CRS-6016: Resource auto-start has completed for server fed01
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.

Preparing packages...
cvuqdisk-1.0.9-1.x86_64
2014/02/28 01:13:27 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Here the cluster (first node) output:

[root@fed01 system]# crsctl stat res -t 
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       fed01                    STABLE
ora.DATA.dg
               ONLINE  ONLINE       fed01                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       fed01                    STABLE
ora.net1.network
               ONLINE  ONLINE       fed01                    STABLE
ora.ons
               ONLINE  ONLINE       fed01                    STABLE
ora.proxy_advm
               ONLINE  OFFLINE      fed01                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       fed01                    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       fed01                    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       fed01                    STABLE
ora.MGMTLSNR
      1        OFFLINE OFFLINE                               STABLE
ora.asm
      1        ONLINE  ONLINE       fed01                    STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       fed01                    STABLE
ora.fed01.vip
      1        ONLINE  ONLINE       fed01                    STABLE
ora.oc4j
      1        ONLINE  ONLINE       fed01                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       fed01                    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       fed01                    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       fed01                    STABLE
--------------------------------------------------------------------------------

4. Redo on all other nodes and complete setup

To complete your cluster setup redo all step above.

For completion the output from the second node (root.sh):

[root@fed02 lib64]# /opt/oracle/12.1.0.1/grid/root.sh 
Performing root user operation for Oracle 12c 

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /opt/oracle/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /opt/oracle/12.1.0.1/grid/crs/install/crsconfig_params

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'fed02'                                                                                                                                                                                                                                
CRS-2673: Attempting to stop 'ora.crsd' on 'fed02'                                                                                                                                                                                                                                                                           
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'fed02'                                                                                                                                                                                                                                           
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'fed02'                                                                                                                                                                                                                                                                        
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'fed02'                                                                                                                                                                                                                                                            
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'fed02'                                                                                                                                                                                                                                                            
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'fed02'                                                                                                                                                                                                                                                            
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'fed02' succeeded                                                                                                                                                                                                                                                             
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'fed02'                                                                                                                                                                                                                                                                      
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'fed02' succeeded                                                                                                                                                                                                                                                             
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'fed02'                                                                                                                                                                                                                                                                      
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'fed02' succeeded                                                                                                                                                                                                                                                             
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'fed02'                                                                                                                                                                                                                                                                      
CRS-2677: Stop of 'ora.scan1.vip' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'fed01'
CRS-2677: Stop of 'ora.scan3.vip' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'fed01'
CRS-2677: Stop of 'ora.scan2.vip' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'fed01'
CRS-2676: Start of 'ora.scan1.vip' on 'fed01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'fed01'
CRS-2676: Start of 'ora.scan3.vip' on 'fed01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'fed01'
CRS-2676: Start of 'ora.scan2.vip' on 'fed01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'fed01'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'fed01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'fed01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'fed01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'fed02'
CRS-2677: Stop of 'ora.asm' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.ASMNET1LSNR_ASM.lsnr' on 'fed02'
CRS-2677: Stop of 'ora.ASMNET1LSNR_ASM.lsnr' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'fed02'
CRS-2677: Stop of 'ora.ons' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'fed02'
CRS-2677: Stop of 'ora.net1.network' on 'fed02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'fed02' has completed
CRS-2677: Stop of 'ora.crsd' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'fed02'
CRS-2673: Attempting to stop 'ora.storage' on 'fed02'
CRS-2673: Attempting to stop 'ora.crf' on 'fed02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'fed02'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'fed02'
CRS-2677: Stop of 'ora.storage' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'fed02'
CRS-2677: Stop of 'ora.crf' on 'fed02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'fed02' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'fed02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'fed02' succeeded
CRS-2677: Stop of 'ora.asm' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'fed02'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'fed02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'fed02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'fed02'
CRS-2677: Stop of 'ora.cssd' on 'fed02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'fed02'
CRS-2677: Stop of 'ora.gipcd' on 'fed02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'fed02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'fed02'
CRS-2672: Attempting to start 'ora.evmd' on 'fed02'
CRS-2676: Start of 'ora.evmd' on 'fed02' succeeded
CRS-2676: Start of 'ora.mdnsd' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'fed02'
CRS-2676: Start of 'ora.gpnpd' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'fed02'
CRS-2676: Start of 'ora.gipcd' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'fed02'
CRS-2676: Start of 'ora.cssdmonitor' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'fed02'
CRS-2672: Attempting to start 'ora.diskmon' on 'fed02'
CRS-2676: Start of 'ora.diskmon' on 'fed02' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'fed02'
CRS-2676: Start of 'ora.cssd' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'fed02'
CRS-2672: Attempting to start 'ora.ctssd' on 'fed02'
CRS-2676: Start of 'ora.ctssd' on 'fed02' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'fed02'
CRS-2676: Start of 'ora.asm' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'fed02'
CRS-2676: Start of 'ora.storage' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'fed02'
CRS-2676: Start of 'ora.crf' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'fed02'
CRS-2676: Start of 'ora.crsd' on 'fed02' succeeded
CRS-6017: Processing resource auto-start for servers: fed02
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'fed02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'fed01'
CRS-2672: Attempting to start 'ora.ons' on 'fed02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'fed01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'fed01'
CRS-2677: Stop of 'ora.scan1.vip' on 'fed01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'fed02'
CRS-2676: Start of 'ora.scan1.vip' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'fed02'
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'fed02'
CRS-2676: Start of 'ora.ons' on 'fed02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'fed02' succeeded
CRS-2676: Start of 'ora.asm' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'fed01'
CRS-2679: Attempting to clean 'ora.proxy_advm' on 'fed02'
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/opt/oracle/12.1.0.1/grid/log/fed01/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2681: Clean of 'ora.proxy_advm' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'fed02'
CRS-2674: Start of 'ora.proxy_advm' on 'fed01' failed
CRS-2679: Attempting to clean 'ora.proxy_advm' on 'fed01'
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/opt/oracle/12.1.0.1/grid/log/fed02/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.proxy_advm' on 'fed02' failed
CRS-2679: Attempting to clean 'ora.proxy_advm' on 'fed02'
CRS-2681: Clean of 'ora.proxy_advm' on 'fed01' succeeded
CRS-2681: Clean of 'ora.proxy_advm' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'fed02'
CRS-2672: Attempting to start 'ora.DATA.dg' on 'fed02'
CRS-2672: Attempting to start 'ora.proxy_advm' on 'fed01'
CRS-2676: Start of 'ora.DATA.dg' on 'fed02' succeeded
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/opt/oracle/12.1.0.1/grid/log/fed02/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-5017: The resource action "ora.proxy_advm start" encountered the following error: 
ORA-03113: end-of-file on communication channel
Process ID: 0
Session ID: 0 Serial number: 0
. For details refer to "(:CLSN00107:)" in "/opt/oracle/12.1.0.1/grid/log/fed01/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.proxy_advm' on 'fed02' failed
CRS-2679: Attempting to clean 'ora.proxy_advm' on 'fed02'
CRS-2674: Start of 'ora.proxy_advm' on 'fed01' failed
CRS-2679: Attempting to clean 'ora.proxy_advm' on 'fed01'
CRS-2681: Clean of 'ora.proxy_advm' on 'fed02' succeeded
CRS-2672: Attempting to start 'ora.proxy_advm' on 'fed02'
===== Summary of resource auto-start failures follows =====
CRS-2807: Resource 'ora.proxy_advm' failed to start automatically.
CRS-6016: Resource auto-start has completed for server fed02
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2014/02/28 01:53:02 CLSRSC-343: Successfully started Oracle clusterware stack

Preparing packages...
cvuqdisk-1.0.9-1.x86_64
2014/02/28 01:53:18 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

The final state:

--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       fed01                    STABLE
               ONLINE  ONLINE       fed02                    STABLE
ora.DATA.dg
               ONLINE  ONLINE       fed01                    STABLE
               ONLINE  ONLINE       fed02                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       fed01                    STABLE
               ONLINE  ONLINE       fed02                    STABLE
ora.net1.network
               ONLINE  ONLINE       fed01                    STABLE
               ONLINE  ONLINE       fed02                    STABLE
ora.ons
               ONLINE  ONLINE       fed01                    STABLE
               ONLINE  ONLINE       fed02                    STABLE
ora.proxy_advm
               ONLINE  OFFLINE      fed01                    STABLE
               ONLINE  OFFLINE      fed02                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       fed02                    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       fed01                    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       fed01                    STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE                                STABLE
ora.asm
      1        ONLINE  ONLINE       fed01                    STABLE
      2        ONLINE  ONLINE       fed02                    STABLE
ora.cvu
      1        ONLINE  ONLINE       fed01                    STABLE
ora.fed01.vip
      1        ONLINE  ONLINE       fed01                    STABLE
ora.fed02.vip
      1        ONLINE  ONLINE       fed02                    STABLE
ora.oc4j
      1        ONLINE  ONLINE       fed01                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       fed02                    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       fed01                    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       fed01                    STABLE
--------------------------------------------------------------------------------

In my next post i’m trying to enable asmlib and acfs support. I hope i’ll get it to work.

DONE & Have Fun!

Oracle RAC with OCR mirrored by ASM pitfalls


Since 11gR2 OCR and Voting files can placed into ASM. Here an overview:

ASM Reduncancy Level  OCR Mirrors  Votingdisks  Failgroups  Min. count of disks
EXTERNAL 1 1 1 1
NORMAL 2 3 3 3
HIGH 3 5 5 5

The most common configuration of RAC is a 2 node RAC. For example Oracle database applicance.
If a RAC has one storage the will be no problem and ASM mirror is not nessecary, EXTERNAL redundancy can be used. If the RAC has two storages there is a problem with the Votingdisks. Therefore a third location will be needed. In most configurations two locations are expensive enough and a third location is not available.

Here a real situation:
Customer has two nodes and two storages. All files are mirrored on ASM with normal redundancy, also OCR diskgroup:
Site A: 2x voting, secondary RAC node
Site B: 1x voting, master RAC node

Suddenly site A will break down due to a site desaster. Some seconds later the RAC node on Site B will shutdown due to OCR errors.
Why does this happens:
ASM mirroring is done at block/extent level.

  • EXTERNAL mirroring does mean no mirror
  • NORMAL = extent will be located in one other failgroup
  • HIGH = extent will be located in two other failgroups

Build up on the count of disks needed and the mirror copies witch should be used?

  1. EXTERNAL = not usable for two storages
  2. NORMAL = 3 Disks with 3 voting disks and an OCR mirror
    therefore segmentation of disks is 2:1, but only 2 mirrored blocks of OCR maybe all OCR blocks are on Site A
  3. HIGH = 5 Disks with 5 voting disks and an HIGH OCR mirror
    therefore segmentation of disks is 3:2, but only 3 mirrored blocks of OCR maybe all OCR blocks are on Site A

So what can be done. The solution: NORMAL redundancy with HIGH redundancy OCR mirror. The following construct will be created:
Disk segmentation is 2:1 and 3 mirrored block of OCR. All blocks of the OCR will be mirrored on every Disk. Whatever witch Site on the disaster will happen, at least one OCR mirror copy will be available.

Here the demonstration on 12.1.0.1 GI:
1. Create Cluster with normal redundancy cluster diskgroup DG_CLUSTER
2. Check asm template of OCR asm diskgroup:

SQL> select * from v$asm_template where group_number=1;

GROUP_NUMBER ENTRY_NUMBER REDUND STRIPE S NAME                           PRIM MIRR     CON_ID
------------ ------------ ------ ------ - ------------------------------ ---- ---- ----------
           1          123 MIRROR COARSE Y VOTINGFILE                     COLD COLD          0
           1          343 MIRROR COARSE Y OCRFILE                        COLD COLD          0

3. Check mirror on OCR:

ASMCMD> ls -l +DG_CLUSTER/vmsvr-clu2/OCRFILE
Type     Redund  Striped  Time             Sys  Name
OCRFILE  MIRROR  COARSE   JUL 23 23:00:00  Y    REGISTRY.255.821572803

4. Check ASM extent distribution

SQL>select g.name
2          ,d.path
3          ,e.XNUM_KFFXP extent
4          ,decode(e.lxn_kffxp,0,'primary',1,'mirror-normal','mirror-high') mirrormeta
5  from x$kffxp        e
6        ,v$asm_alias  a
7        ,v$asm_disk   d
8        ,v$asm_diskgroup  g
9  where e.number_kffxp=a.file_number
10       and e.disk_kffxp=d.disk_number
11       and d.group_number = g.group_number
12       and a.name='REGISTRY.255.821572803'
13*      order by 3,4 desc

NAME                         PATH                               EXTENT MIRRORMETA
---------------------------------------------------------------------- -------------
DG_CLUSTER                   ORCL:ORA_DISK_2                         0 primary
DG_CLUSTER                   ORCL:ORA_DISK_1                         0 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_3                         1 primary
DG_CLUSTER                   ORCL:ORA_DISK_1                         1 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_1                         2 primary
DG_CLUSTER                   ORCL:ORA_DISK_3                         2 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_2                         3 primary
DG_CLUSTER                   ORCL:ORA_DISK_3                         3 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_3                         4 primary
DG_CLUSTER                   ORCL:ORA_DISK_2                         4 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_1                         5 primary
DG_CLUSTER                   ORCL:ORA_DISK_2                         5 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_2                         6 primary
DG_CLUSTER                   ORCL:ORA_DISK_1                         6 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_3                         7 primary
DG_CLUSTER                   ORCL:ORA_DISK_1                         7 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_1                         8 primary
DG_CLUSTER                   ORCL:ORA_DISK_3                         8 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_2                         9 primary
DG_CLUSTER                   ORCL:ORA_DISK_3                         9 mirror-normal
DG_CLUSTER                   ORCL:ORA_DISK_3                        10 primary
DG_CLUSTER                   ORCL:ORA_DISK_2                        10 mirror-normal
...

As you can see the diskgroup is made up of 3 disks (ORA_DISK_1 – 3). Further there are only two mirrors of each extent.

5. Backup OCR

[root ~]# ocrconfig -manualbackup
vmsvredu3     2013/07/24 23:51:17     /opt/oracle/12.1/grid/cdata/vmsvr-clu2/backup_20130724_235117.ocr
vmsvredu3     2013/07/23 22:52:10     /opt/oracle/12.1/grid/cdata/vmsvr-clu2/backup_20130723_225210.ocr
vmsvredu3     2013/07/23 22:45:11     /opt/oracle/12.1/grid/cdata/vmsvr-clu2/backup_20130723_224511.ocr

6. To correct this problem if don’t have an OCR mirror, stop cluster and start one node exclusiv and without crsd

crsctl start crs -excl -nocrs

7. Change asm template

SQL> alter diskgroup dg_cluster modify template OCRFILE attributes (HIGH);

Diskgroup altered.

SQL> select * from v$asm_template where group_number=1;

GROUP_NUMBER ENTRY_NUMBER REDUND STRIPE S NAME                           PRIM MIRR     CON_ID
------------ ------------ ------ ------ - ------------------------------ ---- ---- ----------
           1          120 MIRROR COARSE Y PARAMETERFILE                  COLD COLD          0
           1          121 MIRROR COARSE Y ASMPARAMETERFILE               COLD COLD          0
           1          123 MIRROR COARSE Y VOTINGFILE                     COLD COLD          0
           1          124 MIRROR COARSE Y DUMPSET                        COLD COLD          0
           1          125 HIGH   FINE   Y CONTROLFILE                    COLD COLD          0
           1          126 MIRROR COARSE Y FLASHFILE                      COLD COLD          0
           1          127 MIRROR COARSE Y ARCHIVELOG                     COLD COLD          0
           1          128 MIRROR COARSE Y ONLINELOG                      COLD COLD          0
           1          129 MIRROR COARSE Y DATAFILE                       COLD COLD          0
           1          230 MIRROR COARSE Y TEMPFILE                       COLD COLD          0
           1          231 MIRROR COARSE Y BACKUPSET                      COLD COLD          0
           1          232 MIRROR COARSE Y XTRANSPORT BACKUPSET           COLD COLD          0
           1          233 MIRROR COARSE Y INCR XTRANSPORT BACKUPSET      COLD COLD          0
           1          234 MIRROR COARSE Y AUTOBACKUP                     COLD COLD          0
           1          235 MIRROR COARSE Y XTRANSPORT                     COLD COLD          0
           1          237 MIRROR COARSE Y CHANGETRACKING                 COLD COLD          0
           1          238 MIRROR COARSE Y FLASHBACK                      COLD COLD          0
           1          239 MIRROR COARSE Y KEY_STORE                      COLD COLD          0
           1          340 MIRROR COARSE Y AUTOLOGIN_KEY_STORE            COLD COLD          0
           1          341 MIRROR COARSE Y AUDIT_SPILLFILES               COLD COLD          0
           1          342 MIRROR COARSE Y DATAGUARDCONFIG                COLD COLD          0
           1          343 HIGH   COARSE Y OCRFILE                        COLD COLD          0

22 rows selected.

SQL>

8. Remove old OCR

ASMCMD> ls -l
Type     Redund  Striped  Time             Sys  Name
OCRFILE  MIRROR  COARSE   JUL 24 10:00:00  Y    REGISTRY.255.821572803
ASMCMD> rm -f REGISTRY.255.821572803

9. Restore OCR

[root ~]# ocrconfig -restore /opt/oracle/12.1/grid/cdata/vmsvr-clu2/backup_20130724_235117.ocr

10. Check new OCR

ASMCMD> ls -l
Type     Redund  Striped  Time             Sys  Name
OCRFILE  HIGH    COARSE   JUL 24 10:00:00  Y    REGISTRY.255.821615711
ASMCMD>

11. Check crsd starts

[oracle ~]$ crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crf' on 'vmsvredu3'
CRS-2672: Attempting to start 'ora.storage' on 'vmsvredu3'
CRS-2676: Start of 'ora.storage' on 'vmsvredu3' succeeded
CRS-2676: Start of 'ora.crf' on 'vmsvredu3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'vmsvredu3'
CRS-2676: Start of 'ora.crsd' on 'vmsvredu3' succeeded

12. Restart cluster normal

[root ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'vmsvredu3'
CRS-2673: Attempting to stop 'ora.crsd' on 'vmsvredu3'
CRS-2677: Stop of 'ora.crsd' on 'vmsvredu3' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'vmsvredu3'
CRS-2673: Attempting to stop 'ora.evmd' on 'vmsvredu3'
CRS-2673: Attempting to stop 'ora.storage' on 'vmsvredu3'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'vmsvredu3'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'vmsvredu3'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'vmsvredu3'
CRS-2677: Stop of 'ora.storage' on 'vmsvredu3' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'vmsvredu3'
CRS-2677: Stop of 'ora.drivers.acfs' on 'vmsvredu3' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'vmsvredu3' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'vmsvredu3' succeeded
CRS-2677: Stop of 'ora.evmd' on 'vmsvredu3' succeeded
CRS-2677: Stop of 'ora.asm' on 'vmsvredu3' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'vmsvredu3'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'vmsvredu3' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'vmsvredu3' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'vmsvredu3'
CRS-2677: Stop of 'ora.cssd' on 'vmsvredu3' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'vmsvredu3'
CRS-2677: Stop of 'ora.crf' on 'vmsvredu3' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'vmsvredu3'
CRS-2677: Stop of 'ora.gipcd' on 'vmsvredu3' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'vmsvredu3' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@vmsvredu3 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root ~]#

13. Check ASM mirroring again:

SQL> select g.name
  2          ,d.path
  3          ,e.XNUM_KFFXP extent
  4          ,decode(e.lxn_kffxp,0,'primary',1,'mirror-normal','mirror-high') mirrormeta
  5  from x$kffxp        e
  6        ,v$asm_alias  a
  7        ,v$asm_disk   d
  8        ,v$asm_diskgroup  g
  9  where e.number_kffxp=a.file_number
 10        and e.disk_kffxp=d.disk_number
 11        and d.group_number = g.group_number
 12        and a.name='REGISTRY.255.821615711'
 13        order by 3,4 desc
 14  ;

NAME                           PATH                               EXTENT MIRRORMETA
------------------------------ ------------------------------ ---------- -------------
DG_CLUSTER                     ORCL:ORA_DISK_1                         0 primary
DG_CLUSTER                     ORCL:ORA_DISK_3                         0 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_2                         0 mirror-high
DG_CLUSTER                     ORCL:ORA_DISK_2                         1 primary
DG_CLUSTER                     ORCL:ORA_DISK_3                         1 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_1                         1 mirror-high
DG_CLUSTER                     ORCL:ORA_DISK_3                         2 primary
DG_CLUSTER                     ORCL:ORA_DISK_1                         2 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_2                         2 mirror-high
DG_CLUSTER                     ORCL:ORA_DISK_1                         3 primary
DG_CLUSTER                     ORCL:ORA_DISK_3                         3 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_2                         3 mirror-high
DG_CLUSTER                     ORCL:ORA_DISK_2                         4 primary
DG_CLUSTER                     ORCL:ORA_DISK_3                         4 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_1                         4 mirror-high
DG_CLUSTER                     ORCL:ORA_DISK_3                         5 primary
DG_CLUSTER                     ORCL:ORA_DISK_1                         5 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_2                         5 mirror-high
DG_CLUSTER                     ORCL:ORA_DISK_1                         6 primary
DG_CLUSTER                     ORCL:ORA_DISK_3                         6 mirror-normal
DG_CLUSTER                     ORCL:ORA_DISK_2                         6 mirror-high
...

All done. Now a disaster can come.

References: