Time Required: 30 minutes

Class Materials:

• Oracle 11gR2 Grid Infrastructure software

Next we are going to perform some final steps before we can launch the Oracle Grid Infrastructure install.

The Grid Infrastructure will provide the cluster software that allows the RAC nodes to communicate, as well as the ASM software to manage the shared disks.

To begin, download the zip files from the Oracle software download website and unzip on Ruggero. Make sure you are logged in as the oracle user so that oracle owns the unzipped files.

There are two zip files for 12cR1 Linux 64-bit called linuxamd64_12c_grid_1of2.zip and linuxamd64_12c_grid_2of2.zip.

Unzip these files into a common folder. You will be left with a directory called grid.

First we are going to set up secure password sharing between Ruggero and Magda. This will allow the oracle account on one machine to access resources on the other machine, without being prompted for a password.

Change into the grid/sshsetup directory and we will find a script called sshUserSetup.sh

We are going to use the following command to set up password sharing:

./sshUserSetup.sh \
-user oracle -hosts "ruggero magda" \
-noPromptPassphrase -confirm -advanced

 
This command will not only allow the oracle account on Ruggero to access Magda, but also allow the oracle account on Magda to access Ruggero.

You will be prompted for the oracle password four times during execution. I have included the full output so you can see the progress.

Launch the script as follows:

[oracle@ruggero sshsetup]$ ./sshUserSetup.sh \
> -user oracle -hosts "ruggero magda" \
> -noPromptPassphrase -confirm -advanced
The output of this script is also logged into /tmp/sshUserSetup_2013-07-08-22-45-18.log
Hosts are ruggero magda
user is oracle
Platform:- Linux 
Checking if the remote hosts are reachable
PING ruggero (10.10.1.110) 56(84) bytes of data.
64 bytes from ruggero (10.10.1.110): icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from ruggero (10.10.1.110): icmp_seq=2 ttl=64 time=0.041 ms
64 bytes from ruggero (10.10.1.110): icmp_seq=3 ttl=64 time=0.041 ms
64 bytes from ruggero (10.10.1.110): icmp_seq=4 ttl=64 time=0.044 ms
64 bytes from ruggero (10.10.1.110): icmp_seq=5 ttl=64 time=0.044 ms

--- ruggero ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.032/0.040/0.044/0.007 ms
PING magda (10.10.1.120) 56(84) bytes of data.
64 bytes from magda (10.10.1.120): icmp_seq=1 ttl=64 time=0.486 ms
64 bytes from magda (10.10.1.120): icmp_seq=2 ttl=64 time=0.590 ms
64 bytes from magda (10.10.1.120): icmp_seq=3 ttl=64 time=0.449 ms
64 bytes from magda (10.10.1.120): icmp_seq=4 ttl=64 time=0.435 ms
64 bytes from magda (10.10.1.120): icmp_seq=5 ttl=64 time=0.486 ms

--- magda ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 0.435/0.489/0.590/0.056 ms
Remote host reachability check succeeded.
The following hosts are reachable: ruggero magda.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost ruggero
numhosts 2
The script will setup SSH connectivity from the host ruggero.hadesnet to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host ruggero.hadesnet
and the remote hosts without being prompted for passwords or confirmations.

NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.

NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
directories.

Do you want to continue and let the script make the above mentioned changes (yes/no)?
Confirmation provided on the command line

The user chose yes
User chose to skip passphrase related questions.
Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /home/oracle/.ssh/config, it would be backed up to /home/oracle/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host with empty passphrase
Generating public/private rsa key pair.
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
9b:18:cb:fc:b4:cd:98:9b:e8:18:6e:e4:f8:53:fa:9b oracle@ruggero.hadesnet
The key's randomart image is:
+--[ RSA 1024]----+
|                 |
|                 |
|                 |
|                 |
|      . S        |
|    .o.+ o       |
|   +.o= +        |
|  ..=o = B       |
|   o++E.B.o      |
+-----------------+
Creating .ssh directory and setting permissions on remote host ruggero
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host ruggero. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host ruggero.
Warning: Permanently added 'ruggero,10.10.1.110' (RSA) to the list of known hosts.
oracle@ruggero's password: 
Done with creating .ssh directory and setting permissions on remote host ruggero.
Creating .ssh directory and setting permissions on remote host magda
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR oracle. THIS IS AN SSH REQUIREMENT.
The script would create ~oracle/.ssh/config file on remote host magda. If a config file exists already at ~oracle/.ssh/config, it would be backed up to ~oracle/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host magda.
Warning: Permanently added 'magda,10.10.1.120' (RSA) to the list of known hosts.
oracle@magda's password: 
Done with creating .ssh directory and setting permissions on remote host magda.
Copying local host public key to the remote host ruggero
The user may be prompted for a password or passphrase here since the script would be using SCP for host ruggero.
oracle@ruggero's password: 
Done copying local host public key to the remote host ruggero
Copying local host public key to the remote host magda
The user may be prompted for a password or passphrase here since the script would be using SCP for host magda.
oracle@magda's password: 
Done copying local host public key to the remote host magda
Creating keys on remote host ruggero if they do not exist already. This is required to setup SSH on host ruggero.

Creating keys on remote host magda if they do not exist already. This is required to setup SSH on host magda.
Generating public/private rsa key pair.
Your identification has been saved in .ssh/id_rsa.
Your public key has been saved in .ssh/id_rsa.pub.
The key fingerprint is:
92:73:f2:da:38:c5:e6:41:c2:c2:f3:eb:68:5a:90:fa oracle@magda.hadesnet
The key's randomart image is:
+--[ RSA 1024]----+
|                 |
|                 |
|   . .           |
|   .+ o..        |
|  o  +=+S        |
| . .  .*=        |
|.   .  =..       |
| . ...o+.        |
|  Eo..+..        |
+-----------------+
Updating authorized_keys file on remote host ruggero
Updating known_hosts file on remote host ruggero
Updating authorized_keys file on remote host magda
Updating known_hosts file on remote host magda
cat: /home/oracle/.ssh/known_hosts.tmp: No such file or directory
cat: /home/oracle/.ssh/authorized_keys.tmp: No such file or directory
SSH setup is complete.

------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user oracle.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. ~oracle or ~oracle/.ssh on the remote host may not be owned by oracle.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the /sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--ruggero:--
Running /usr/bin/ssh -x -l oracle ruggero date to verify SSH connectivity has been setup from local host to ruggero.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Mon Jul  8 22:45:52 PDT 2013
------------------------------------------------------------------------
--magda:--
Running /usr/bin/ssh -x -l oracle magda date to verify SSH connectivity has been setup from local host to magda.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Mon Jul  8 22:45:53 PDT 2013
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from ruggero to ruggero
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Mon Jul  8 22:45:53 PDT 2013
------------------------------------------------------------------------
------------------------------------------------------------------------
Verifying SSH connectivity has been setup from ruggero to magda
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.
Mon Jul  8 22:45:53 PDT 2013
------------------------------------------------------------------------
-Verification from complete-
SSH verification complete.

 
Next we are going to load a missing RPM. The cvuqdisk-1.0.9-1.rpm is located in the grid install directory under the RPM sub-directory.

Log in as root and load the missing RPM.

[root@ruggero ~]# cd /media/ORASOL/linux_x86/grid/rpm
[root@ruggero rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
Using default group oinstall to install package
   1:cvuqdisk               ########################################### [100%]
[root@ruggero rpm]#

 
Remember this RPM must be loaded on the Magda as well.

Next we are going to fix a problem with runcluvfy.sh on Red Hat and Centos 6.2 and above systems. The runcluvfy.sh script is used to ensure your cluster is ready for the grid install, but if you run this on a Red Hat or CentOS system on 6.2 or greater, you get the following error:

ERROR: 
Reference data is not available for verifying prerequisites on this operating system distribution
Verification cannot proceed

 
This is due to bug 15973656 which states that runcluvfy.sh cannot properly detect the operating system. MOS provides a zip file attached to the bug report called clupack.zip. This zip file contains several files to patch the tool, but for Oracle 12cR1 we only need one, a dummy RPM file called redhat-release-6Server-1.noarch.rpm.

If you want to use runcluvfy.sh, and I strongly recommend that you do, you will need to access MOS for bug 15973656 and download the attached zip file.

Then load the RPM:

[root@ruggero clufix]# rpm -ivh redhat-release-6Server-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:redhat-release 

 
We can now run the runcluvfy.sh tool to test if our cluster is ready for the install.

Log into Ruggero as the oracle user and navigate back to the grid install directory you created when you unzipped the 12c grid files.

In the grid directory you should see a file called runcluvfy.sh. The command we are going to use is as follows.

./runcluvfy.sh stage -pre crsinst -n ruggero,magda

The above command will check the cluster against nodes ruggero and magda. If you followed all the steps carefully, you should get the following output:

[oracle@ruggero grid]$ ./runcluvfy.sh stage -pre crsinst -n ruggero,magda

Performing pre-checks for cluster services setup 

Checking node reachability...
Node reachability check passed from node "ruggero"


Checking user equivalence...
User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...

Verification of the hosts config file successful

Node connectivity passed for subnet "192.168.0.0" with node(s) ruggero,magda
TCP connectivity check passed for subnet "192.168.0.0"

Node connectivity passed for subnet "10.10.1.0" with node(s) ruggero,magda
TCP connectivity check passed for subnet "10.10.1.0"

Node connectivity passed for subnet "10.10.2.0" with node(s) ruggero,magda
TCP connectivity check passed for subnet "10.10.2.0"


Interfaces found on subnet "192.168.0.0" that are likely candidates for VIP are:
ruggero eth0:192.168.0.4
magda eth0:192.168.0.3

Interfaces found on subnet "10.10.1.0" that are likely candidates for a private interconnect are:
ruggero eth1:10.10.1.110
magda eth5:10.10.1.120

Interfaces found on subnet "10.10.2.0" that are likely candidates for a private interconnect are:
ruggero eth2:10.10.2.110
magda eth4:10.10.2.120

WARNING: 
Could not find a suitable set of interfaces with the same name for the private interconnect
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.0.0".
Subnet mask consistency check passed for subnet "10.10.1.0".
Subnet mask consistency check passed for subnet "10.10.2.0".
Subnet mask consistency check passed.

Node connectivity check passed

Checking multicast communication...

Checking subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251"...
Check of subnet "192.168.0.0" for multicast communication with multicast group "224.0.0.251" passed.

Check of multicast communication passed.

Checking ASMLib configuration.
Check for ASMLib configuration passed.
Total memory check passed
Available memory check passed
Swap space check passed
Free disk space check passed for "ruggero:/usr,ruggero:/var,ruggero:/etc,ruggero:/sbin,ruggero:/tmp"
Free disk space check passed for "magda:/usr,magda:/var,magda:/etc,magda:/sbin,magda:/tmp"
Check for multiple users with UID value 501 passed 
User existence check passed for "oracle"
Group existence check passed for "oinstall"
Group existence check passed for "dba"
Membership check for user "oracle" in group "oinstall" [as Primary] passed
Membership check for user "oracle" in group "dba" passed
Run level check passed
Hard limits check passed for "maximum open file descriptors"
Soft limits check passed for "maximum open file descriptors"
Hard limits check passed for "maximum user processes"
Soft limits check passed for "maximum user processes"
System architecture check passed
Kernel version check passed
Kernel parameter check passed for "semmsl"
Kernel parameter check passed for "semmns"
Kernel parameter check passed for "semopm"
Kernel parameter check passed for "semmni"
Kernel parameter check passed for "shmmax"
Kernel parameter check passed for "shmmni"
Kernel parameter check passed for "shmall"
Kernel parameter check passed for "file-max"
Kernel parameter check passed for "ip_local_port_range"
Kernel parameter check passed for "rmem_default"
Kernel parameter check passed for "rmem_max"
Kernel parameter check passed for "wmem_default"
Kernel parameter check passed for "wmem_max"
Kernel parameter check passed for "aio-max-nr"
Package existence check passed for "binutils"
Package existence check passed for "compat-libcap1"
Package existence check passed for "compat-libstdc++-33(x86_64)"
Package existence check passed for "libgcc(x86_64)"
Package existence check passed for "libstdc++(x86_64)"
Package existence check passed for "libstdc++-devel(x86_64)"
Package existence check passed for "sysstat"
Package existence check passed for "gcc"
Package existence check passed for "gcc-c++"
Package existence check passed for "ksh"
Package existence check passed for "make"
Package existence check passed for "glibc(x86_64)"
Package existence check passed for "glibc-devel(x86_64)"
Package existence check passed for "libaio(x86_64)"
Package existence check passed for "libaio-devel(x86_64)"
Package existence check passed for "nfs-utils"

Checking availability of ports "6200,6100" required for component "Oracle Notification Service (ONS)"
Port availability check passed for ports "6200,6100"
Check for multiple users with UID value 0 passed 
Current group ID check passed

Starting check for consistency of primary group of root user

Check for consistency of root user's primary group passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
NTP Configuration file check passed

Checking daemon liveness...
Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes

NTP common Time Server Check started...
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Clock time offset check passed

Clock synchronization check using Network Time Protocol(NTP) passed

Core file name pattern consistency check passed.

User "oracle" is not part of "root" group. Check passed
Default user file creation mask check passed
Checking integrity of file "/etc/resolv.conf" across nodes

"domain" and "search" entries do not coexist in any  "/etc/resolv.conf" file
All nodes have same "search" order defined in file "/etc/resolv.conf"
The DNS response time for an unreachable node is within acceptable limit on all nodes

Check for integrity of file "/etc/resolv.conf" passed

Time zone consistency check passed

Checking integrity of name service switch configuration file "/etc/nsswitch.conf" ...
All nodes have same "hosts" entry defined in file "/etc/nsswitch.conf"
Check for integrity of name service switch configuration file "/etc/nsswitch.conf" passed


Checking daemon "avahi-daemon" is not configured and running
Daemon not configured check passed for process "avahi-daemon"
Daemon not running check passed for process "avahi-daemon"

Starting check for /dev/shm mounted as temporary file system ...

Check for /dev/shm mounted as temporary file system passed

Pre-check for cluster services setup was successful. 

 
If you do not get a successful message then check your output carefully. You can also add the -verbose switch to get extra information.

Note that many problems reported by runcluvfy.sh can be safely ignored.

We are now ready to launch the grid installer!