FYI – I used Oracle 12.2.0.1 Enterprise DB on Oracle Linux 6.9 and installed the 12c rpm:
[root@rac1 Desktop]# rpm -qa | grep -i dbms
oracle-rdbms-server-12cR1-preinstall-1.0-16.el6.x86_64
[root@rac1 Desktop]# cat /etc/oracle-release
Oracle Linux Server release 6.9
[root@rac1 Desktop]# uname -a
Linux rac1.localdomain 3.8.13-118.18.4.el6uek.x86_64 #2 SMP Tue May 30 15:01:52 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Clusterware Configuration Files and Locations
Two text files are used to bootstrap Oracle Clusterware they are: <hostname>.olr and the GPnP profile. GPnP profile records a lot of important information about the cluster, such as the network profile and the VD.
Additionally information in the GPnP profile is used when adding a node to a cluster. This file default location is $GRID_HOME/gpnp/<hostname>/profiles/peer/profile.xml.
/etc/oracle directory contains two location files olr.loc and ocr.loc (among others). The file olr.loc shows the local location of the configuration file also used during the boot strap process of Oracle Clusterware. And ocr.loc shows the location usually pointing at an ASM based disk group / path of ocr.loc with the detailed configuration of the cluster.
[oracle@rac1 oracle]$ ls
lastgasp maps ocr.loc ocr.loc.orig olr.loc olr.loc.orig oprocd scls_scr setasmgid
[oracle@rac1 oracle]$ cat olr.loc
olrconfig_loc=/u01/app/12.2.0.1/grid/cdata/rac1.olr
crs_home=/u01/app/12.2.0.1/grid
[oracle@rac1 oracle]$ cat ocr.log
cat: ocr.log: No such file or directory
[oracle@rac1 oracle]$ cat ocr.loc
#Device/file +DATA getting replaced by device +DATA/rac-cluster/OCRFILE/registry.255.947347231
ocrconfig_loc=+DATA/rac-cluster/OCRFILE/registry.255.947347231
local_only=false[oracle@rac1 oracle]$ pwd
/etc/oracle
How Oracle Clusterware Automatically Bootstraps and Shutdown on OS start and stop
The initialization process of Oracle Clusterware starts with the following configuration file:
[oracle@rac1 oprocd]$ cat /etc/init/oracle-ohasd.conf
# Copyright (c) 2001, 2011, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup
start on runlevel [35]
stop on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
[oracle@rac1 oprocd]$ ls -al /etc/init/oracle-ohasd.conf
-rw-r–r– 1 root root 220 Jun 22 15:58 /etc/init/oracle-ohasd.conf
The initial ohasd daemon process is invoked on RHEL 6.x via the rc.d system:
[root@rac1 init.d]# pwd
/etc/rc.d/init.d
[root@rac1 init.d]# ls -al oh*
-rwxr-xr-x 1 root root 7394 Jun 22 15:57 ohasd
[root@rac1 init.d]# head ohasd
#!/bin/sh
#
# Copyright (c) 2001, 2016, Oracle and/or its affiliates. All rights reserved.
#
# ohasd.sbs – Control script for the Oracle HA Services daemon
# This script is invoked by the rc system.
#
# Note:
# For security reason, all cli tools shipped with Clusterware should be
# executed as HAS_USER in init.ohasd and ohasd rc script for SIHA. (See bug
High Availability Cluster Service stack (OHAS) is a set of background processes.
Once OHAS has started another group of processes are then spawned collectively referred to as CRS Cluster Ready Services.
Both process groups OHAS and CRS contain a large # of processes beyond the scope of the blog – to support all phases of HA and cluster management and configuration for support of Oracle RAC 12cR2.
Clusterware Manual Shutdown and Startup
As mentioned, both groups of processes are automatically start and stop during startup and shutdown of the OS.
They can be manually stopped and started as follows:
run as root:
crsctl stop cluster -all
crsctl start cluster -all
Cluster shutdown
[root@rac1 Desktop]# /u01/app/12.2.0.1/grid/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘rac2’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server ‘rac1’
CRS-2673: Attempting to stop ‘ora.cdbrac1.db’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.chad’ on ‘rac1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server ‘rac2’
CRS-2673: Attempting to stop ‘ora.qosmserver’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.cdbrac1.db’ on ‘rac2’
CRS-2677: Stop of ‘ora.qosmserver’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.cdbrac1.db’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘rac1’
CRS-2677: Stop of ‘ora.cdbrac1.db’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘rac2’
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac1’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.rac1.vip’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘rac1’
CRS-2677: Stop of ‘ora.cvu’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘rac2’
CRS-2677: Stop of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘rac1’
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.rac1.vip’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.chad’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.chad’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.ons’ on ‘rac1’
CRS-2677: Stop of ‘ora.ons’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘rac1’
CRS-2677: Stop of ‘ora.net1.network’ on ‘rac1’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘rac1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.storage’ on ‘rac1’
CRS-2677: Stop of ‘ora.storage’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac1’
CRS-2677: Stop of ‘ora.ctssd’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.chad’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.mgmtdb’ on ‘rac2’
CRS-2677: Stop of ‘ora.mgmtdb’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.MGMTLSNR’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘rac2’
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac2’
CRS-2677: Stop of ‘ora.asm’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘rac2’
CRS-2677: Stop of ‘ora.MGMTLSNR’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.rac2.vip’ on ‘rac2’
CRS-2677: Stop of ‘ora.rac2.vip’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘rac2’
CRS-2677: Stop of ‘ora.ons’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘rac2’
CRS-2677: Stop of ‘ora.net1.network’ on ‘rac2’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘rac2’ has completed
CRS-2677: Stop of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘rac1’
CRS-2677: Stop of ‘ora.crsd’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘rac2’
CRS-2673: Attempting to stop ‘ora.storage’ on ‘rac2’
CRS-2677: Stop of ‘ora.storage’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘rac1’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘rac2’
CRS-2677: Stop of ‘ora.ctssd’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.cssd’ on ‘rac1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘rac2’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘rac2’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘rac2’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘rac2’
CRS-2677: Stop of ‘ora.cssd’ on ‘rac2’ succeeded
[root@rac1 Desktop]#
And cluster startup
[root@rac1 Desktop]# /u01/app/12.2.0.1/grid/bin/crsctl start cluster -all
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘rac1’
CRS-2672: Attempting to start ‘ora.evmd’ on ‘rac1’
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘rac2’
CRS-2672: Attempting to start ‘ora.evmd’ on ‘rac2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘rac1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘rac1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘rac1’
CRS-2676: Start of ‘ora.diskmon’ on ‘rac1’ succeeded
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘rac2’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘rac2’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘rac2’
CRS-2676: Start of ‘ora.diskmon’ on ‘rac2’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘rac1’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘rac2’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘rac1’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘rac1’
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘rac1’
CRS-2676: Start of ‘ora.cssd’ on ‘rac2’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘rac2’
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘rac2’
CRS-2676: Start of ‘ora.ctssd’ on ‘rac1’ succeeded
CRS-2676: Start of ‘ora.ctssd’ on ‘rac2’ succeeded
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘rac1’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘rac1’
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘rac2’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘rac2’
CRS-2676: Start of ‘ora.asm’ on ‘rac2’ succeeded
CRS-2672: Attempting to start ‘ora.storage’ on ‘rac2’
CRS-2676: Start of ‘ora.asm’ on ‘rac1’ succeeded
CRS-2672: Attempting to start ‘ora.storage’ on ‘rac1’
CRS-2676: Start of ‘ora.storage’ on ‘rac1’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac1’
CRS-2676: Start of ‘ora.crsd’ on ‘rac1’ succeeded
CRS-2676: Start of ‘ora.storage’ on ‘rac2’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2’
CRS-2676: Start of ‘ora.crsd’ on ‘rac2’ succeeded
[root@rac1 Desktop]#
[oracle@rac2 agent13c]$ srvctl status database -db cdbrac1
Instance cdbrac11 is running on node rac1
Instance cdbrac12 is running on node rac2
exit
logout
Sourcing the environment to GRID_HOME / ORACLE_SID=+ASM1
su – oracle
. ./grid_env
# here is what grid_env source script looks like
[oracle@rac1 ~]$ cat grid_env
export ORACLE_SID=+ASM1
export ORACLE_HOME=$GRID_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
[oracle@rac1 ~]$
Yes, in this case / installation the oracle unix user owns both the grid environment and the oracle RAC DB environment
I always login to the VM as root and su – oracle and source either the grid or the db env depending on what I want to do.
So now that we have sourced the Oracle user with the grid environment, show the status of CRS.
Show CRS Status
[oracle@rac1 ~]$ crsctl status resource -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.DATA.dg
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.proxy_advm
OFFLINE OFFLINE rac1 STABLE
OFFLINE OFFLINE rac2 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE rac1 169.254.253.71 192.1
68.11.150,STABLE
ora.asm
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.cdbrac1.db
1 ONLINE ONLINE rac1 Open,HOME=/u01/app/o
racle/product/12.2.0
.1/db_1,STABLE
2 ONLINE ONLINE rac2 Open,HOME=/u01/app/o
racle/product/12.2.0
.1/db_1,STABLE
ora.cvu
1 ONLINE ONLINE rac1 STABLE
ora.mgmtdb
1 ONLINE ONLINE rac1 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE rac1 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan2.vip
1 ONLINE ONLINE rac1 STABLE
ora.scan3.vip
1 ONLINE ONLINE rac1 STABLE
——————————————————————————–
[oracle@rac1 ~]$ exit
logout
Sourcing the environment database environment ORACLE_HOME / database / ORACLE_SID=cdbrac1
[root@rac1 Desktop]# su – oracle
[oracle@rac1 ~]$ db_env
[oracle@rac1 ~]$ alias
alias db_env=’. /home/oracle/db_env’
alias grid_env=’. /home/oracle/grid_env’
…[oracle@rac1 ~]$
Here is what my “db_env” environment script looks like
[oracle@rac1 ~]$ cat db_env
export ORACLE_SID=cdbrac11
export ORACLE_HOME=$DB_HOME
export PATH=$ORACLE_HOME/bin:$BASE_PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libexport LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/libexport CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib[oracle@rac1 ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Tue Aug 1 23:28:52 2017
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production
SQL> select name, open_mode,log_mode, database_role from v$database;
NAME OPEN_MODE LOG_MODE DATABASE_ROLE——— ——————– ———— —————-CDBRAC1 READ WRITE NOARCHIVELOG PRIMARY
To be a bit more informative – here is my /etc/host file on rac1 it’s the same on rac2 and I’m using dnsmasq for the 3 SCAN IP addresses
[root@rac1 Desktop]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.150 rac1.localdomain rac1
192.168.1.151 rac2.localdomain rac2
# vip – no nic
192.168.1.152 rac1-vip.localdomain rac1-vip
192.168.1.153 rac2-vip.localdomain rac2-vip
# private Interconnect
192.168.11.150 rac1-priv.localdomain rac1-priv
192.168.11.151 rac2-priv.localdomain rac2-priv
#
192.168.1.154 scan-ip.localdomain scan-ip
192.168.1.155 scan-ip.localdomain scan-ip
192.168.1.156 scan-ip.localdomain scan-ip
192.168.1.37 sampson.localdomain sampson
[root@rac1 Desktop]# rpm -qa | grep -i dnsmasq
dnsmasq-2.48-17.el6.x86_64
Here are the grid logs
[root@rac1 Desktop]# su – oracle
[oracle@rac1 ~]$ grid_env
[oracle@rac1 ~]$ cd $GRID_HOME
[oracle@rac1 grid]$ cd log
[oracle@rac1 log]$ find . -name “*.log” -exec ls -al {} \;
-rw-r–r– 1 oracle oinstall 644 Jun 23 06:19 ./rac1/client/gpnp_12819.log
-rw-r–r– 1 oracle oinstall 393 Jun 23 06:19 ./rac1/client/gpnp_9869.log
#… and many more in between
-rw-r–r– 1 oracle oinstall 393 Jun 22 17:53 ./rac1/client/gpnp_5196.log
-rw-r–r– 1 oracle oinstall 644 Jul 30 02:42 ./rac1/client/gpnp_2944.log
-rw-r–r– 1 oracle oinstall 644 Jun 22 17:53 ./rac1/client/gpnp_6031.log
-rw-rw-r– 1 oracle oinstall 0 Jun 22 15:56 ./rac1/alertrac1.log
-rw-r–r– 1 oracle oinstall 6290 Jun 22 18:28 ./diag/asmcmd/user_oracle/rac1.localdomain/alert/alert.log
[oracle@rac1 log]$
Reference:
https://docs.oracle.com/database/121/RACAD/E48838-11.pdf
The book Expert Oracle RAC 12c.
https://oracle-base.com/articles/12c/oracle-db-12cr1-rac-installation-on-oracle-linux-6-using-virtualbox