Oracle 11g RAC 添加节点故障之--CRS资源启动故障

系统环境：

操作系统：RedHat EL5.5

集群软件： GI 11G

数据库软件：Oracle 11.2.0.1

故障原因：

由于新节点（node3）是从原有的老节点（node2），克隆而来，在添加新节点时，忘了修改新节点的主机名，导致出现以下故障：

1、新节点CRS service启动正常

[root@node3 ~]# crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

2、listener 资源启动失败

[root@node3 ~]# crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.DG1.dg ora....up.type ONLINE ONLINE node1

ora....ER.lsnr ora....er.type ONLINE ONLINE node1

ora....N1.lsnr ora....er.type ONLINE ONLINE node1

ora....VOTE.dg ora....up.type ONLINE ONLINE node1

ora.RCY.dg ora....up.type ONLINE ONLINE node1

ora.asm ora.asm.type ONLINE ONLINE node1

ora.eons ora.eons.type ONLINE ONLINE node1

ora.gsd ora.gsd.type ONLINE ONLINE node1

ora....network ora....rk.type ONLINE ONLINE node1

ora....SM1.asm application ONLINE ONLINE node1

ora....E1.lsnr application ONLINE ONLINE node1

ora.node1.gsd application ONLINE ONLINE node1

ora.node1.ons application ONLINE ONLINE node1

ora.node1.vip ora....t1.type ONLINE ONLINE node1

ora....SM2.asm application ONLINE ONLINE node2

ora....E2.lsnr application ONLINE ONLINE node2

ora.node2.gsd application ONLINE ONLINE node2

ora.node2.ons application ONLINE ONLINE node2

ora.node2.vip ora....t1.type ONLINE ONLINE node2

ora....SM3.asm application ONLINE ONLINE node3

ora....E3.lsnr application OFFLINE OFFLINE

ora.node3.gsd application ONLINE ONLINE node3

ora.node3.ons application ONLINE ONLINE node3

ora.oc4j ora.oc4j.type ONLINE ONLINE node2

ora.ons ora.ons.type ONLINE ONLINE node1

ora.pmydb.db ora....se.type OFFLINE OFFLINE

ora....taf.svc ora....ce.type OFFLINE OFFLINE

ora.prod.db ora....se.type OFFLINE OFFLINE

ora....ry.acfs ora....fs.type ONLINE ONLINE node1

ora.scan1.vip ora....ip.type ONLINE ONLINE node1

[root@node3 ~]# crs_stat |grep lsn

NAME=ora.LISTENER.lsnr

NAME=ora.LISTENER_SCAN1.lsnr

NAME=ora.node1.LISTENER_NODE1.lsnr

NAME=ora.node2.LISTENER_NODE2.lsnr

NAME=ora.node3.LISTENER_NODE3.lsnr

NAME=ora.node3.LISTENER_NODE3.lsnr 服务启动失败

3、listener 资源故障解决方法：

手工启动：

[root@node3 ~]# crs_start ora.node3.LISTENER_NODE3.lsnr -f

CRS-2527: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on 'ora.cluster_vip_net1.type'

CRS-2525: All instances of the resource 'ora.node1.vip' are already running; relocate is not allowed because the force option was not specified

CRS-2525: All instances of the resource 'ora.node2.vip' are already running; relocate is not allowed because the force option was not specified

CRS-0222: Resource 'ora.node3.LISTENER_NODE3.lsnr' has dependency error.

从以上日志可以看出，listener 在启动时，缺少vip的支持（node3 vip service）

[root@node3 ~]# crs_stat |grep vip

NAME=ora.node1.vip

TYPE=ora.cluster_vip_net1.type

NAME=ora.node2.vip

TYPE=ora.cluster_vip_net1.type

NAME=ora.scan1.vip

TYPE=ora.scan_vip.type

缺少node vip service！

添加vip service：

[root@node3 ~]# srvctl add -h

The SRVCTL add command adds the configuration and the Oracle Clusterware application to the OCR for the cluster database, named instances, named services, or for the named nodes.

Usage: srvctl add vip -n <node_name> -k <network_number> -A <name|ip>/<netmask>/[if1[|if2...]] [-v]

[root@node1 ~]# cat /etc/hosts

# Do not remove the following line, or various programs

# that require network functionality will fail.

127.0.0.1 localhost

192.168.8.41 node1

192.168.8.43 node1-vip

10.1.1.1 node1-priv

192.168.8.42 node2

192.168.8.44 node2-vip

10.1.1.2 node2-priv

192.168.8.45 scan_ip

192.168.8.46 node3

192.168.8.47 node3-vip

10.1.1.3 node3-priv

[root@node3 ~]# srvctl add vip -n node3 -A 192.168.8.47/255.255.255.0/eth0 -k 0

PRCN-2049 : The network attributes specified (network number: 0, subnet: 192.168.8.0, adapters: eth0) conflict with an already registered network (network number: 1, subnet: 192.168.8.0, adapters: eth0)

[root@node3 ~]# srvctl add vip -n node3 -A 192.168.8.47/255.255.255.0/eth0 -k 1

[root@node3 ~]#

添加node3 vip service 成功！

[root@node3 ~]# crs_stat |grep vip

NAME=ora.node1.vip

TYPE=ora.cluster_vip_net1.type

NAME=ora.node2.vip

TYPE=ora.cluster_vip_net1.type

NAME=ora.node3.vip

TYPE=ora.cluster_vip_net1.type

NAME=ora.scan1.vip

TYPE=ora.scan_vip.type

[root@node3 ~]#

启动ora.node3.LISTENER_NODE3.lsnr service：

[root@node3 ~]# crs_stat |grep lsn

NAME=ora.LISTENER.lsnr

NAME=ora.LISTENER_SCAN1.lsnr

NAME=ora.node1.LISTENER_NODE1.lsnr

NAME=ora.node2.LISTENER_NODE2.lsnr

NAME=ora.node3.LISTENER_NODE3.lsnr

[root@node3 ~]# crs_start -f ora.node3.LISTENER_NODE3.lsnr

Attempting to start `ora.node3.vip` on member `node3`

Start of `ora.node3.vip` on member `node3` succeeded.

Attempting to start `ora.LISTENER.lsnr` on member `node3`

Start of `ora.LISTENER.lsnr` on member `node3` succeeded.

[root@node3 ~]#

node3 Grid 日志：

[root@node3 node3]# pwd

/u01/11.2.0/grid/log/node3

[root@node3 node3]# tail -f alertnode3.log

[gpnpd(7901)]CRS-2332:Error pushing GPnP profile to "mdns:service:gpnp._tcp.local.://node1:12802/agent=gpnpd,cname=node-cluster,host=node1,pid=3431/gpnpd h:node1 c:node-cluster".

2014-04-17 14:47:42.399