Patch - 22654153 Oracle Grid Infrastructure Patch Set Update 12.1.0.1.160419 (Apr2016)

[root@rac2 ~]#  export ORACLE_HOME=/u01/app/12.1.0/grid

[root@rac2 ~]# export PATH=$PATH:$ORACLE_HOME/OPatch

[root@rac2 ~]# cd $ORACLE_HOME/OPatch

[root@rac2 OPatch]# ./opatchauto apply /home/oracle/22654153 -ocmrf /home/oracle/ocm.rsp

OPatch Automation Tool

Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10

OUI Version        : 12.1.0.1.0

Running from       : /u01/app/12.1.0/grid

opatchauto log file: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/22654153/opatch_gi_2016-04-24_17-45-26_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /home/oracle/22654153

Grid Infrastructure Patch(es): 17077442 17303297 22291141

DB Patch(es): 17077442 22291141

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 17077442]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 17303297]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 21951844]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 20831107]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 19769486]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 21352619]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 20299016]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/12.1.0/grid

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 17077442]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/oracle/product/12.1.0/dbhome_1

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 21951844]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/oracle/product/12.1.0/dbhome_1

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 20831107]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/oracle/product/12.1.0/dbhome_1

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 19769486]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/oracle/product/12.1.0/dbhome_1

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 21352619]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/oracle/product/12.1.0/dbhome_1

The following patch(es) are duplicate patches with patches installed in the Oracle Home.

 [ 20299016]

You have already installed same patch(es) with same UPI(s) or same version(s).

These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /u01/app/oracle/product/12.1.0/dbhome_1

Patch Validation: Successful

Grid Infrastructure home:

/u01/app/12.1.0/grid

DB home(s):

/u01/app/oracle/product/12.1.0/dbhome_1

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/u01/app/oracle/product/12.1.0/dbhome_1" ...

Patch "/home/oracle/22654153/17077442" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".

Patch "/home/oracle/22654153/22291141" successfully applied to "/u01/app/oracle/product/12.1.0/dbhome_1".

Applying patch(es) to "/u01/app/12.1.0/grid" ...

Command "/u01/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /u01/app/12.1.0/grid/oraInst.loc -oh /u01/app/12.1.0/grid -silent -ocmrf /home/oracle/ocm.rsp" execution failed:

UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.

Log file Location for the failed command: /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2016-04-24_17-52-09PM_1.log

For more details, please refer to the log file "/u01/app/12.1.0/grid/cfgtoollogs/opatchauto/22654153/opatch_gi_2016-04-24_17-45-26_deploy.debug.log".

Apply Summary:

Following patch(es) are successfully installed:

DB Home: /u01/app/oracle/product/12.1.0/dbhome_1: 17077442,22291141

Following patch(es) failed to be installed:

GI Home: /u01/app/12.1.0/grid: 17077442,17303297,22291141

Command failure exception

opatchauto failed with error code 2.

Checked the logfiles and oracle unable to stop one of the process which impact to apply the patch.

[oracle@rac2 ~]$ tail -1000f /u01/app/12.1.0/grid/cfgtoollogs/opatch/opatch2016-04-24_17-52-09PM_1.log

[Apr 24, 2016 5:52:20 PM]    Start fuser command /sbin/fuser /u01/app/12.1.0/grid/lib/libclntsh.so.12.1 at Sun Apr 24 17:52:20 IST 2016

[Apr 24, 2016 5:52:20 PM]    Finish fuser command /sbin/fuser /u01/app/12.1.0/grid/lib/libclntsh.so.12.1 at Sun Apr 24 17:52:20 IST 2016

[Apr 24, 2016 5:52:20 PM]    Files in use by a process: /u01/app/12.1.0/grid/lib/libclntsh.so.12.1 PID(  3487 )

[Apr 24, 2016 5:52:20 PM]    Following executables are active :

                             /u01/app/12.1.0/grid/lib/libclntsh.so.12.1

[Apr 24, 2016 5:52:20 PM]    Prerequisite check "CheckActiveFilesAndExecutables" failed.

                             The details are:

                             Following executables are active :

                             /u01/app/12.1.0/grid/lib/libclntsh.so.12.1

[Apr 24, 2016 5:52:20 PM]    Start fuser command /sbin/fuser /u01/app/12.1.0/grid/bin/extjob at Sun Apr 24 17:52:20 IST 2016

[oracle@rac2 ~]$ fuser /u01/app/12.1.0/grid/lib/libclntsh.so.12.1

/u01/app/12.1.0/grid/lib/libclntsh.so.12.1:  3487m

[oracle@rac2 ~]$ ps -ef  | grep grid

oracle    3453     1  0 13:32 ?        00:00:00 /bin/sh /u01/app/12.1.0/grid/bin/cluvfy comp healthcheck -mandatory -_format

oracle    3487  3453  0 13:32 ?        00:00:21 /u01/app/12.1.0/grid/jdk/bin/java -classpath /u01/app/12.1.0/grid/jdk/lib/rt.jar:/u01/app/12.1.0/grid/jlib/cvu.jar:/u01/

app/12.1.0/grid/jlib/srvm.jar:/u01/app/12.1.0/grid/oui/jlib/OraInstaller.jar:/u01/app/12.1.0/grid/oui/jlib/jsch.jar:/u01/app/12.1.0/grid/oui/jlib/OraPrereq.jar:/u01/app

/12.1.0/grid/oui/jlib/prov_fixup.jar:/u01/app/12.1.0/grid/oui/jlib/xmlparserv2.jar:/u01/app/12.1.0/grid/oui/jlib/share.jar:/u01/app/12.1.0/grid/oui/jlib/orai18n-mapping

.jar:/u01/app/12.1.0/grid/jlib/srvmhas.jar:/u01/app/12.1.0/grid/jdbc/lib/ojdbc6.jar:/u01/app/12.1.0/grid/jlib/netcfg.jar:/u01/app/12.1.0/grid/jlib/ldapjclnt12.jar -DCV_

DESTLOC=/tmp -DCV_HOME=/u01/app/12.1.0/grid -DRUNNING.MODE=cvuresource -DCV_LOG_LOCATION=/u01/app/12.1.0/grid/log/rac2/cvu/cvulog/ -DCV_TRACELOC=/u01/app/12.1.0/grid/lo

g/rac2/cvu/cvutrc/ oracle.ops.verification.client.CluvfyDriver comp healthcheck -mandatory -_format

oracle   31503 29848  0 17:57 pts/1    00:00:00 grep grid

[oracle@rac2 ~]$ kill -9 3487

[oracle@rac2 ~]$ ps -ef  | grep grid

oracle   31508 29848  0 17:57 pts/1    00:00:00 grep grid

Now, Patch activity completed partially which may create inconsistency. We need to follow many steps manually to complete the patch and start the CRS service. Also, we can’t simply rerun the job.

 [root@rac2 OPatch]# ./opatchauto apply /home/oracle/22654153 -ocmrf /home/oracle/ocm.rsp

OPatch Automation Tool

Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10

OUI Version        : 12.1.0.1.0

Running from       : /u01/app/12.1.0/grid

opatchauto log file: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/22654153/opatch_gi_2016-04-24_17-58-13_deploy.log

Clusterware is either not running or not configured. You have the following 2 options:

1. Configure and start the Clusterware on this node and re-run the tool

2. Run the tool with '-oh ' to first patch the Grid Home, then invoke tool with '-database ' or '-oh ' to patch the RAC home

Parameter Validation: FAILED

Opatchauto Session failed: Parameter validation failed

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Stream closed

        at oracle.opatchauto.gi.GILogger.writeWithoutTimeStamp(GILogger.java:432)

        at oracle.opatchauto.gi.GILogger.printStackTrace(GILogger.java:447)

        at oracle.opatchauto.gi.OPatchauto.main(OPatchauto.java:97)

Caused by: java.io.IOException: Stream closed

        at java.io.BufferedWriter.ensureOpen(BufferedWriter.java:98)

        at java.io.BufferedWriter.write(BufferedWriter.java:203)

        at java.io.Writer.write(Writer.java:140)

        at oracle.opatchauto.gi.GILogger.writeWithoutTimeStamp(GILogger.java:426)

        ... 2 more

opatchauto failed with error code 1.

To avoid manual effort, follow the below step immediate. OPatch will resume the patch activity from where it stopped.

[root@rac2 OPatch]# ./opatchauto resume

OPatch Automation Tool

Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10

OUI Version        : 12.1.0.1.0

Running from       : /u01/app/12.1.0/grid

OPatchauto will attempt to resume the last run session. This might take several minutes...

opatchauto log file: /u01/app/12.1.0/grid/cfgtoollogs/opatchauto/22654153/opatch_gi_2016-04-24_17-59-26_deploy.log

Executing command:

/u01/app/12.1.0/grid/OPatch/opatch napply -phBaseFile /tmp/OraGI12Home1_oracle_patchList -local  -invPtrLoc /u01/app/12.1.0/grid/oraInst.loc -oh /u01/app/12.1.0/grid -silent -ocmrf /home/oracle/ocm.rsp

 as owner "oracle"

... above command successful.

Executing command:

/u01/app/12.1.0/grid/rdbms/install/rootadd_rdbms.sh

 as owner "root"

... above command successful.

Executing command:

/u01/app/12.1.0/grid/perl/bin/perl -I/u01/app/12.1.0/grid/perl/lib -I/u01/app/12.1.0/grid/OPatch/opatchautotemp_rac2/patchwork/crs/install /u01/app/12.1.0/grid/OPatch/opatchautotemp_rac2/patchwork/crs/install/rootcrs.pl -postpatch

 as owner "root"

... above command successful.

Executing command:

/bin/bash -c 'ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 ORACLE_SID=testdb2 /u01/app/oracle/product/12.1.0/dbhome_1/OPatch/datapatch'

 as owner "oracle"

... above command successful.

OPatchauto was able to resume from the previous patching session and complete successfully.

opatchauto succeeded.

[root@rac2 OPatch]#