Oracle Class – Wissem's Oracle and NOSQL Tips – Tunisia's First Oracle, NOSQL Website

Pdf version of the post available for download here
Introduction:

In this post, I am going to explain you how to successfully apply a Patch Set Update 3 (PSU 3) on the top of the Patch Set Update 2 (11.2.0.2.2 version).
In my previous article “RAC 11.2.0.2 Patch 12311357 – 11.2.0.2.2 GI Patch Set Update and mutex patch – 4 Nodes on Redhat Linux”, I demonstrated how to apply the patch set update 2 on the top of patch set version 2 (11.2.0.2.0). You may ask this question; what is difference between Patch Set and Patch Set Update?
I will try to answer to that question in the following paragraph.

Definition:

Yesterday, (14 September 2011) Oracle released Patch set number 3 (11.2.0.3) on Linux x86 and Linux x86-64 (refer to My Oracle Support 10404530 for download. So what is Patch Set?
Patch Set is the The fourth digit, component specific release number of 11.2.0.3. Oracle Support adds new features to the Patch Set.
Patch Set Update is the fifth number 11.2.0.2.3; Every 3 months or so, Oracle Support release a Patch Set Update (PSU) to fix Patch Set bugs and does not add any new features to the PSU.

Now, you may ask what about the other 3 digits; the first, the second and the third?

The first digit is the major version, in our case 11g. The second digit is the release number, in our case it is release 2 also called (11g release 2 or shortly 11gR2) with major upgrades and major new features. The third digit is Oracle application specific number, always equal to 0 for database software.

RAC rolling upgrade:

Only the node that is currently being patched has its cluster resources down (database instance, services, db console, …) where the other nodes of the cluster remain available. In this post we will use roll upgrade method to patch the cluster.

Patch Set Update 3:

Patch number 12419353 includes the grid infrastructure and database patch set update number 3.
Since, we are going to patch a RAC (Real Application Cluster) environment, we should apply this patch number. In the case, you need to patch only the database, you can use the patch number: 12419331.
Actually, there are different issues with the opatch auto method to patch the grid infrastructure and database homes. In this post, I am going to use the manual method.

Prerequisites for manual patching:

These are the prerequisites to patch the RAC environment:

1- OPatch utility version 11.2.0.1.5 or higher (Recommended the latest version).
2- Validate the Oracle Inventory
3- Cluster resources must be down for the node that is currently being patched.

Patching Steps:

The following steps must be performed on all nodes of the cluster.

1- Download the latest opatch version: In this post we have downloaded the opatch version 11.2.0.1.6. Please refer to the document ID: 6880880 for opatch download.
2- Donwload the PSU number 3: From my Oracle Support web site download the patch number 12419353.
3- Check the download files:
Check Sum the download files, the result should be similar to this output:
You can check the integrity of a patch using the below:
Note 549617.1 How To Verify The Integrity Of A Patch/Software Download?

[root@dddb01 patch3]# md5sum p12419353_112020_Linux-x86-64.zip

1fcd06f26eccf5bbf49ab30bc6b7d7cf  p12419353_112020_Linux-x86-64.zip

[root@dddb01 patch3]# md5sum p6880880_112000_Linux-x86-64.zip

017e3e62f7e7da6fdb897afb228f6f0b  p6880880_112000_Linux-x86-64.zip

[root@dddb01 patch3]#

Check the zipped files:

[root@dddb01 patch3]# unzip -t p12419353_112020_Linux-x86-64.zip

...

...

/rep/editUserSummary.uix   OK

    testing: 12419331/etc/            OK

    testing: 12419331/etc/xml/        OK

    testing: 12419331/etc/xml/GenericActions.xml   OK

    testing: 12419331/etc/xml/ShiphomeDirectoryStructure.xml   OK

    testing: 12419331/etc/config/     OK

    testing: 12419331/etc/config/inventory.xml   OK

    testing: 12419331/etc/config/actions.xml   OK

    testing: 12419331/etc/config/deploy.xml   OK

    testing: bundle.xml               OK

    testing: README.txt               OK

    testing: README.html              OK

No errors detected in compressed data of p12419353_112020_Linux-x86-64.zip.
[root@dddb01 patch3]# unzip -t p6880880_112000_Linux-x86-64.zip

4- Install the opatch in grid infrastructure home:

The GRID_HOME directory is /opt/11.2.0/grid/
.
For RAC installation, we used the role separation; Oracle system user owner of Oracle Software and grid system user Cluster owner.

[root@dddb01 patch3]# cd /opt/patch3/

[root@dddb01 patch3]# cp p6880880_112000_Linux-x86-64.zip /opt/11.2.0/grid/

[root@dddb01 db_1]# cd /opt/11.2.0/grid/

[root@dddb01 grid]# mv OPatch OPatch.20110923.OLD

[root@dddb01 grid]# unzip p6880880_112000_Linux-x86-64.zip

[root@dddb01 db_1]# rm -f p6880880_112000_Linux-x86-64.zip

[root@dddb01 grid]# chown -R grid:oinstall /opt/11.2.0/grid/OPatch

[root@dddb01 grid]# su - grid

-bash-3.2$ which opatch

/opt/11.2.0/grid/OPatch/opatch

-bash-3.2$ opatch version

Invoking OPatch 11.2.0.1.6

OPatch Version: 11.2.0.1.6

OPatch succeeded.

-bash-3.2$ exit;

logout

[root@dddb01 grid]#

5- Install the opatch in database home:
The ORACLE_HOME directory is opt/app/oracle/product/11.2.0/db_1

[root@dddb01 patch3]# cd /opt/patch3/

[root@dddb01 patch3]# cp p6880880_112000_Linux-x86-64.zip /opt/app/oracle/product/11.2.0/db_1

[root@dddb01 patch3]# cd /opt/app/oracle/product/11.2.0/db_1

[root@dddb01 db_1]# mv OPatch OPatch.20110923.OLD

[root@dddb01 db_1]# unzip p6880880_112000_Linux-x86-64.zip

[root@dddb01 db_1]# rm -f p6880880_112000_Linux-x86-64.zip

[root@dddb01 db_1]# chown -R oracle:oinstall /opt/app/oracle/product/11.2.0/db_1/OPatch

[root@dddb01 db_1]# su - oracle

-bash-3.2$ which opatch

/opt/app/oracle/product/11.2.0/db_1/OPatch/opatch

-bash-3.2$ opatch version

Invoking OPatch 11.2.0.1.6

OPatch Version: 11.2.0.1.6

OPatch succeeded.

-bash-3.2$ exit;

6- Validate the Oracle Inventory for grid and Oracle users:

[root@dddb01 grid]# su - grid

-bash-3.2$ opatch lsinventory -detail -oh $GRID_HOME

....

...

..

 Patch Location in Inventory:

     /opt/11.2.0/grid/inventory/oneoffs/12311357

   Patch Location in Storage area:

     /opt/11.2.0/grid/.patch_storage/12311357_Apr_5_2011_09_13_41

Rac system comprising of multiple nodes

  Local node = dddb01

  Remote node = dddb02

  Remote node = dddb03

  Remote node = dddb04

  Remote node = dddb05

  Remote node = dddb06

--------------------------------------------------------------------------------

OPatch succeeded.

-bash-3.2$ exit;

Oracle Home:

[root@dddb01 grid]# su - oracle

-bash-3.2$ opatch lsinventory -detail -oh $ORACLE_HOME

....

...

..

Rac system comprising of multiple nodes

  Local node = dddb01

  Remote node = dddb02

  Remote node = dddb03

  Remote node = dddb04

  Remote node = dddb05

  Remote node = dddb06

--------------------------------------------------------------------------------

OPatch succeeded.

-bash-3.2$ exit;

6- Check for patch Conflicts:
Unzip the patch file and check for patch conflict.
Patch unzipped location is /opt/patch3/.

[root@dddb01 patch3]# unzip  p12419353_112020_Linux-x86-64.zip

[oracle@dddb01 patch_psu_3]$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/patch_psu_3/12419331

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6
Copyright (c) 2010, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /opt/app/oracle/product/11.2.0/db_1

Central Inventory : /opt/app/oraInventory

   from           : /etc/oraInst.loc

OPatch version    : 11.2.0.1.6
OUI version       : 11.2.0.2.0

OUI location      : /opt/app/oracle/product/11.2.0/db_1/oui

Log file location : /opt/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-09-21_01-38-05AM.log

Patch history file: /opt/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt

Invoking prereq "checkconflictagainstohwithdetail"

ZOP-47: The patch(es) has supersets with other patches installed in the Oracle Home (or) among themselves.

Prereq "checkConflictAgainstOHWithDetail" failed.

Summary of Conflict Analysis:

Patches that can be applied now without any conflicts are :

12419331

Following patches are not required, as they are subset of the patches in Oracle Home or subset of the patches in the given list :

12431716, 11724916

Following patches will be rolled back from Oracle Home on application of the patches in the given list :

12431716, 11724916

Conflicts/Supersets for each patch are:

Patch : 12419331

        Bug Superset of 12431716

        Super set bugs are:

        12431716

        Bug Superset of 11724916

        Super set bugs are:

        10151017,  10158965,  11724916,  10190642,  10129643,  10018789,  9744252,  10248523,  9956713,  10356513,  9715581,  9770451,  10170431,  10425676,  10222719,  9591812,  10127360,  10094201,  11069199,  10245086,  10205230,  10052141,  11818335,  11830776,  11830777,  9905049,  11830778,  10077191,  10358019,  10219576,  10264680,  10209232,  10102506,  11067567,  9881076,  10040531,  10218814,  9788588,  9735237,  10230571,  10079168,  10228151,  10013431,  10217802,  10238786,  10052956,  11699057,  10080579,  10332111,  10227288,  10329146,  10332589,  10110863,  10073683,  10019218,  10229719,  10373381,  11724984,  9539440,  10411618,  10022980,  10187168,  6523037,  9724970,  10084145,  10157402,  9651350,  10299224

OPatch succeeded.

[oracle@dddb01 patch_psu_3]$

From the above output, we get these conclusions:

Patches that can be applied now without any conflicts are :
12419331

Following patches are not required, as they are subset of the patches in Oracle Home or subset of the patches in the given list :
12431716, 11724916

Following patches will be rolled back from Oracle Home on application of the patches in the given list :
12431716, 11724916

We can apply the PSU.

7- Stop Enterprise Manager Console:

In my environment, I have DB console running from first node of the cluster with agent deployed on all remaining nodes of the cluster. For automatic restart of the DB console, I have added it as a cluster resource (for more information of how to do it, you can refer to my previous post: 11gR2 RAC: add Enterprise Manager Database Control to crs resources for automatic restart (http://www.oracle-class.com/?p=2389).

Stop the DB console resource from first node and as grid user:
Do this step once as it will stop the cluster resource on all nodes of the cluster.

crsctl stop res crs_dbconsole

Stop the DB console from Node being patched if it is not stopped after running the above command:

[oracle@dddb01 ~]$ emctl stop dbconsole

Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0

Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.

https://dddb01.dddb.ddomain.com:5500/em/console/aboutApplication

Stopping Oracle Enterprise Manager 11g Database Control ...

 ...  Stopped.

[oracle@dddb01 ~]$

From now on, please do the following steps on each node one at a time; (this is how I did, I didn’t tested the result of doing the following steps on all nodes in parallel).

8- Check you have enough disk space: The patch may fail with a fatal error if you don’t have enough space on disk.
9- Stop the cluster resources on the node being patched:

[oracle@dddb01 ~]$ su - oracle

[oracle@dddb01 ~]$ srvctl stop home -o $ORACLE_HOME -s /tmp/res_stat.out -n dddb01

10- As a root user run the following:

[oracle@dddb01 ~]$ su -

[root@dddb01 ~]# /opt/11.2.0/grid/crs/install/rootcrs.pl -unlock

Using configuration parameter file: /opt/11.2.0/grid/crs/install/crsconfig_params

Successfully unlock /opt/11.2.0/grid

[root@dddb01 ~]#

[root@dddb01 patch3]# chown -R grid:oinstall /opt/patch3/

11- Grid Infrastructure Home patching:

[root@dddb01 patch3]# su - grid

[grid@dddb01 ~]$ which opatch

/opt/11.2.0/grid/OPatch/opatch

[grid@dddb01 ~]$ opatch version

Invoking OPatch 11.2.0.1.6

OPatch Version: 11.2.0.1.6

OPatch succeeded.

[grid@dddb01 ~]$

[grid@dddb01 ~]$  /opt/11.2.0/grid/OPatch/opatch napply -oh $GRID_HOME -local /opt/patch3/12419353

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6

Copyright (c) 2011, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /opt/11.2.0/grid

Central Inventory : /opt/app/oraInventory

   from           : /etc/oraInst.loc

OPatch version    : 11.2.0.1.6

OUI version       : 11.2.0.2.0

Log file location : /opt/11.2.0/grid/cfgtoollogs/opatch/opatch2011-09-22_02-58-0

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Patch : 12419353

        Bug Superset of 12311357

        Super set bugs are:

        12311357,  10425672,  10244210,  11655840,  10634513,  9891341,  1178242,  10089120,  10057296,  10053985,  9864003,  10044622,  9812970,  10083789,  10185,  10071992,  10038791,  10048487,  9926027,  10260251,  10052721,  10028235, 10072474,  10036834,  10028637,  10029900,  9974223,  9979706,  10016083,  1001,  10305361,  10029119,  10233159,  10083009,  9812956,  10008467,  10036193,  19401,  10111010,  10075643,  10057680,  10205290,  10124517,  10078086,  9944948  10168006,  9949676,  10228079,  10015603,  10241696,  9861790,  10069698,  10019796,  9975343,  10216878,  9906432,  10045316,  10029794,  10425675,  10061534

Patches [   12311357 ] will be rolled back.

Do you want to proceed? [y|n]

y

User Responded with: Y

OPatch continues with these patches:   12419353

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name:

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local sy

(Oracle Home = '/opt/11.2.0/grid')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '12419353' to OH '/opt/11.2.0/grid'

Rolling back interim patch '12311357' from OH '/opt/11.2.0/grid'

Patching component oracle.crs, 11.2.0.2.0...

Patching component oracle.usm, 11.2.0.2.0...

RollbackSession removing interim patch '12311357' from inventory

OPatch back to application of the patch '12419353' after auto-rollback.

Patching component oracle.crs, 11.2.0.2.0...

Copying file to "/opt/11.2.0/grid/crs/install/crsconfig_lib.pm"

Copying file to "/opt/11.2.0/grid/crs/install/crspatch.pm"

Copying file to "/opt/11.2.0/grid/crs/install/s_crsconfig_lib.pm"

Patching component oracle.usm, 11.2.0.2.0...

Patches 12419353 successfully applied.

OPatch Session completed with warnings.

Log file location: /opt/11.2.0/grid/cfgtoollogs/opatch/opatch2011-09-22_02-58-07AM.log

OPatch completed with warnings.

[grid@dddb01 ~]$

you can safely ignore the warning as it is only related to “You have not provided an email address for notification of security issues.”

[grid@dddb01 ~]$ /opt/11.2.0/grid/OPatch/opatch napply -oh $GRID_HOME -local /opt/patch3/12419331/

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6

Copyright (c) 2011, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /opt/11.2.0/grid

Central Inventory : /opt/app/oraInventory

   from           : /etc/oraInst.loc

OPatch version    : 11.2.0.1.6

OUI version       : 11.2.0.2.0

Log file location : /opt/11.2.0/grid/cfgtoollogs/opatch/opatch2011-09-22_03-48-08AM.log

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Patch : 12419331

        Bug Superset of 12431716

        Super set bugs are:

        12431716

        Bug Superset of 11724916

        Super set bugs are:

        10151017,  10158965,  11724916,  10190642,  10129643,  10018789,  9744252,  10248523,  9956713,  10 9591812,  10127360,  10094201,  11069199,  10245086,  10205230,  10052141,  11818335,  11830776,  118307774680,  10209232,  10102506,  11067567,  9881076,  10040531,  10218814,  9788588,  9735237,  10230571,  1007 11699057,  10080579,  10332111,  10227288,  10329146,  10332589,  10110863,  10073683,  10019218,  102297187168,  6523037,  9724970,  10084145,  10157402,  9651350,  10299224

Patches [   12431716   11724916 ] will be rolled back.

Do you want to proceed? [y|n]

y

User Responded with: Y

OPatch continues with these patches:   12419331

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name:

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/opt/11.2.0/grid')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '12419331' to OH '/opt/11.2.0/grid'

ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.2.0 ] , [ oracle.sysman.oms.core, 10n.plugin.db.main.repository, 11.2.0.2.0 ]  not present in the Oracle Home or a higher version is found.

Rolling back interim patch '12431716' from OH '/opt/11.2.0/grid'

Patching component oracle.rdbms, 11.2.0.2.0...

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

RollbackSession removing interim patch '12431716' from inventory

Rolling back interim patch '11724916' from OH '/opt/11.2.0/grid'

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

Patching component oracle.rdbms, 11.2.0.2.0...

Patching component oracle.ldap.rsf, 11.2.0.2.0...

Deleting "ssl_getconnectionrenegotiationoptions.o" from archive "/opt/11.2.0/grid/lib/libnnz11.a"

Deleting "ssl_getrenegotiationoptions.o" from archive "/opt/11.2.0/grid/lib/libnnz11.a"

Deleting "ssl_hshk_priv_dooptionsallowrenegotoation.o" from archive "/opt/11.2.0/grid/lib/libnnz11.a"

Deleting "ssl_modifyrenegotiationoptions.o" from archive "/opt/11.2.0/grid/lib/libnnz11.a"

Deleting "ssl_setrenegotiationoptions.o" from archive "/opt/11.2.0/grid/lib/libnnz11.a"

Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...

RollbackSession removing interim patch '11724916' from inventory

OPatch back to application of the patch '12419331' after auto-rollback.

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

Patching component oracle.rdbms, 11.2.0.2.0...

Copying file to "/opt/11.2.0/grid/psu/11.2.0.2.3/catpsu.sql"

Copying file to "/opt/11.2.0/grid/psu/11.2.0.2.3/catpsu_rollback.sql"

Copying file to "/opt/11.2.0/grid/cpu/scripts/patch_8837510.sql"

Copying file to "/opt/11.2.0/grid/cpu/scripts/emdb_recomp_invalids.sql"

Patching component oracle.ldap.rsf, 11.2.0.2.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...

Patching component oracle.rdbms.rman, 11.2.0.2.0...

Patches 12419331 successfully applied.

OPatch Session completed with warnings.

Log file location: /opt/11.2.0/grid/cfgtoollogs/opatch/opatch2011-09-22_03-48-08AM.log

OPatch completed with warnings.

[grid@dddb01 ~]$ exit;

12- Oracle Database Home patching:

[root@dddb01 patch3]# chown -R oracle:oinstall /opt/patch3/

[root@dddb01 patch3]#

[root@dddb01 patch3]# su - oracle

[oracle@dddb01 ~]$ which opatch

/opt/app/oracle/product/11.2.0/db_1/OPatch/opatch

[oracle@dddb01 ~]$ opatch version

Invoking OPatch 11.2.0.1.6

OPatch Version: 11.2.0.1.6

OPatch succeeded.

[oracle@dddb01 ~]$

[oracle@dddb01 ~]$ /opt/patch3/12419353/custom/server/12419353/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME

/opt/patch3/12419353/custom/server/12419353/custom/scripts/prepatch.sh completed successfully.

[oracle@dddb01 ~]$

[oracle@dddb01 ~]$ $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /opt/patch3/12419353/custom/server/12419353

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6

Copyright (c) 2011, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /opt/app/oracle/product/11.2.0/db_1

Central Inventory : /opt/app/oraInventory

   from           : /etc/oraInst.loc

OPatch version    : 11.2.0.1.6

OUI version       : 11.2.0.2.0

Log file location : /opt/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-09-22_03-55-25AM.log

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Patch : 12419353

        Bug Superset of 12311357

        Super set bugs are:

        12311357,  10425672,  10157506,  11655840,  10634513,  9891341,  11782423,  11077756,  10376847,  10178670,  9959110,  10314123,  10014392,  10157622,  10089120,  10057296,  10053985,  9864003,  10044622,  9812970,  10083789,  10073372,  9876201,  9963327,  10375649,  9336825,  10062301,  10018215,  10105195,  10007185,  10071992,  10038791,  10048487,  9926027,  10260251,  10052721,  10028235,  10027079,  10028343,  10231906,  10065216,  10045436,  9907089,  10175855,  10284828,  10072474,  10036834,  10028637,  10029900,  9974223,  9979706,  10016083,  10015460,  9918485,  9971646,  10040647,  9978765,  10069541,  9915329,  10107380,  10110969,  10305361,  10029119,  10233159,  10083009,  9812956,  10008467,  10036193,  10048027,  10040109,  10015210,  9944978,  10033106,  9978195,  10042143,  10284693,  9679401,  10111010,  10075643,  10057680,  10205290,  10124517,  10078086,  9944948,  10146768,  10052529,  10011084,  10073075,  10248739,  10236074,  10128191,  9975837,  10168006,  9949676

Patches [   12311357 ] will be rolled back.

Do you want to proceed? [y|n]

y

User Responded with: Y

OPatch continues with these patches:   12419353

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/opt/app/oracle/product/11.2.0/db_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '12419353' to OH '/opt/app/oracle/product/11.2.0/db_1'

Rolling back interim patch '12311357' from OH '/opt/app/oracle/product/11.2.0/db_1'

Patching component oracle.rdbms, 11.2.0.2.0...

RollbackSession removing interim patch '12311357' from inventory

OPatch back to application of the patch '12419353' after auto-rollback.

Patching component oracle.rdbms, 11.2.0.2.0...

Patches 12419353 successfully applied.

OPatch Session completed with warnings.

Log file location: /opt/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-09-22_03-55-25AM.log

OPatch completed with warnings.

[oracle@dddb01 ~]$

[oracle@dddb01 ~]$ $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /opt/patch3/12419331

Invoking OPatch 11.2.0.1.6

Oracle Interim Patch Installer version 11.2.0.1.6

Copyright (c) 2011, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /opt/app/oracle/product/11.2.0/db_1

Central Inventory : /opt/app/oraInventory

   from           : /etc/oraInst.loc

OPatch version    : 11.2.0.1.6

OUI version       : 11.2.0.2.0

Log file location : /opt/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-09-22_03-57-08AM.log

Verifying environment and performing prerequisite checks...

Conflicts/Supersets for each patch are:

Patch : 12419331

        Bug Superset of 12431716

        Super set bugs are:

        12431716

        Bug Superset of 11724916

        Super set bugs are:

        10151017,  10158965,  11724916,  10190642,  10129643,  10018789,  9744252,  10248523,  9956713,  10356513,  9715581,  9770451,  10170431,  10425676,  10222719,  9591812,  10127360,  10094201,  11069199,  10245086,  10205230,  10052141,  11818335,  11830776,  11830777,  9905049,  11830778,  10077191,  10358019,  10219576,  10264680,  10209232,  10102506,  11067567,  9881076,  10040531,  10218814,  9788588,  9735237,  10230571,  10079168,  10228151,  10013431,  10217802,  10238786,  10052956,  11699057,  10080579,  10332111,  10227288,  10329146,  10332589,  10110863,  10073683,  10019218,  10229719,  10373381,  11724984,  9539440,  10411618,  10022980,  10187168,  6523037,  9724970,  10084145,  10157402,  9651350,  10299224

Patches [   12431716   11724916 ] will be rolled back.

Do you want to proceed? [y|n]

y

User Responded with: Y

OPatch continues with these patches:   12419331

Do you want to proceed? [y|n]

y

User Responded with: Y

All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/opt/app/oracle/product/11.2.0/db_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

Applying interim patch '12419331' to OH '/opt/app/oracle/product/11.2.0/db_1'

Rolling back interim patch '12431716' from OH '/opt/app/oracle/product/11.2.0/db_1'

Patching component oracle.rdbms, 11.2.0.2.0...

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

RollbackSession removing interim patch '12431716' from inventory

Rolling back interim patch '11724916' from OH '/opt/app/oracle/product/11.2.0/db_1'

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

Patching component oracle.rdbms, 11.2.0.2.0...

Patching component oracle.sysman.console.db, 11.2.0.2.0...

Updating jar file "/opt/app/oracle/product/11.2.0/db_1/sysman/jlib/emCORE.jar" with "/opt/app/oracle/product/11.2.0/db_1/.patch_storage/11724916_Apr_1_2011_07_09_05/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/admin/rep/AdminResourceBundle.class"

Updating jar file "/opt/app/oracle/product/11.2.0/db_1/sysman/jlib/emCORE.jar" with "/opt/app/oracle/product/11.2.0/db_1/.patch_storage/11724916_Apr_1_2011_07_09_05/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/admin/rep/AdminResourceBundleID.class"

Updating jar file "/opt/app/oracle/product/11.2.0/db_1/sysman/jlib/emCORE.jar" with "/opt/app/oracle/product/11.2.0/db_1/.patch_storage/11724916_Apr_1_2011_07_09_05/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/admin/rep/UserData.class"

Patching component oracle.sysman.oms.core, 10.2.0.4.3...

Updating jar file "/opt/app/oracle/product/11.2.0/db_1/sysman/jlib/emCORE.jar" with "/opt/app/oracle/product/11.2.0/db_1/.patch_storage/11724916_Apr_1_2011_07_09_05/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/admin/rep/AdminResourceBundle.class"

Updating jar file "/opt/app/oracle/product/11.2.0/db_1/sysman/jlib/emCORE.jar" with "/opt/app/oracle/product/11.2.0/db_1/.patch_storage/11724916_Apr_1_2011_07_09_05/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/admin/rep/AdminResourceBundleID.class"

Updating jar file "/opt/app/oracle/product/11.2.0/db_1/sysman/jlib/emCORE.jar" with "/opt/app/oracle/product/11.2.0/db_1/.patch_storage/11724916_Apr_1_2011_07_09_05/files//sysman/jlib/emCORE.jar/oracle/sysman/eml/admin/rep/UserData.class"

Patching component oracle.ldap.rsf, 11.2.0.2.0...

Deleting "ssl_getconnectionrenegotiationoptions.o" from archive "/opt/app/oracle/product/11.2.0/db_1/lib/libnnz11.a"

Deleting "ssl_getrenegotiationoptions.o" from archive "/opt/app/oracle/product/11.2.0/db_1/lib/libnnz11.a"

Deleting "ssl_hshk_priv_dooptionsallowrenegotoation.o" from archive "/opt/app/oracle/product/11.2.0/db_1/lib/libnnz11.a"

Deleting "ssl_modifyrenegotiationoptions.o" from archive "/opt/app/oracle/product/11.2.0/db_1/lib/libnnz11.a"

Deleting "ssl_setrenegotiationoptions.o" from archive "/opt/app/oracle/product/11.2.0/db_1/lib/libnnz11.a"

Patching component oracle.rdbms.dv, 11.2.0.2.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...

RollbackSession removing interim patch '11724916' from inventory

OPatch back to application of the patch '12419331' after auto-rollback.

Patching component oracle.rdbms.rsf, 11.2.0.2.0...

Patching component oracle.rdbms, 11.2.0.2.0...

Copying file to "/opt/app/oracle/product/11.2.0/db_1/psu/11.2.0.2.3/catpsu.sql"

Copying file to "/opt/app/oracle/product/11.2.0/db_1/psu/11.2.0.2.3/catpsu_rollback.sql"

Copying file to "/opt/app/oracle/product/11.2.0/db_1/cpu/scripts/patch_8837510.sql"

Copying file to "/opt/app/oracle/product/11.2.0/db_1/cpu/scripts/emdb_recomp_invalids.sql"

Patching component oracle.sysman.console.db, 11.2.0.2.0...

Patching component oracle.sysman.oms.core, 10.2.0.4.3...

Patching component oracle.ldap.rsf, 11.2.0.2.0...

Patching component oracle.rdbms.dv, 11.2.0.2.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.2.0...

Patching component oracle.sysman.plugin.db.main.repository, 11.2.0.2.0...

Patching component oracle.rdbms.rman, 11.2.0.2.0...

Patches 12419331 successfully applied.

OPatch Session completed with warnings.

Log file location: /opt/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-09-22_03-57-08AM.log

OPatch completed with warnings.

[oracle@dddb01 ~]$

[oracle@dddb01 ~]$ /opt/patch3/12419353/custom/server/12419353/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME

Reading /opt/app/oracle/product/11.2.0/db_1/install/params.ora..

Reading /opt/app/oracle/product/11.2.0/db_1/install/params.ora..

Parsing file /opt/app/oracle/product/11.2.0/db_1/bin/racgwrap

Parsing file /opt/app/oracle/product/11.2.0/db_1/bin/srvctl

Parsing file /opt/app/oracle/product/11.2.0/db_1/bin/srvconfig

Parsing file /opt/app/oracle/product/11.2.0/db_1/bin/cluvfy

Verifying file /opt/app/oracle/product/11.2.0/db_1/bin/racgwrap

Verifying file /opt/app/oracle/product/11.2.0/db_1/bin/srvctl

Verifying file /opt/app/oracle/product/11.2.0/db_1/bin/srvconfig

Verifying file /opt/app/oracle/product/11.2.0/db_1/bin/cluvfy

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/racgwrap

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/srvctl

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/srvconfig

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/cluvfy

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/diskmon.bin

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/lsnodes

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/osdbagrp

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/rawutl

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/srvm/admin/ractrans

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/srvm/admin/getcrshome

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/gnsd

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/bin/crsdiag.pl

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libclsra11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libdbcfg11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libocr11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libocrb11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libocrutl11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libuini11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/librdjni11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libgns11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libgnsjni11.so

Reapplying file permissions on /opt/app/oracle/product/11.2.0/db_1/lib/libagfw11.so

[oracle@dddb01 ~]$

13- Run post patch scripts:

[root@dddb01 patch3]# su -

[root@dddb01 patch3]# /opt/11.2.0/grid/rdbms/install/rootadd_rdbms.sh

[root@dddb01 patch3]#

[root@dddb01 patch3]# /opt/11.2.0/grid/crs/install/rootcrs.pl -patch

Using configuration parameter file: /opt/11.2.0/grid/crs/install/crsconfig_params

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9312: Existing ADVM/ACFS installation detected.

ACFS-9314: Removing previous ADVM/ACFS installation.

ACFS-9315: Previous ADVM/ACFS components successfully removed.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9309: ADVM/ACFS installation correctness verified.

CRS-4123: Oracle High Availability Services has been started.

[root@dddb01 patch3]#

14- Start Cluster resources on node being patched:

[root@dddb01 patch3]# su - oracle

[oracle@dddb01 ~]$ srvctl start home -o $ORACLE_HOME -s /tmp/res_stat.out -n dddb01

[oracle@dddb01 ~]$

[oracle@dddb01 ~]$ srvctl status service -d dddb

Post Patch verification:

One of the simple method I used to verify that patch has been applied successfully; I know there is a bug number 10636231 – High version count for INSERT .. RETURNING statements with reason … has been fixed by this patch. So, I am going to see if this bug number is listed fixed in the oracle inventory.
You can run the following the following commands:

su - root user:

script /tmp/opatch.txt

=> do this if you want to debug opatch: export OPATCH_DEBUG=TRUE

su - oracle

$ORACLE_HOME/OPatch/opatch lsinventory

opatch lsinventory -detail -oh $ORACLE_HOME

date

su - grid

opatch lsinventory -detail -oh $GRID_HOME

date

exit

Open the /tmp/opatch.txt file and check if bug number 10636231 is listed in the fixed bugs.
Now, you can start the DB console resource again if not started, as grid user run ;

crsctl start res crs_dbconsole