EM12c upgrade from 12.1.0.2 to 12.1.0.3

Now that Oracle Enterprise Manager Cloud Control 12c Release 3 (aka EM12cR3!) has been out for a few weeks, I decided to brave it and upgrade our Production stack today.

In this post, I’ll be upgrading an EM12cR2 (12.1.0.2) installation which consists of two OMSes, running on Linux x86-64 (RHEL5) which uses an 11.2.0.3 database management repository.  I do not have BI Publisher installed here, if you do, then take a look at this.

Before you begin, make sure you have plenty of disk space.  The EM12cR3 disk requirements have changed slightly, in that it now wants a minimum of 14GB available per OMS (that’s with the EM software library configured), as opposed to 7GB in EM12cR2 (without the software library configured).  My EM software library sits on a different filesystem, so I ended up having to get another 7GB of space added to the OMS filesystem before I could carry on with the upgrade (the installer checks part way through).

 

Download and extract the software

First of all, download the zip parts here and extract everything to a staging area:

unzip /tmp/em12103_linux64_disk1.zip -d /u01/app/oracle/software
unzip /tmp/em12103_linux64_disk2.zip -d /u01/app/oracle/software
unzip /tmp/em12103_linux64_disk3.zip -d /u01/app/oracle/software *

* this part actually caught me out straight away:

unzip /tmp/em12103_linux64_disk3.zip -d /u01/app/oracle/software
unzip: cannot find or open /tmp/em12103_linux64_disk3.zip, /tmp/em12103_linux64_disk3.zip.zip or /tmp/em12103_linux

Whilst it states on the download page that you need to use unzip version 6, I’d completely overlooked it, and the 3rd zip file would refuse to unpack using unzip v5.52.  You can download unzip 6 here for your OS, after using that, everything extracted for me just fine 🙂

Clean up the zip files once extracted, as they take up quite a bit of space:

rm /tmp/em12103_linux64_disk*.zip

Backup your installation and database

Shutdown our OMSes and central agents:

export OMS_HOME=/u03/app/oracle/middleware/oms12c/oms
export AGENT_HOME=/u03/app/oracle/middleware/agent12c/core/12.1.0.2.0
$AGENT_HOME/bin/emctl stop agent
$OMS_HOME/bin/emctl stop oms -all

…then backup your OMS installation, for example:

tar -cvf /u01/app/oracle/oms12c-pre-rel3-oraInv.tar /u03/app/oracle/oraInventory
tar -cvf /u01/app/oracle/oms12c-pre-rel3.tar /u03/app/oracle/middleware

Also, make sure you take a full backup of your Management Repository database, and of your software library.

Prepare the Management Repository (Database)

Check for EM snapshots

Ensure that the tables in the Management Repository do not contain any shapshots.  To check this, run the following SQL as the SYSMAN user:

select master, log_table from all_mview_logs where log_owner='SYSMAN';

If there are any, then you’ll need to drop them as mentioned in step 2c of the upgrade documentation here.

Copy the EMKey

Next, copy the emkey from the existing OMS to the Management Repository.  To do so, run the following commands on the OMS home you are about to upgrade, entering the SYSMAN password when prompted:

export OMS_HOME=/u03/app/oracle/middleware/oms12c/oms
$OMS_HOME/bin/emctl start oms

$OMS_HOME/bin/emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

Then shutdown the OMS again:

$OMS_HOME/bin/emctl stop oms -all

Apply required database patch(es)

To avoid any database upgrade failures, make sure you apply any pre-requisite patches accordingly as detailed in step 2d here.  In my case, I have an 11.2.0.3 database repository, so need to ensure generic patch 11061801 is applied:

$ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc $ORACLE_HOME/oraInst.loc | grep 11061801

Shutdown the database, listener, and apply the patch:

export PATH=$PATH:$ORACLE_HOME/OPatch
mkdir $ORACLE_HOME/Patches
unzip /tmp/p11061801_112030_Generic.zip -d $ORACLE_HOME/Patches
cd $ORACLE_HOME/Patches/11061801
opatch prereq CheckConflictAgainstOHWithDetail -ph ./ -invPtrLoc $ORACLE_HOME/oraInst.loc
opatch apply -invPtrLoc $ORACLE_HOME/oraInst.loc

Once applied, connect as SYS and run the following in order:

sqlplus / as sysdba
startup
@?/rdbms/admin/owmctrg.plb
alter trigger wmsys.no_vm_ddl enable;
alter trigger wmsys.no_vm_drop_a enable;

Once everything has been shutdown, backed up and prepared as above, make sure any cron jobs etc. are also disabled.  As an example, I have jobs which use EMCLI commands which run quite frequently.

Start the upgrade of your OMS (1-System Upgrade)

Start the installer from your primary OMS first.

/u01/app/oracle/software/runInstaller -invPtrLoc $OMS_HOME/oraInst.loc

Once the GUI starts, respond as follows:

• My Oracle Support Details
  • Uncheck ‘I wish to receive security updates via My Oracle Support’
  • Next
  • Confirm with ‘Yes’

 

•  Software Updates
  • Skip software updates
  • Next

• Prerequisites Checks
  • All prerequisite checks should complete successfully here, if not, then obviously they will need addressing before proceeding.
  • Next

• Installation Types
  • Select ‘Upgrade an existing Enterprise Manager System’
  • Select ‘One System Upgrade’
  • Select the OMS you want to upgrade
  • Next

• Installation Details
  • Middleware Home Location: /u03/app/oracle/middleware/oms12c_r3
  • Next
  • NOTE: Upgrade to 12c Release 3 (12.1.0.3) is an out-of-place upgrade, therefore you must either select an existing middleware home (where Enterprise Manager is not already installed) or enter a new home.

• Database Connection Details
  • Check the ‘Connect Descriptor’
  • Enter the ‘SYS Password’
  • Enter the ‘SYSMAN Password’
  • Tick to ‘Confirm that you have backed up the Management Repository’ (if you still haven’t, do it now!)
  • Next

• At this point, the installer will perform a series of pre-requisite checks against your database repository so don’t expect it to jump onto the next step straight away 😉
• After a while the following recommendations were reported, which I accepted with responding ‘Yes’

• Again, you’ll need to be patient here whilst the changes are implemented.
• Further checks then revealed the following, so I granted the permission by responding ‘OK’

• Plug-in Upgrade
  • This page will obviously vary depending on your installation.  Check the version details.
  • Next

• Select Plug-ins
  • Here you have the option of deploying additional plug-ins as part of the upgrade. I left all unchecked.
  • Next

• Extend WebLogic Sever Domain
  • Enter your Admin Server host, port, and WebLogic user details
  • OMS Instance Base Location: /u03/app/oracle/middleware/gc_inst
  • Next
  • NOTE: I was tempted here to change the default location to /u03/app/oracle/middleware/oms12c_r3/gc_inst, as I prefer to keep everything under the MW_HOME if possible, but this note about it persuaded me to leave as is, in case it causes any complications further down the line.

• Review
  • Check the install locations, ports etc. and make sure you’re happy with everything 🙂
  • Install

• You can monitor the installation progress by clicking on the ‘View Log’ link

     

• When prompted, run the following script as root (or via sudo):
  • /u03/app/oracle/middleware/oms12c_r3/oms/allroot.sh

     

From the point of clicking install, the upgrade process of the primary OMS and database repository took 1hr 10mins in total.

Upgrade additional OMSes

Once the primary OMS has been upgraded, run the upgrade above against each additional OMS.  The only differences I observed during the installation wizard were as follows:

1. Database Connection Details: whilst they’re still required, no pre-checks are done here as the management repository has already been upgraded.
2. Extend WebLogic Sever Domain: ensure the Admin Server details linked to your primary OMS are used (the installer should default to it).
3. Port Configuration Details: optionally, change the ports allocated to your Node Manager and Managed Server (OMS).

     

From the point of clicking install, the upgrade process of my second OMS took 20 minutes.

Upgrade the Management Agents

After upgrading all of the OMS instances, upgrade the Management Agents, starting with the ‘central agents’ that were installed with old OMS(es) first.  This is not done automatically as part of the OMS upgrade.

export AGENT_HOME=/u03/app/oracle/middleware/agent12c/core/12.1.0.2.0
$AGENT_HOME/bin/emctl start agent

Log into the EM12c console as SYSMAN and navigate to: Help > About Enterprise Manager

…and you’ll see confirmation that you’re running release 3 now 😉

Navigate to: Setup >Manage Cloud Control > Upgrade Agents.

Add your central Management Agent(s) for upgrade, check the credentials etc. and hit submit:

You may see a warning about privileges and the root.sh script.  If you do, just continue and remember to run the root.sh script manually after the agent has been successfully upgraded:

/u01/app/oracle/product/agent12c/core/12.1.0.3.0/root.sh

Repeat this process so that all Management Agents are upgraded to version 12.1.0.3.

Alternatively, you can you EMCLI commands to upgrade your agents.

NOTE: You may need to download new 12.1.0.3 versions of the agent software using ‘Self Update’ before all Management Agents can be upgraded.

Remove the old agent software

Once that’s done, navigate to: Navigate to: Setup >Manage Cloud Control > Upgrade Agents > Post Agent Upgrade Task

When you’re confident everything is working as expected with each agent, you can submit clean-up jobs to delete the obsolete directories ($AGENT_HOME/core/12.1.0.2.0/* and $AGENT_HOME/backup_*).

Remove the old OMS software

I won’t go into this in much detail, but when you’re ready to the move the old OMS software, follow the process documentated here by starting off with the following:

export OMS_HOME=/u03/app/oracle/middleware/oms12c/oms
$OMS_HOME/oui/bin/runInstaller -deinstall -removeAllFiles -invPtrLoc $OMS_HOME/oraInst.loc

Don’t forget to update any scripts that refer to the OMS and agent homes accordingly, as they will have changed:

OMS_HOME=/u03/app/oracle/middleware/oms12c_r3/oms
AGENT_HOME=/u03/app/oracle/middleware/agent12c/core/12.1.0.3.0

Out-of-the-box incident rules

Oracle documentation states….

“As a best practice, you should create your own copies of out-of-box rule sets and then subscribe to the rule set copies rather than subscribing directly to the out-of-box rule sets.”

This is important….I noticed that as part of the upgrade from EM12cR2 to EM12cR3, users previously subscribed to the “Incident management rule set for all targets” system generated rule, had been removed.  As a result, notifications weren’t being sent out after the upgrade, as they were previously, so make sure you use a custom rule set, even if it’s just a copy of the out-of-the-box one 😉 …otherwise you may well be wondering what’s happened to your notifications after the upgrade!

Overall, the upgrade went pretty smoothly for both OMSes, the Management Repository, and Management Agents, hope this helps…

References:
Upgrading Primary Enterprise Manager
Upgrading OMS and Management Repository
Performing Post Upgrade Tasks
Deleting Old OMS Home
What’s New in Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3)
Understanding Enterprise Manager 12.1.0.3 Upgrade and Agent Upgrades