Automating relocation of targets within EM for HP-UX Serviceguard

Recently, I found myself in a situation whereby multiple database instances were running on a HP-UX clustered environment to ensure high availability, each with their own Serviceguard package, on an agent per package basis.

Now on the plus side, if the SG package is failed over to another node, the agent goes with it, and you’ve nothing to worry about…but if you’ve multiple database instances running, each with their own agent installed, then you can potentially end up with a lot of overhead, thus wasting resources, and additional maintenance…not to mention all the extra disk space!

Based on my previous post about relocating targets with EMCLI via Enterprise Manager, I wanted to implement an automated solution so that when a Serviceguard package is failed over to another node, and the database/listener targets are moved as part of that, and I don’t have to issue relocate commands using EMCLI manually…

First of all,  I removed all of the existing Oracle Management Agent installations from the packages.  Then I installed a single agent on a local mount point (not part of any SG package) for each node, and configured the database/listener targets using the EM console.

I then created this shell script: em_relocate_checker.sh

…which when executed, does the following

1. Compiles a list of all running Oracle PMON processes (exceptions can be added)
2. Connects to EM Cloud Control and fetches the host name registered against the database instance (if the target doesn’t exist in EM, it will send an email to let you know!)
3. Checks the local host name and compares it with the one from EM
4. If they match, ignore that instance and check the next one
5. If they differ, then execute a function within the scripts which relocates the database and listener targets to the correct host within EM, and fires off an email 🙂

This script assumes the following:

1. EMCLI has been installed and configured on the machine.
2. EMCLI has been setup to use ‘auto-login’.
3. Only database and listener targets are being migrated.
4. EMREP is the name of the EM database repository and preferred credentials (SYSDBA) have been defined for EMREP within EM.

The following variables should be set to your environment as appropriate:

export AGENT_HOME=/emagent/agent12c
export AGENT_PORT=3872
export FQDN=local.domain
export MAIL_LIST=mail@example.com

Example usages of the script would be as follows…

Check against all running database instances (default):

em_relocate_checker.sh all_sids

Check against all running database instances, except for SIDs test1, test2, test3:

em_relocate_checker.sh "test1|test2|test3"

Check against all running database instances, except for the SID test:

em_relocate_checker.sh test

On each of the Serviceguard nodes, I scheduled in a cron job to run the em_relocate_checker.sh script every 15 minutes:

# Checks current ora_pmon processes against EM12c and relocates DB/LSN targets if targets have moved hosts.
0,15,30,45 * * * * /emagent/emcli/em_relocate_checker.sh all_sids >> /dev/null 2>&1

There are a number of tweaks I could make to improve this script, so it’s by no means perfect (no error handling yet for when the OMS is down or unreachable for example), but I’ve implemented this in Production across multiple nodes/SG packages, and it’s served the purpose I wanted it to for now, so hopefully it will be of some use to someone else out there looking to reduce the number of agents in a HP-UX Serviceguard setup 🙂