Monitoring a Grid Control Installation

Posted at Wednesday, September 24, 2008

This posting covers how to monitor your 10g Grid Control installation. This is referred to as OOB (Out-of-Bound) Notification. I am using a version 10.2.0.4 OMS (Oracle Management Service), Repository Database, and Agent on Enterprise Linux.

In essence, the agent local to the OMS must be used in order to monitor your Grid Control installation via the Oracle-provided method (i.e. series of perl scripts, run by the agent).

This post assumes that you already have a local mail service - in my case sendmail - established and functioning.

Check the Agent and Verify the OMS Target

$ export ORACLE_SID=agent10g
$ . oraenv
$ emctl config agent listtargets | grep oracle_emrep

If the OMS target is missing then add it to the $AGENT_HOME/sysman/emd/targets.xml file

<target type="oracle_emrep" name="Management Services and Repository" version="1.0">
<property value="(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac1.colestock.test)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))" name="ConnectDescriptor"></property>
<property value="sysman" name="UserName" encrypted="FALSE"></property>
<property value="password" name="password" encrypted="FALSE"></property></target>

Adjust the aforementioned values to suit your environment

Reload the Agent

$ emctl reload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully

This will encrypt the username and password properties in the targets.xml

The agent should now be monitoring the OMS as a target

$ emctl config agent listtargets | grep oracle_emrep
[Management Services and Repository, oracle_emrep]

Update the $AGENT_HOME/sysman/config/emd.properties

Configure the email properties for the agent in emd.properties

emd_email_address=oracle@rac1.colestock.test,james@colestock.com
emd_email_gateway=localhost
emd_from_email_address=

Reload the agent again

$ emctl reload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD reload completed successfully

Verify $AGENT_HOME/bin/emrepdown.pl

This file may be missing from your agent installation. If so, copy the file from the OMS' $ORACLE_HOME to $AGENT_HOME/bin/emrepdown.pl

Review this file closely. For example, if you specified a value for emd_from_email_address earlier, then notifications might fail. This is due to the fact that the script uses the -r or "recipient" option of mailx; this option is not available on all Linux/UNIX-based distributions. Make the changes that are necessary for your environment.

Test Notification

Shut down the OMS and wait to see whether the notification email fires off.

$ export ORACLE_SID=oms10g
$ . oraenv
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...

The notification should take at least a few minutes to appear.

If you were to trace the agent's activity (DEBUG level) you would see the following in $AGENT_HOME/sysman/log/emagent_perl.trc

emrepnotif.pl: Wed Sep 24 18:28:55 2008: DEBUG: emrepnotif: Connectdescriptor (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac1.colestock.test)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))
emrepnotif.pl: Wed Sep 24 18:28:55 2008: DEBUG: emrepnotif: emConsoleMode=STANDALONE jobLike=EMD_MAINTENANCE%
emrepnotif.pl: Wed Sep 24 18:28:55 2008: DEBUG: emrepnotif: , /tmp/_emrepnotif
emrepnotif.pl: Wed Sep 24 18:28:55 2008: DEBUG: emrepnotif: sql is: SELECT
(SELECT count(broken) FROM user_jobs
WHERE what LIKE('EMD_MAINTENANCE%')
AND broken = 'Y'),
(SELECT MIN(SYSDATE-next_date) FROM user_jobs
WHERE what LIKE('EMD_MAINTENANCE%')),
NVL((SELECT AVG(value) FROM mgmt_system_performance_log
WHERE job_name like('EMD_MAINTENANCE%')
AND name='Queued Notifications'
AND time>(SYSDATE-(1/24))),0),
(SELECT DECODE(COUNT(a.device_name), 0, -1, COUNT(a.device_name)) - COUNT(b.device_name)
FROM mgmt_notify_devices a, mgmt_notify_devices b
WHERE b.status = 0) FROM DUAL
emrepnotif.pl: Wed Sep 24 18:28:55 2008: DEBUG: emrepnotif: query result is:
em_result=0|0|-1

emrepnotif.pl: Wed Sep 24 18:28:55 2008: DEBUG: emrepnotif: Time in emrepnotif: 61.2359046936035
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: Connectdescriptor (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=rac1.colestock.test)(PORT=1521)))(CONNECT_DATA=(SID=emrep)))
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: , /tmp/_emrepresp
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: sql is: select count(distinct host_url) from mgmt_failover_table where sysdate-last_time_stamp < 300
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: sql1 is: SELECT
(SELECT count(broken) FROM user_jobs
WHERE what LIKE('EMD_COLLECTION.%')
AND broken = 'Y'),
(SELECT MIN(SYSDATE-next_date) FROM user_jobs
WHERE what LIKE('EMD_COLLECTION.%')) FROM DUAL
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: No active OMSs
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: exists=, accesstime=, interval=0.0416666666666667 mailscriptexists=
emrepresp.pl: Wed Sep 24 18:29:15 2008: ERROR: emrepresp: processfailure /u01/app/oracle/product/agent10g/bin/emrepdown.pl, Message:No active Management Services were found, Subject:Severe Enterprise Manager problem
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: opened /tmp/sysman1234_emrepdown
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: list command1=cat /u01/app/oracle/product/agent10g/sysman/config/emd.properties | grep -i EMD_EMAIL_ADDRESS= | sed s?EMD_EMAIL_ADDRESS=??i | awk '{print }'
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: return command1=cat /u01/app/oracle/product/agent10g/sysman/config/emd.properties | grep -i EMD_FROM_EMAIL_ADDRESS= | sed s?EMD_FROM_EMAIL_ADDRESS=??i | awk '{print }'
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: list=oracle@rac1.colestock.test,james@colestock.com
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: return=
emrepresp.pl: Wed Sep 24 18:29:15 2008: DEBUG: command1=`mailx -s "Severe Enterprise Manager problem" oracle@rac1.colestock.test,james@colestock.com < /tmp/sysman1234_emrepdown`
emrepresp.pl: Wed Sep 24 18:29:16 2008: DEBUG: out=
emrepresp.pl: Wed Sep 24 18:29:16 2008: DEBUG: emrepresp: Time in emrepresp: 1205.62410354614

The referenced perl scripts are the ones involved in checking the status of the Grid Control installation and sending the email

If everything works, you should receive an email similar to the following

From oracle@colestock.com  Wed Sep 24 18:29:16 2008
Return-Path: 
Received: from rac1.colestock.test (localhost.colestock.test [127.0.0.1])
        by rac1.colestock.test (8.13.1/8.13.1) with ESMTP id m8P0TFTp006370;
        Wed, 24 Sep 2008 18:29:16 -0600
Received: (from oracle@localhost)
        by rac1.colestock.test (8.13.1/8.13.1/Submit) id m8P0TFpN006368;
        Wed, 24 Sep 2008 18:29:15 -0600
Date: Wed, 24 Sep 2008 18:29:15 -0600
From: Oracle Software Owner 
Message-Id: <200809250029.m8P0TFpN006368@rac1.colestock.test>
To: oracle@rac1.colestock.test, james@colestock.com
Subject: Severe Enterprise Manager problem


Wed Sep 24 18:29:15 MDT 2008
Severe Enterprise Manager problem
Error message: No active Management Services were found

For more help, refer to this Metalink article: 429257.1

Labels: Grid Control, Monitoring

How to Install Enterprise Manager 10g (Grid Control) - 10.2.0.4 - on Linux

Posted at Tuesday, April 15, 2008

In the following example, I install Enterprise Manager (Grid Control)- 10.2.0.4 - against an existing 10.2.0.3 RAC database on Linux x86.

In order to accomplish this, the base version of Enterprise Manager needs to be installed - 10.2.0.1 - and then patched as appropriate.

Needless to say, Grid Control is not extremely stable and has many bugs. Accordingly, the installation is a little more detailed and prone to error. Be sure that you pay careful attention to prerequisites, etc. for your environment (reference documents listed below). I won't go into all the O/S requirements here - it is assumed that things like semaphores and file descriptor limits, etc. have been attended to already.

Download the base release and patchset Enterprise Manager Software

Install the base version - 10.2.0.1 - of Enterprise Manager

Perform prerequisite checks via Universal Installer (as oracle)

# /u02/oracle/software/oms/install/runInstaller -prereqchecker PREREQ_CONFIG_LOCATION=/u02/oracle/software/oms/rdbms/Disk1/stage/prereq -entryPoint oracle.sysman.top.oms_Core -prereqLogLoc /home/oracle -silent

# /u02/oracle/software/oms/install/runInstaller -prereqchecker PREREQ_CONFIG_LOCATION=/u02/oracle/software/oms/rdbms/Disk1/stage/prereq -entryPoint oracle.sysman.top.em_seed_Core -prereqLogLoc /home/oracle -silent

Prepare the Database for the install

FGA must be installed:

SQL> select value from v$option where parameter = 'Fine-grained access control';

VALUE
----------------------------------------------------------------
TRUE

DBMS_SHARED_POOL must be installed:

SQL> select count(*) from DBA_OBJECTS where object_name='DBMS_SHARED_POOL'
  2  ;

  COUNT(*)
----------
         0

SQL> @$ORACLE_HOME/rdbms/admin/dbmspool.sql

Package created.


Grant succeeded.


View created.


Package body created.

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql

SQL> select count(*) from DBA_OBJECTS where object_name='DBMS_SHARED_POOL'
  2  ;

  COUNT(*)
----------
         2

Prepare the environment for Installation

Update /etc/oratab:

# echo oms10g:$ORACLE_BASE/product/oms10g:N >> /etc/oratab
# echo agent10g:$ORACLE_BASE/product/agent10g:N >> /etc/oratab

Set the environment:

# export ORACLE_SID=oms10g
# . oraenv
# export DISPLAY=192.168.1.2:0.0
# export TMP=/tmp
# export TEMP=/tmp

Preform the Installation

# ./runInstaller

Choose 'Using an Existing Database' option

Specify Parent Location

Verify the prerequisite checks. The failures for compat* libraries can be ignored on Linux

Specify Database Connection information, including file locations for Enterprise Manager's tablespaces

In response, unset dispatchers parameter and click 'Retry':

SQL> alter system set dispatchers='' scope=both sid='*';

System altered.

In response, set session_cached_cursors and aq_tm_processes parameters upward and click 'OK':

SQL> alter system set session_cached_cursors=200 scope=spfile sid='*';

System altered.

SQL> alter system set aq_tm_processes=4 scope=spfile sid='*';

System altered.

Cycle the database for the changes to take effect:

# srvctl stop database -d em
# srvctl start database -d em

In response, set sga_target and sga_max_size parameters upward and click 'OK':

SQL> alter system set sga_target=256M scope=spfile sid='*';

System altered.

SQL> alter system set sga_max_size=256M scope=spfile sid='*';

System altered.

Once again, cycle the database for the changes to take effect:

# srvctl stop database -d em
# srvctl start database -d em

Specify Optional Configuration Information

Provide Security Passwords

Review the Summary Screen, click 'Install'

When prompted run the allroot.sh

# cd /u01/app/oracle/product/oms10g
# ./allroot.sh

Then Click 'OK'

Upon successful completion, test Grid Control to ensure that it is working

Configure Enterprise Management Tablespaces per your requirements:

SQL> alter tablespace MGMT_TABLESPACE read only;

Tablespace altered.

SQL> alter tablespace MGMT_TABLESPACE offline;

Tablespace altered.

SQL> ! mv /u02/oracle/ora1/em/mgmt.dbf /u02/oracle/ora2/em/mgmt.dbf

SQL> alter database rename file '/u02/oracle/ora1/em/mgmt.dbf' to '/u02/oracle/ora2/em/mgmt.dbf';

Database altered.

SQL> alter tablespace MGMT_TABLESPACE online;

Tablespace altered.

SQL> alter tablespace MGMT_TABLESPACE read write;

Tablespace altered.

SQL> alter database datafile '/u02/oracle/ora2/em/mgmt.dbf' resize 10G;

Database altered.

SQL> alter database datafile '/u02/oracle/ora2/em/mgmt.dbf' autoextend off;

Database altered.

Patch the Enterprise Manager installation to 10.2.0.4 (Patch #3731593)

Stop Services in Preparation for Patching:

# export ORACLE_SID=oms10g
# . oraenv
# $ORACLE_HOME/bin/emctl stop oms
Oracle Enterprise Manager 10g Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
Oracle Management Server is Down.

# $ORACLE_HOME/bin/emctl stop iasconsole
Oracle Enterprise Manager 10g Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.

Stopping Oracle 10g Application Server Control ...  ...  Stopped.

# $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...

# export AGENT_HOME=$ORACLE_BASE/product/agent10g
# $AGENT_HOME/bin/emctl stop agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

Patch the Repository and Management Service to 10.2.0.4:

# export DISPLAY=192.168.1.2:0.0
# /u02/oracle/software/3731593/Disk1/runInstaller &

Specify Software Home to Patch

Note the Provisioning Archive Message

Set up the Software Library

Navigate to the Software Library Configuration Page and specify a valid directory

Afterwords, run the following command:

# export ORACLE_SID=oms10g
# . oraenv
# /u01/app/oracle/product/oms10g/bin/PARDeploy -action deploy -parDir /u01/app/oracle/product/oms10g/sysman/prov/paf -force

Secure the OMS

# export ORACLE_SID=oms10g
# . oraenv
# emctl status emkey
Oracle Enterprise Manager 10g Release 4 Grid Control
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Please enter repository password:

The Em Key  is configured properly, but is not secure. Secure the Em Key by running "emctl config emkey -remove_from_repos".

# emctl config emkey -remove_from_repos
Oracle Enterprise Manager 10g Release 4 Grid Control
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Please enter repository password:

The Em Key has been removed from the Management Repository.
Make a backup copy of OH/sysman/config/emkey.ora file and store it on another machine.
WARNING: Encrypted data in Enterprise Manager will become unusable if the emkey.ora file is lost or corrupted.

# emctl status emkey
Oracle Enterprise Manager 10g Release 4 Grid Control
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Please enter repository password:

The Em Key is configured properly.

Upgrade the Management Agent on the OMS Node

# export ORACLE_SID=agent10g
# . oraenv
# which emctl
/u01/app/oracle/product/agent10g/bin/emctl
# emctl stop agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
Agent is Not Running

Run the installer, just as you did for patching the OMS:

# export DISPLAY=192.168.1.2:0.0
# /u02/oracle/software/3731593/Disk1/runInstaller &

You will see screens similar to that of the last install; simply follow the prompts as you did before.

Secure the agent:

# emctl secure agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Enter Agent Registration password :
Agent successfully stopped...   Done.
Securing agent...   Started.
Requesting an HTTPS Upload URL from the OMS...   Done.
Requesting an Oracle Wallet and Agent Key from the OMS...   Done.
Check if HTTPS Upload URL is accessible from the agent...   Done.
Configuring Agent for HTTPS in CENTRAL_AGENT mode...   Done.
EMD_URL set in /u01/app/oracle/product/agent10g/sysman/config/emd.properties
publicKeyFile = /u01/app/oracle/product/agent10g/sysman/config/nmosudo.props
Generating RSA keys.

Saving private key in wallet.
SaveKeyWallet: Cannot open wallet (error=28759): file:/u01/app/oracle/product/agent10g/sysman/config
SaveKeyWallet: Re-creating wallet.

Reading private key from wallet.

Saving public key into file.

Reading public key from file.

Encrypting clear-text: gensudoprops pass.

Decrypting encrypted-text.
Decrypted Text: gensudoprops pass
Securing agent...   Successful.
Agent successfully restarted...   Done.

Reconfigure the agent:

# agentca -f

Stopping the agent using /u01/app/oracle/product/agent10g/bin/emctl  stop agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.
Running agentca using /u01/app/oracle/product/agent10g/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/oracle/product/agent10g ACTION=Configure MODE=Perform RESPONSE_FILE=/u01/app/oracle/product/agent10g/response_file RERUN=TRUE INV_PTR_LOC=/etc/oraInst.loc COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform - mode is starting for action: Configure


Perform - mode finished for action: Configure

You can see the log file: /u01/app/oracle/product/agent10g/cfgtoollogs/oui/configActions2008-04-17_06-54-10-PM.log

# emctl upload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

# emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.2.0.4.0
OMS Version       : 10.2.0.4.0
Protocol Version  : 10.2.0.4.0
Agent Home        : /u01/app/oracle/product/agent10g
Agent binaries    : /u01/app/oracle/product/agent10g
Agent Process ID  : 12778
Parent Process ID : 12736
Agent URL         : https://rac2.colestock.test:3872/emd/main/
Repository URL    : https://rac2.colestock.test:1159/em/upload
Started at        : 2008-04-17 18:59:20
Started by user   : oracle
Last Reload       : 2008-04-17 19:01:24
Last successful upload                       : 2008-04-17 19:01:45
Total Megabytes of XML files uploaded so far :    10.19
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    74.73%
Last successful heartbeat to OMS             : 2008-04-17 19:01:27
---------------------------------------------------------------
Agent is Running and Ready

Remember to configure any applicable targets so that you can monitor the target via Grid Control. You will first have to unlock the dbsnmp user for databases (if not done already).

SQL> alter user dbsnmp account unlock;

User altered.

SQL> alter user dbsnmp identified by "password";

User altered.

At this point your Grid Control Installation is ready for use.

Optionally, Setup Mass Deployment Agent

Download the Mass Deployment Agent Software from Oracle

Stage the Agent Software on the Management Service

# export ORACLE_SID=oms10g
# . oraenv
# echo $ORACLE_HOME
/u01/app/oracle/product/oms10g
# mkdir -p $ORACLE_HOME/sysman/agent_download/10.2.0.4.0

# unzip /u02/oracle/software/Linux_Grid_Control_agent_download_10_2_0_4_0.zip -d /u01/app/oracle/product/oms10g/sysman/agent_download/10.2.0.4.0

Edit the agentDownload script, specifying the appropriate values for the following variables

vi /u01/app/oracle/product/oms10g/sysman/agent_download/10.2.0.4.0/linux/agentDownload.linux

# Variables instantiated during the OMS install
# Machine hosting the OMS
OMShost=rac2.colestock.test
# Default HTTP port for EM console
httpPort=4889

Test Installing the Agent via the Mass Deployment Agent

# echo agent10g:$ORACLE_BASE/product/agent10g:N >> /etc/oratab

# export ORACLE_SID=agent10g
# . oraenv

# export AGENT_INSTALL_PASSWORD=password

# scp -p rac2.colestock.test:/u01/app/oracle/product/oms10g/sysman/agent_download/10.2.0.4.0/linux/agentDownload.linux .

The authenticity of host 'rac2.colestock.test (192.168.1.102)' can't be established.
RSA key fingerprint is 1e:7c:5c:fa:da:29:f0:62:cd:30:0a:57:67:85:f1:77.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac2.colestock.test' (RSA) to the list of known hosts.
oracle@rac2.colestock.test's password:
agentDownload.linux                           100%   28KB  27.9KB/s   00:00

# chmod u+x agentDownload.linux

 ./agentDownload.linux -b $ORACLE_BASE/product -dt

 su - root
Password:
[root@rac1 ~]#  /u01/app/oracle/product/agent10g/root.sh

Start and Test the Agent

[oracle@rac1 tmp]$ emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
Starting agent ...... started.

# emctl upload agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

# emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.2.0.4.0
OMS Version       : 10.2.0.4.0
Protocol Version  : 10.2.0.4.0
Agent Home        : /u01/app/oracle/product/agent10g
Agent binaries    : /u01/app/oracle/product/agent10g
Agent Process ID  : 15529
Parent Process ID : 15478
Agent URL         : https://rac1.colestock.test:3872/emd/main/
Repository URL    : https://rac2.colestock.test:1159/em/upload
Started at        : 2008-04-17 19:58:33
Started by user   : oracle
Last Reload       : 2008-04-17 19:58:33
Last successful upload                       : 2008-04-17 19:58:55
Total Megabytes of XML files uploaded so far :     0.67
Number of XML files pending upload           :        3
Size of XML files pending upload(MB)         :     2.02
Available disk space on upload filesystem    :    87.75%
Last successful heartbeat to OMS             : 2008-04-17 19:58:37
---------------------------------------------------------------
Agent is Running and Ready