OMS responded illegally [ERROR- Failed to Update Target Type Metadata] in 13c R2

I want to discuss about this OMS error coming from the agent’s heartbeat status. The agent shows with an unreachable status but is actually up and running.

$: emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.

---------------------------------------------------------------

Agent Version          : 13.2.0.0.0

OMS Version            : (unknown)

…

Last Reload            : (none)

Last successful upload                       : (none)

Last attempted upload                        : (none)

Total Megabytes of XML files uploaded so far : 0

Number of XML files pending upload           : 2,210

Size of XML files pending upload(MB)         : 2.89

Available disk space on upload filesystem    : 56.39%

Collection Status                            : Collections enabled

Heartbeat Status       : OMS responded illegally [ERROR- Failed to Update Target Type Metadata]

Last attempted heartbeat to OMS              : 2017-01-24 21:09:21

Last successful heartbeat to OMS             : (none)

Next scheduled heartbeat to OMS              : 2017-01-24 21:09:51

---------------------------------------------------------------

Agent is Running and Ready

In this case the agent is not able to upload the metadata because is different in the OMS.

This is mainly due to a patch in the agent side that is missing in the OMS. By mistake I applied the latest bundle patch to the agent before applying it to the OMS.

Just another proof that any patches have to be installed to the OMS first then the agents.

Thanks,

Alfredo

Upgrade OPatch for OEM 13c R2 Agents

Opatch utility has to be updated often as Oracle delivers regular updates to it. I was used to version 12 and it only required an overwrite of the OPatch directory. Well, not anymore with version 13c R2.

I will walk you through an OPatch upgrade from 13.8 to 13.9 version.

First export your agent ORACLE_HOME:

$: export ORACLE_HOME=/u01/oracle/oemagent/agent_13.2.0.0.0/

Verify the current OPatch version:

$: $ORACLE_HOME/OPatch/opatch version

OPatch Version: 13.8.0.0.0

OPatch succeeded.

Download and unzip the 13.9 version in a temporary directory:

$: unzip p6880880_139000_Generic.zip

Archive:  p6880880_139000_Generic.zip

   creating: 6880880/

  inflating: 6880880/README.txt

  inflating: 6880880/opatch_generic.jar

  inflating: 6880880/version.txt

As you can see, now we have some zip files and a java file. Execute java to upgrade OPatch as follows:

$: $ORACLE_HOME/jdk/bin/java -jar opatch_generic.jar -silent oracle_home=$ORACLE_HOME

Launcher log file is /tmp/OraInstall2017-01-19_08-47-21PM/launcher2017-01-19_08-47-21PM.log.

Extracting the installer . . . . Done

Checking if CPU speed is above 300 MHz.   Actual 3000.000 MHz    Passed

Checking swap space: must be greater than 512 MB.   Actual 16383 MB    Passed

Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)

Checking temp space: must be greater than 300 MB.   Actual 18842 MB    Passed

Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2017-01-19_08-47-21PM

Installation Summary

Disk Space : Required 6 MB, Available 5,229 MB

Feature Sets to Install:

        Next Generation Install Core 13.9.1.0.0

        OPatch 13.9.1.0.0

        OPatch Auto OPlan 13.9.1.0.0

Session log file is /tmp/OraInstall2017-01-19_08-47-21PM/install2017-01-19_08-47-21PM.log

Loading products list. Please wait.

 1%

 40%

Loading products. Please wait.

 43%

 …

 97%

 99%

Updating Libraries

Starting Installations

 1%

 …

 91%

 92%

Install pending

Installation in progress

 Component : oracle.swd.opatch 13.9.1.0.0

Copying files for 'oracle.swd.opatch 13.9.1.0.0 '

 Component : oracle.glcm.osys.core 13.9.1.0.0

Copying files for 'oracle.glcm.osys.core 13.9.1.0.0 '

 Component : oracle.glcm.oplan.core 13.9.1.0.0

Copying files for 'oracle.glcm.oplan.core 13.9.1.0.0 '

 Component : oracle.glcm.opatch.common.api 13.9.1.0.0

Copying files for 'oracle.glcm.opatch.common.api 13.9.1.0.0 '

 Component : oracle.glcm.opatchauto.core 13.9.1.0.0

Copying files for 'oracle.glcm.opatchauto.core 13.9.1.0.0 '

Install successful

Post feature install pending

Post Feature installing

 Feature Set : apache_commons_cli_lib

 Feature Set : oracle.glcm.opatchauto.core.actions.classpath

Post Feature installing 'apache_commons_cli_lib'

 Feature Set : oracle.glcm.opatchauto.core.binary.classpath

 Feature Set : oracle.glcm.osys.core.classpath

Post Feature installing 'oracle.glcm.opatchauto.core.binary.classpath'

Post Feature installing 'oracle.glcm.osys.core.classpath'

 Feature Set : apache_commons_compress_lib

 Feature Set : oracle.glcm.opatchauto.core.wallet.classpath

Post Feature installing 'oracle.glcm.opatchauto.core.actions.classpath'

Post Feature installing 'apache_commons_compress_lib'

 Feature Set : oracle.glcm.opatchauto.core.classpath

Post Feature installing 'oracle.glcm.opatchauto.core.wallet.classpath'

Post Feature installing 'oracle.glcm.opatchauto.core.classpath'

 Feature Set : oracle.glcm.opatch.common.api.classpath

 Feature Set : oracle.glcm.oplan.core.classpath

Post Feature installing 'oracle.glcm.opatch.common.api.classpath'

Post Feature installing 'oracle.glcm.oplan.core.classpath'

Post feature install complete

String substitutions pending

String substituting

 Component : oracle.swd.opatch 13.9.1.0.0

String substituting 'oracle.swd.opatch 13.9.1.0.0 '

 Component : oracle.glcm.osys.core 13.9.1.0.0

String substituting 'oracle.glcm.osys.core 13.9.1.0.0 '

 Component : oracle.glcm.oplan.core 13.9.1.0.0

String substituting 'oracle.glcm.oplan.core 13.9.1.0.0 '

 Component : oracle.glcm.opatch.common.api 13.9.1.0.0

String substituting 'oracle.glcm.opatch.common.api 13.9.1.0.0 '

 Component : oracle.glcm.opatchauto.core 13.9.1.0.0

String substituting 'oracle.glcm.opatchauto.core 13.9.1.0.0 '

String substitutions complete

Link pending

Linking in progress

 Component : oracle.swd.opatch 13.9.1.0.0

Linking 'oracle.swd.opatch 13.9.1.0.0 '

 Component : oracle.glcm.osys.core 13.9.1.0.0

Linking 'oracle.glcm.osys.core 13.9.1.0.0 '

 Component : oracle.glcm.oplan.core 13.9.1.0.0

Linking 'oracle.glcm.oplan.core 13.9.1.0.0 '

 Component : oracle.glcm.opatch.common.api 13.9.1.0.0

Linking 'oracle.glcm.opatch.common.api 13.9.1.0.0 '

 Component : oracle.glcm.opatchauto.core 13.9.1.0.0

Linking 'oracle.glcm.opatchauto.core 13.9.1.0.0 '

Linking in progress

Link successful

Setup pending

Setup in progress

 Component : oracle.swd.opatch 13.9.1.0.0

Setting up 'oracle.swd.opatch 13.9.1.0.0 '

 Component : oracle.glcm.osys.core 13.9.1.0.0

Setting up 'oracle.glcm.osys.core 13.9.1.0.0 '

 Component : oracle.glcm.oplan.core 13.9.1.0.0

Setting up 'oracle.glcm.oplan.core 13.9.1.0.0 '

 Component : oracle.glcm.opatch.common.api 13.9.1.0.0

Setting up 'oracle.glcm.opatch.common.api 13.9.1.0.0 '

 Component : oracle.glcm.opatchauto.core 13.9.1.0.0

Setting up 'oracle.glcm.opatchauto.core 13.9.1.0.0 '

Setup successful

Save inventory pending

Saving inventory

 93%

Saving inventory complete

 94%

Configuration complete

Logs successfully copied to /u01/app/oraInventory/logs.

Verify the new OPatch version

$: $ORACLE_HOME/OPatch/opatch version

OPatch Version: 13.9.1.0.0

OPatch succeeded. 

Thanks,

Alfredo

ORA-01503: CREATE CONTROLFILE failed on RMAN duplicate between RAC databases

It was while since I performed an RMAN duplicate from a RAC database to another RAC database. You may find this post useless as there’s a lot of information out there about the error I faced, but all my posts are kind of my own little diary.

So, basically. In order to successfully duplicate one database to another you may need this.

    - TNS setup on the server where the database to be duplicated is hosted. You need to be able to tnsping both, the target and the auxiliary database.

     - Create a parameter file and password file for the auxiliary and startup nomount the first instance.

    - Create a full backup of the target database including archivelogs.

    - Connect to rman to both the target and the auxiliary databases from the auxiliary server:

rman target sys/*****@<target_db> auxiliary /

    - Issue the duplicate command:

 duplicate target database to <auxiliary_db>;

This should successfully finish but I got this error:

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 01/03/2017 18:29:58

RMAN-05501: aborting duplication of target database

RMAN-06136: ORACLE error from auxiliary database: ORA-01503: CREATE CONTROLFILE failed

ORA-12720: operation requires database is in EXCLUSIVE mode

I searched this and the culprit is that my parameter file for the auxiliary had this:

cluster_database = true

By MOS note “Rman duplicate fail with RMAN-06136, ORA-01503, ORA-12720, ORA-00494 enqueue [CF] (Doc ID 1335479.1)”. Duplicate from a RAC database to another RAC database is not supported, hence we need to modify the parameter cluster_database to false in the auxiliary database.

After doing this the duplicate finished correctly.

Thanks,

Alfredo

OEM 13c New Features - Corrective Actions

Today’s post is about another new feature of Oracle Enterprise Manager (OEM) 13c. Well, not a new feature itself but some awesome enhancements to Corrective Actions (CA).

In this new release 13c several new CA types were added, including “Add Space To Tablespace”, “Chef Recipe” for those who like automation and now can be added to Service Level Agreement (SLA) alerts and job status events.

I’m a big enthusiast of OEM jobs and now having the option to set a CA for them is just more than cool!

I wrote an article a year ago on IOUG’s Select journal about Corrective Actions and didn’t want to finish my day without setting a CA for job. But wait a minute, I know I can set CA for metrics using the “Metric and Collection Settings” page, but there’s no such thing for jobs.

Here’s how we setup a corrective action for jobs. First we need to enable events for jobs, this means that we as administrators decide what kind of job status will create an event.

In order to do this, you need to navigate to Setup -> Incidents -> Job Events

Step 1. Here you decide what job status and job severity generates an event.

Step 2. You decide if you want to generate events for jobs without targets.

Step 3. Choose the target you want to generate job events for.

Once you have the proper events setup, then the next step is to catch this event and do something with it. In this case fire a Corrective Action (CA). The only place we can catch these events and do something with them is an Incident Rule.

Navigate to Setup -> Incident -> Incident Rules

Then create or modify an existing Incident Rule Set. Inside this Rule Set create a new Rule to catch these job events. Inside the Rule Actions there’s a new section to Submit a Corrective Action.

Click on the “Select corrective action” and choose the CA from your CA Library.

Cool doesn’t it! One of the uses I can think of is with Database backup jobs. Sometimes the RMAN backup fails when is not able to find an archivelog and the solution is to run a crosscheck command and retry the backup job. This CA can do that for us and just notify us once it completes.

It still needs further testing but is another tool to save us time.

If you want information about the “Add Space To Tablespace” CA you better check Kellyn’s post about it.

http://dbakevlar.com/2016/04/em13c-corrective-actions/

Thanks,

Alfredo