Unable to Extend OEM 13.3 Domain to Add an Additional OMS

While trying to add an additional OMS to the current OEM 13.3 GCDomain using the provisioning console there’s an error in the Install homes step.

The error contains the following message:

Caused by: oracle.sysman.oii.oiif.oiifb.OiifbEndIterateException:

oracle.sysman.oii.oiif.oiifb.OiifbEndIterateException: SEVERE:Values for the following variables could not

be obtained from the command line or response file(s):

MYORACLESUPPORT_USERNAME(MyOracleSupportUsername)

Silent install cannot continue.

This issue is related to Bug 28290139 and the workaround can be found in MOS 2432095.1

After performing the workaround the OMS addition job needs to be executed from scratch. This resolves the issue.

Thanks,

Alfredo

OEM 12c/13c Metric Extension - Collection Result Maximum Flood Control

I recently had an issue in OEM 12c while creating a Metric Extension that was returning more than 5000 records. In OEM 12c versions 12.1.0.4 and 12.1.0.5 there a flood control mechanism in order to limit the maximum number of rows for a metric.

You can also verify this in the agent_inst/sysman/log/gcagent.log file:

2018-10-08 13:38:50,330 [66:F119EDFE:GC.Executor.1 (host:hostname:ll_host_config) (host:hostname:ll_host_config:ECM_OS_COMPONENT)] WARN - Result set exceeding min flood control level

The solution is provided in the MOS Doc Id 1499381.1

We didn’t want to proceed to apply the patch but rather we limited the number of rows from the Metric Extension.

This OMS patch is available also for 13.1 version and the bug states that is fixed in 13.2 versions.

Thanks,

Alfredo

EBS ADOP Phase Failing Due to ORA-600 Error

There’s an issue in EBS 12.2 and database 12.1.0.2 versions related to Materialized Views. While executing ADOP (prepare, apply, finalize or cutover) phases, the cycle could fail due to an ORA-600 : internal error code, arguments: [kqllod:no stub for dependency parent] error.

After working with Oracle, they provided information about BUG 27883586 related to MView Refresh or Create while they are editioned.

The solution is to apply patch 27883586 to the database which is currently available for August 2017 release and then recreate the MViews. Oracle also released a merge patch on top if this (28820125).

If you need it for a different release, please contact Oracle to provide it.

Happy patching,

Alfredo

OEM 12c Agent Crashing on Solaris

I recently had an issue with an OEM agent 12.1.0.4 version running on Solaris. Looks like there’s a Java related bug which causes the agent to consume high amounts of CPU in the box. But the interesting part is this only happens when Listeners are being monitored. If you blackout the Listener targets this stops.

If this’s the case then you are hitting Bug 15953286.

The solution is comprised by various steps outlined on MOS “EM 12c: Enterprise Manager 12c Cloud Control Agent CPU Spiking and High Utilization on Solaris with Many Database and Listener Targets (Doc ID 1536871.1)”.

Thanks,

Alfredo

ORA-01555 query duration 0 seconds with Dataguard

How many times we have calls from users complaining about their process that failed due to an ORA-01555 error?

We know that if the queries are not well tuned and they modify a lot of data, the image held in the UNDO Tablespace could not be consistent with the real data. But have you ever seen this error right away after executing a SQL statement against a table?

I just did couple of days ago. Here’s the story:

ORA-01555 error appeared in the alertlog’s database with a query duration of 0 seconds.

ORA-01555 caused by SQL statement below (SQL ID: d3rt4tyudufeu, Query Duration=0 sec, SCN: 0x034f.34f660b4)

Any queries plus an analyze table failed right away with ORA-01555:

ERROR at line 1: ORA-0155: snapshot too old: rollback segment number 10 with name "SYSSMU11_1072300523734$" too small

So weird.

After researching a bit on MOS, found a note regarding a bug.

Some minutes later we also started to receive ORA-600 errors related so scn numbers.

ORA-error stack (00600[ktbdchk1: bad dscn])

The MOS note mentions the ORA-01555 and the ORA-600 errors as part of bug 22241601 with a Dataguard configuration. Is worth to mention that yes, we were doing switchover testing recently in this 12.1.0.2 environment.

ALERT Bug 22241601 ORA-600 [kdsgrp1] / ORA-1555 / ORA-600 [ktbdchk1: bad dscn] / ORA-600 [2663] due to Invalid Commit SCN in INDEX (Doc ID 1608167.1)

The solution is to apply the patch but there’s also a tested workaround that is to rebuild online all the indexes of that table.

Hope this helps.

Alfredo