Oracle Enterprise Manager – Reducing the noise, Part 1

Enterprise Manager 12c is a great monitoring tool, with it you can monitor a wide range of target types from databases to middleware; although out-of-the-box metrics can suit your monitoring requirements they can generate a considerable amount of white noise. In order to reduce this noise first you have to identify which are the top alerts in your system; Cloud Control comes with several predefined reports that help you to dig into multiple areas of your system, there’s a report “20 Most Common Alerts” which shows you the incidence of common alerts.

In the picture above, you can clearly see that metric “Database Time Spent Waiting (%)” appears twice in my Top 3, let’s find out our metric setting for my DB targets; in order to do this we must go to a DB home page then Oracle Database -> Monitoring -> Metrics and Collection Settings.  

 

Wait a minute! Why I’m receiving alerts if there are no thresholds setup for any of those metrics?, this behavior is clearly explained in MOS note 1500074.1 about a default warning threshold of 30% inside the database configuration. Let’s take a look to dba_threshold to confirm.

set lines 300

column METRICS_NAME format a30

column WARNING_OPERATOR format a30

column WARNING_VALUE format a30

column CRITICAL_OPERATOR format a30

column CRITICAL_VALUE format a30

SELECT METRICS_NAME,WARNING_OPERATOR ,WARNING_VALUE,CRITICAL_OPERATOR ,CRITICAL_VALUE FROM DBA_THRESHOLDS;

METRICS_NAME                        WARNING_OPERATOR               WARNING_VALUE                  CRITICAL_OPERATOR              CRITICAL_VALUE

----------------------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             30                             NONE

Average Users Waiting Counts        GT                             30                             NONE

Blocked User Session Count          GT                             0                              NONE

Current Open Cursors Count          GT                             1200                           NONE

Database Time Spent Waiting (%)     GT                             30                             NONE

Database Time Spent Waiting (%)     GT                             30                             NONE

Database Time Spent Waiting (%)     GT                             30                             NONE

Database Time Spent Waiting (%)     GT                             30                             NONE

Database Time Spent Waiting (%)     GT                             30                             NONE

Database Time Spent Waiting (%)     GT                             30                             NONE

Database Time Spent Waiting (%)     GT                             50                             NONE

Database Time Spent Waiting (%)     GT                             50                             NONE

Logons Per Sec                      GE                             100                            NONE

Session Limit %                     GT                             90                             GT                             97

Tablespace Bytes Space Usage        DO NOT CHECK                   0                              DO_NOT_CHECK                   0

Tablespace Space Usage              GE                             85                             GE                             97

22 rows selected.

There you go!, all metrics for “Database Time Spent Waiting (%)” are set to 30% or 50% values, now the trick to disable these metrics is to set them to a different value like 99%; this will override the default value as follows:

  

Let’s look at the database setting again:

set lines 300

column METRICS_NAME format a30

column WARNING_OPERATOR format a30

column WARNING_VALUE format a30

column CRITICAL_OPERATOR format a30

column CRITICAL_VALUE format a30

METRICS_NAME                        WARNING_OPERATOR               WARNING_VALUE                  CRITICAL_OPERATOR              CRITICAL_VALUE

----------------------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             30                             NONE

Average Users Waiting Counts        GT                             30                             NONE

Blocked User Session Count          GT                             0                              NONE

Current Open Cursors Count          GT                             1200                           NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Database Time Spent Waiting (%)     GT                             99                             NONE

Logons Per Sec                      GE                             100                            NONE

Session Limit %                     GT                             90                             GT                             97

Tablespace Bytes Space Usage        DO NOT CHECK                   0                              DO_NOT_CHECK                   0

Tablespace Space Usage              GE                             85                             GE                             97

25 rows selected.

We successfully modified these metrics to a very high value; at this point you can decide to stay at 99% or you can remove that threshold in order to completely disable them.

Now let’s confirm those settings in the database:

set lines 300

column METRICS_NAME format a30

column WARNING_OPERATOR format a30

column WARNING_VALUE format a30

column CRITICAL_OPERATOR format a30

column CRITICAL_VALUE format a30

METRICS_NAME                        WARNING_OPERATOR               WARNING_VALUE                  CRITICAL_OPERATOR              CRITICAL_VALUE

----------------------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             10                             NONE

Average Users Waiting Counts        GT                             30                             NONE

Average Users Waiting Counts        GT                             30                             NONE

Blocked User Session Count          GT                             0                              NONE

Current Open Cursors Count          GT                             1200                           NONE

Logons Per Sec                      GE                             100                            NONE

Session Limit %                     GT                             90                             GT                             97

Tablespace Bytes Space Usage        DO NOT CHECK                   0                              DO_NOT_CHECK                   0

Tablespace Space Usage              GE                             85                             GE                             97

14 rows selected.

The metrics are not there anymore and hopefully the alerts neither. This behavior is also noted for “Average Users Waiting Counts” metric, if you are receiving considerable white noise for this metric you can disable as well following the same procedure. A good practice is to create a Monitoring template to help you modify these thresholds for multiple targets at once.

Stay tuned for my next post about reducing OEM 12c noise.

Thanks,

Alfredo