The typical troubleshooting scenario we like to get as much information as we can. However, we have to balance that with the cost of collecting the information. For example, in a large SOA installation with several hindered or perhaps millions of composite instances, setting the console to track the ‘state’ of those instances is abusive and time consuming. One should only enable the composite instance state monitoring in the SOA console if that information is actually more valuable than the delay in getting the information. Collecting this and other ‘optional’ information can bring the OEM console to its knees.

The guidance: Only collect the information that you are currently and actively using

Option: use a different channel to collect the same information. Perhaps a separate query directly to the database will reveal the desired information without crushing the console’s responsiveness.

Capture Composite Instance State

Also note the very explicit (and counter-intuitive) warnings about how the console displays the state information (highlighted in bold/red)

Reference:

http://download.oracle.com/docs/cd/E12839_01/integration.1111/e10226/soainfra_config.htm#BHCHBAIA

…
input archive log thread=2 sequence=46274 recid=166663 stamp=714935844
input archive log thread=2 sequence=46275 recid=166665 stamp=714935889
input archive log thread=2 sequence=46276 recid=166667 stamp=714939059
input archive log thread=2 sequence=46277 recid=166668 stamp=714939083
channel ch0: starting piece 1 at 29-MAR-10
released channel: ch0
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch0 channel at 03/29/2010 18:27:07
ORA-19506: failed to create sequential file, name=”r2l9q6n6_1_1″, parms=”"
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text:
VxBSACreateObject: Failed with error:
Server Status: Communication with the server has not been iniatated or the server status has not been retrieved from the server.

The fix was to remove the client identification from the rman script.

Got this error in the emoms.trc log and thousands of .xml files in the recv directory.
2009-12-22 18:13:37,659 [XMLLoader0 30000031529.xml] ERROR emdrep.XMLLoaderContext flushTableStatements.1587 – Error while flushing insert statementjava.sql.SQLException: ORA-14400: inserted partition key does not map to any partition

The fix was found on metalink as [ID 317046.1] Problem: Many Ora-14400 Errors Showing In Management Services And Repository Ora-14400 [ID 317046.1]

SQL> exec emd_maintenance.analyze_emd_schema('SYSMAN');

PL/SQL procedure successfully completed.
Also make sure your job_queue_processes is set to 10 or above.
Hope that helps

While upgrading OEM from 10.2.0.4 to 10.2.0.5 I got the following error:

[Text: Required RDBMS package: DBMS_AQ doesnt exist on this database. This package is needed for the proper functioning of the EM product. Please contact Oracle support to diagnose the RDBMS and once resolved, retry EM install.]

This was caused by the sys password not being the same on all instances. The root cause was that a DBA had changed the sys password on a single instance to do some maintenance and did not change it back. This caused sys connections to the rac database to succeed and fail intermittently.

The fix was to resync the sys password by running the following from sqlplus as the sysdba:
SQL> alter user sys identified by “correct_password”;
Once the passwords were correctly synced, the upgrade ran fine.

The TNS_ADMIN environment is used by the installation to find the location of the tnsnames.ora and the listener.ora files…

Default location

The default location of the tnsnames.ora and listener.ora files are in $ORACLE_HOME/network/admin. If you want to change the location you have to show Oracle where to find the files. This is done by setting the TNS_ADMIN environment settings as follows:

To setup

Many people run Oracle RMAN scripts from crontab and an OS shell script. We prefer to run all of our RMAN backups via OEM and the database scheduler instead of from a specific server or instance. There are many benefits OEM/database scheduler backups a few of which are:

• Centralized administration makes it easy to manage 100′s of databases
• Database affinity means that backups run even if a server crashes
• OEM monitors the output of the RMAN and can alert the DBA if there are errors

One thing that we have noticed is that when we used our OS (server) RMAN shell scripts we had a string similar to the following:

send ‘NB_ORA_CLIENT=gmrac1

This parameter is optional for Networker and is used to assist Networker with recovery operations to alternate servers.

Occasionally, the RMAN would fail for no immediately obvious reason. On closer inspection we saw Networker was timing out while connecting to the target server. Everytime the database RMAN job was launched from gmrac1 it would succeed. However when it was run from any other server in the cluster the job would fail with the following:

RMAN-03009: failure of backup command on ch0 channel at 02/05/2009 16:51:46

ORA-19506: failed to create sequential file, name=”mqk6k505_1_1″, parms=”"
ORA-27028: skgfqcre: sbtbackup returned error
ORA-19511: Error received from media manager layer, error text: VxBSACreateObject: Failed with error: Server Status: Communication with the server has not been iniatated or the server status has not been retrieved from the server.

[sic]

The fix was to remove the optional parameter NB_ORA_CLIENT=gmrac1 and let Networker do the right thing. For reference, our broken RMAN script is shown below:

run {
sql ‘alter system archive log current’;
allocate channel ch0 type ‘SBT_TAPE’;
send ‘NB_ORA_CLIENT=gmrac1,NB_ORA_POLICY=PROD_ora,
NB_ORA_SERV=backup.betterlogic.com’;
backup
tag “prod_archivelogs_only”
archivelog all not backed up;
delete noprompt archivelog until time ‘sysdate -2′ backed up 1 times to ‘SBT_TAPE’;
release channel ch0;}

I liked this book. It had a lot of good ideas for a co-operative leader, one that people want to follow and receive inspiration from.

Wonderful, must-read article with a super quote.

“The ultimate result of shielding men from the effects of folly is to fill the world with fools.” – Herbert Spencer

I am encouraging everyone to work harder and smarter. Efficiency of output and execution will restore profits and save jobs. Working smarter is only one piece of the solution.

The link below summarizes the problem. This article is a must read for every working class citizen who thinks we can just let someone else drive the country.
http://www.minyanville.com/articles/Bernanke-Paulson-Fed-Credit-crunch-recession/index/a/19357

STTY

In the .profile or .bashrc add the following lines:

stty ek
stty erase ^H
EDITOR=vi
export EDITOR

Before typing ^H or ^? type ^V

EXECUTE DBMS_MONITOR.SERV_MOD_ACT_TRACE_ENABLE(service_name => ‘betterlogic.com’, waits => TRUE, binds => TRUE, instance_name => ‘bpel1′);

select * from dba_enabled_traces;

EXECUTE DBMS_MONITOR.SERV_MOD_ACT_TRACE_DISABLE(service_name => ‘cmprod.colo.cmarket.com’, instance_name => ‘bpel1′);

oracle@rac34 udump]$ trcsess output=”dump.file” service=”betterlogic.com” module=”JDBC Thin Client”

Oracle forms compile failed with “Out of file descriptors”.

Fix: projmod -s -K “process.max-file-descriptor=(basic,10000,deny)” oracle

Reference: IBM Resource Limit Configuration

testing post of image

this is a test

The disk mount point was cleared to get ready for 300G disk migration.

Dellâ„¢ PowerEdgeâ„¢ 2900

Systems Hardware Owner’s Manual

EM can automatically connect to Metalink to check for required patches for each target database and download them to a software library.

Over the next few months I will add monitoring, alerts, admin accounts, developer user accounts (so they can get their own ADDM (statspack) results), and groups. I will also work to add every db to the Grid.

This is just the start. EM is pretty cool and I expect we will be able to:

* Increase uptime and proactive management
* Automate the instance reporting (for licensing contract purposes)
* Improve general and custom reporting for all instances
* Centralize much of the management
* Compare database settings and schemas between two databases or between a stored baseline and any database
* Create new users in multiple databases using any other user as a starting template
* Handle configuration management
* Assist with change control
* Keep configuration histories
* Make it simple to answer questions like:
o “What was the database doing yesterday at 2:22PM?”
o “When was this database last backed up?”
o “Are their any blocking locks?”
o “What are the top SQL queries?”, “What are the top five wait events?”
o “What hosts are connected to my database?” (i.e. are development boxed connecting to production databases)
o “What is the explain plan for this query?”
* Job queuing and command execution library

More to follow…

-Rick

Management Commands

Commands for managing project attributes include the following:

• projects: Displays project memberships for users, lists projects from the project database, prings information on given projects.
• newtask: Executes the shell or command in a new task in the current project.
• projadd: Adds a new entry to the /etc/project entry.
• projmod: Modifies information for a project in /etc/project.
• projdel: Deletes a project from /etc/project.
• rctladm: Displays/modifies global state of active resource controls, sets logging or actions.
• prctl: Displays/modifies local resource controls.
• ipcs: Identifies which IPC objects are being used in a project.
• rcapadm: Manages rcapd memory-capping daemon.
• prstat -J: Displays resource consumption on a per-project basis
• priocntl -i project-name: Sets/displays scheduling parameters of the project.
• poolbind -i project-name: Assigns a project to a resource pool.

Reference

Recovery example
SQL> select status, enabled, online_time, name from v$datafile order by name;

RMAN>