Quantcast
Channel: berxblog
Viewing all 172 articles
Browse latest View live

enabling Database Vault on e-business RAC database

$
0
0
Right now I'm in a process to setup Database Vault for an E-Business suite database. This is a 2 node RAC cluster.
The DB is 12.1.0.2 with April 17 BP.

As the DB exists already, I followed How To Enable Database Vault in a 12c database ? (Doc ID 2112167.1).
Everything looks smooth, but unfortunately, at the Configuration of DV
exec dvsys.configure_dv('DVOWNER','DVMANAGER');
failes with
ERROR at line 1: 
ORA-47500: Database Vault cannot be configured.
ORA-06512: at "DVSYS.CONFIGURE_DV", line 23
ORA-06512: at "DVSYS.CONFIGURE_DV", line 72
ORA-06512: at line 2

ORA-47500 tells
47500, 00000, "Database Vault cannot be configured."
// *Cause: The Database Vault configuration failed because some Database Vault
// objects were missing.
// *Action: Run the dvremov.sql script to remove Database Vault and then
// reinstall Database Vault.

This was the first try: remove DV and start again.
Nothing changed.


Some research on MOS led to Known Issues In A Database Vault Environment (Doc ID 2330775.1)
there the explanation for ORA-47500 is:

Cause: There are many invalid objects in the database.

Solution: Run $ORACLE_HOME/rdbms/admin/utlrp.sql to validate the invalid objects

So all invalid objects were removed/fixed and - DV enabled again.
With same error.


A lot more investigation and a SR was required here.

When catmac.sql was run it completed and didn't show any hints it failed at the end.
Especially register DB and it's validation were shown successful:
SQL> DECLARE
2 num number;
3 BEGIN
4 dbms_registry.loaded('DV');
5 SYS.validate_dv;
6 END;
7 /

PL/SQL procedure successfully completed.


After a lot more investigation, an issue in the (manually activated) spool of catmac.sql was found:
ERROR at line 1:
ORA-31011: XML parsing failed
ORA-19213: error occurred in XML processing at lines 49
LPX-00217: invalid character 21 (U+0015)
ORA-06512: at "DVSYS.DBMS_MACADM", line 160
ORA-06512: at line 2
and this error stack leads to Installation Of Database Vault Fails With ORA-19213, LPX-00217: Invalid Character 21 (U+0015) (Doc ID 2324871.1)
As so often: the problem looks obvious, but only after it was found.


The problem is the environment variable
ORA_NLS10=<ORACLE_HOME>/nls/data/9idata
Which is a requirement for Oracle E-business Suite.

Here RAC has it fair share of complexity as this environment is not (only) defined in a .bash_profile or similar, but also in cluster registry:
srvctl getenv database -d <DB_UNIQUE_NAME>
EBSSIB01_SITE1:
ORA_NLS10=<ORACLE_HOME>/nls/data/9idata

The fix was to disable this setting:
srvctl unsetenv database -d <DB_UNIQUE_NAME> -t ORA_NLS10
restart all instances
and now successfully enable DV at last!

Of course, afterwards ORA_NLS10 must be set again
srvctl setenv  database -d <DB_UNIQUE_NAME> -t "ORA_NLS10=<ORACLE_HOME>/nls/data/9idata"
and instances restarted once more.

missing oraagent.bin process

$
0
0
It all started with a small error when I tried to stop/start a service:

srvctl start service -d cmpp01_site1 -s cmpp01
PRCD-1084 : Failed to start service CMPP01 
PRCR-1079 : Failed to start resource ora.cmpp01_site1.cmpp01.svc
CRS-2680: Clean of 'ora.cmpp01_site1.cmpp01.svc' on 'av2l954p' failed
CRS-5802: Unable to start the agent process

It's not fine the agent can not be started. so let's check the logfile crsd_oraagent_oracle.trc:
2018-07-23 21:29:14.339 : CRSCOMM:2939654272:  Ipc: Starting send thread
2018-07-23 21:29:14.340 :GIPCXCPT:2939654272: gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)
2018-07-23 21:29:14.340 :GIPCXCPT:2939654272: gipcConnectSyncF [connectToServer : clsIpcClient.cpp : 380]: EXCEPTION[ ret gipcretConnectionRefused (29) ] failed sync connect endp 0x1cbb5b0 [0000000000000090] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_IPC_SOCKET_11)(GIPCID=00000000-00000000-0))', numPend 0, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj 0x1cbe170, sendp 0x1cbdf20 status 13flags 0xa108871a, flags-2 0x0, usrFlags 0x30000 }, addr 0x1cbc990 [0000000000000097] { gipcAddress : name 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_IPC_SOCKET_11)(GIPCID=00000000-00000000-0))', objFlags 0x0, addrFlags 0x4 }, flags 0x0
2018-07-23 21:29:14.340 : CRSCOMM:2939654272: IpcC: gipcConnect() failed, rc= 29
2018-07-23 21:29:14.340 : CRSCOMM:2939654272: [FFAIL] IpcC: Could not connect to (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_IPC_SOCKET_11)) ret = 29

It clearly says there is something going wrong, unfortunately it doesn't tell in detail what's going wrong or what's required.

Maybe the most beautiful but useless analysis here was provided by tfactl summary -crs:
DETAILS                            STATUS    COMPONENT  
+----------------------------------+---------+-----------+
.------------------------------. PROBLEM CRS
| CRS_SERVER_STATUS : ONLINE |
| CRS_STATE : ONLINE |
| CRS_INTEGRITY_CHECK : FAIL |
| CRS_RESOURCE_STATUS : ONLINE |
'------------------------------'
+----------------------------------+---------+-----------+

As neither My Oracle Support nor google helped a lot here, I started a random cry on twitter:


And Anil had mercy on me :
This was not the answer to my problem, but it started a very valuable conversation in which I learned enough to identify the issue.

It's important to know that (this) oraagent.bin process is started by crsd.bin, and it is restarted if it disappears for any reason.
So to see what's really going and and leads to the error above, running strace on crsd.bin and it's forks/clones:
strace -f -p <pid_of_crsd.bin> -o <outfile>

as expected crsd.bin creates a new process for oraagent.bin:
18631 clone( <unfinished ...>
...
18631 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe273dfe9d0) = 18633
...
18633 execve("/appl/grid/grid_12201/bin/oraagent.bin", ["/appl/grid/grid_12201/bin/oraage"...], [/* 52 vars */]) = 0

and if everything runs as expected the socket handling looks like
18633 socket(AF_LOCAL, SOCK_STREAM, 0 
...
18633 <... socket resumed> ) = 69 <== the creation of the socked
...
18633 connect(69, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sCRSD_IPC_SOCKET_11"}, 110 <unfinished ...>
...
18633 <... connect resumed> ) = 0 <== the successful connection of the sockets file descriptor 69 to /var/tmp/.oracle/sCRSD_IPC_SOCKET_11

But in my problematic case it's this:
22612 connect(69, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sCRSD_IPC_SOCKET_11"}, 110  <unfinished ...>
...
22612 <... connect resumed> ) = -1 ENOENT (No such file or directory)

The first finding here:
The error
gipcInternalConnectSync: failed sync request, ret gipcretConnectionRefused (29)
in the logfile correlates with
<... connect resumed> )           = -1 ENOENT (No such file or directory)

With this specific information a short check in /var/tmp/.oracle shows the missing file is not there.
A crosscheck in /proc/<pid_of_crsd.bin>/fd/ shows
lrwx------. 1 root root 64 Jul 26 22:00 /proc/2144/fd/214 -> /var/tmp/.oracle/sCRSD_IPC_SOCKET_11_lock (deleted)

this all happens on grid binaries 12.2 with latest patch set.


I showed now:
1) if oraagent.bin disappears, crsd.bin creates a new oraagent.bin process
2) the oraagent.bin process tries to connect a socket on /var/tmp/.oracle/sCRSD_IPC_SOCKET_11 to communicate with crsd.bin
3) if 2) fails, gipcretConnectionRefused (29) is shown in the logfile.



Right now I know:
If /var/tmp/.oracle/sCRSD_IPC_SOCKET_11 is missing but oraagent.bin does run, I can operate the cluster resources, but whenever oraagent.bin disappears, cluster resources can not be managed anymore.

If this is the only problem (and no other sockets in /var/tmp/.oracle are missing) I can restart crsd which will also restart oraagent.bin and everything works like expected.
crsctl stop res ora.crsd -init
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded

crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'node1'
CRS-2676: Start of 'ora.crsd' on 'node1' succeeded


My next questions to answer is
1) why does oraagent.bin disappear at all?
2) why does files in /var/tmp/.oracle disappear?

This is still not answered and under investigation.

unwanted cleanup in /var/tmp/.oracle

$
0
0
In my previous blog missing oraagent.bin process I raised a question:
why does files in /var/tmp/.oracle disappear? 
At least this question I am now able to answer.

By the help of Linux kernel's audit system and a proper rule:
-a always,exit -F arch=b64 -S rename,rmdir,unlink,unlinkat,renameat -F path=/var/tmp/.oracle -F key=ora.delete

I was able to see the culprit process:
----
type=PROCTITLE msg=audit(08/06/2018 17:13:48.382:58122) : proctitle=/usr/bin/systemd-tmpfiles --clean
type=PATH msg=audit(08/06/2018 17:13:48.382:58122) : item=1 name=sCRSD_IPC_SOCKET_11 inode=50924979 dev=00:25 mode=socket,777 ouid=root ogid=root rdev=00:00 nametype=DELETE
type=PATH msg=audit(08/06/2018 17:13:48.382:58122) : item=0 name=/ inode=45518 dev=00:25 mode=dir,sticky,777 ouid=root ogid=root rdev=00:00 nametype=PARENT
type=CWD msg=audit(08/06/2018 17:13:48.382:58122) : cwd=/
type=SYSCALL msg=audit(08/06/2018 17:13:48.382:58122) : arch=x86_64 syscall=unlinkat success=yes exit=0 a0=0x5 a1=0x556fb11fb17b a2=0x0 a3=0x3 items=2 ppid=1 pid=25307 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-tmpfile exe=/usr/bin/systemd-tmpfiles key=ora.delete
----

So it was a feature of systemd: systemd-tmpfiles.

In detail it comes from
/usr/lib/tmpfiles.d/tmp.conf 
this particular line:
v /var/tmp 1777 root root 30dy

The fix for this is easy: create a proper file (copy) in
/etc/tmpfiles.d
and add these lines at the end:
# Exclude /var/tmp/.oracle
x /var/tmp/.oracle
But as a research at MOS and google did not come up with any issue with systemd-tmpfiles and crsd.bin, oraagent.bin or any other keyword, something else must be a problem here as well.
This (and the other question from my previous post) are questions for further investigations.

So much for now.

RBAL process needs to release FORMER disk

$
0
0
Oracles Automatic Storage Management (ASM) has many nice advantages against other volume managers. One of my favorite is to add and remove or rebalance disks without any interruption for the database. Unfortunately the remove of devices after the rebalance is completed is not as straight forward as expected: 
In a current environment (Grid Infrastructure: 12.2.0.1.180417, rdbms: 12.1.0.2.180116 and others) after the ALTER DISKGROUP xxx DROP DISK yyy specific discs are in status FORMER
Due to  a specific schema of directories & symlinks it takes some effort to come to the real /dev/dm-xxx device. 
Still several RBAL processes of different DBs (I did not find a derterministic pattern which processes or DBs) hold a file handle on this specific device. To make it more interesting is the fact that the diskgroup (DG) or disk was never used by any of those DBs. We have a strict mapping of DBs to DGs, so it's easy for me to claim that. 

To tell RBAL processes to release this "stale" filehandle, I have to run a
  ALTER DISKGROUP zzz REBALANCE
Any DG does the trick, it is not required to be related to the DB of RBAL process at all. 
As diskgroups are normally balanced, this is a kind of NOP. Still it's required to wake up RBAL and makes it release unused filehandles. 

I still have no idea why these RBAL processes open a device it's DB never needs at all; 
but as I have my proper workaround, I don't investigate any further. 

Might this help someone out there, or at least me when I hit this odd behavior again ;-) 

TFA reports full dba_feature_usage_statistics to MyOracleSupport

$
0
0
Those who deal with MyOracleSupport (MOS) regularly will have noticed their demand for Trace File Analyzer (TFA) diagnostic collections.
Few will know they are also providing a full content of their DBs DBA_FEATURE_USAGE_STATISTICS view; at least when running the parameters
tfactl diagcollect -database <DB_UNIQUE_NAME>

The report calls tfa_home/resources/sql/db_feature_usage.sql and it's result is stored in the <hostname>.tfa_<timestamp>.zip file.

I did not find any mention about this behavior in the documentation, so it might be worth to post about it.

This is at least true for version 18.3.0.0.0 (latest version available at 2018-09-04)

seing your DB as it was some minutes ago

$
0
0
Last week during a discussion with a colleague we thought if it would make sense to have in SQLDeveloper the possibility to see the system "as it was some minutes ago".
Small errors can happen and also resource control isn't always perfect in every company.

Björn Rost suggested to use DBMS_FLASHBACK.ENABLE_AT_TIME:

A very clever suggestion!

Let's first look at the documentation:



DBMS_FLASHBACK


Using DBMS_FLASHBACK, you can flash back to a version of the database at a specified time or a specified system change number (SCN).

and in more detail:

DBMS_FLASHBACK Overview

DBMS_FLASHBACK provides an interface for the user to view the database at a particular time in the past, with the additional capacity provided by transaction back out features that allow for selective removal of the effects of individual transactions. This is different from a flashback database which moves the database back in time.
When DBMS_FLASHBACK is enabled, the user session uses the Flashback version of the database, and applications can execute against the Flashback version of the database.
DBMS_FLASHBACK is relevant only for the session in which it's called, so it doesn't change the database.
But it's unclear what's meant by database in this context.

Here a small testcase which shows some unexpected results of DBMS_FLASHBACK:
(I slightly edite the text for better readability)

drop table x1;
create table x1 (u number);

insert into x1 (u) values (1);

CREATE OR REPLACE FUNCTION "RETURN_SOMETHING" (
i NUMBER
) RETURN VARCHAR2 AS
-- Version 1
BEGIN
IF ( i = 1 ) THEN
RETURN 'red';
ELSE
RETURN 'blue';
END IF;
END return_something;
/


select to_char(sysdate, 'YYYY-MM-DD HH24:MI:SS') now, u, return_something(u) rrr,
dbms_flashback.get_system_change_number SCN
from x1;

NOW U RRR SCN
------------------- ---------- ----- ----------
2018-10-01 19:37:56 1 red 24042787


exec sys.DBMS_SESSION.sleep(60);
There should be nothing fancy up to this time.
The table and function are created and the select works fine.
I need the DBMS_SESSION.sleep to copy/paste the proper timestamp into the next block of code:

update x1 set u = 2 where u = 1;

1 row updated.

CREATE OR REPLACE FUNCTION "RETURN_SOMETHING" (
i NUMBER
) RETURN VARCHAR2 AS
-- Version 2
BEGIN
IF ( i = 1 ) THEN
RETURN 'oans';
ELSE
RETURN 'zwoa';
END IF;
END return_something;
/

select to_char(sysdate, 'YYYY-MM-DD HH24:MI:SS') now, u, return_something(u) rrr
from x1;

NOW U RRR
------------------- ---------- -----
2018-10-01 19:39:32 2 zwoa


Still nothing spectacular here - just the preparation done.
Now onto the interesting part:

EXEC dbms_flashback.enable_at_time(to_timestamp('2018-10-01 19:38:00','YYYY-MM-DD HH24:MI:SS'));


select to_char(sysdate, 'YYYY-MM-DD HH24:MI:SS') now, u, return_something(u) rrr
from x1;

NOW U RRR
------------------- ---------- -----
2018-10-01 19:39:33 1 oans



col text for A50
select text
from dba_source
where owner='BERX'
and name='RETURN_SOMETHING'
order by LINE asc;

TEXT
--------------------------------------------------
FUNCTION "RETURN_SOMETHING" (
i NUMBER
) RETURN VARCHAR2 AS
-- Version 1
BEGIN
IF ( i = 1 ) THEN
RETURN 'red';
ELSE
RETURN 'blue';
END IF;
END return_something;

11 rows selected.


Here we can see how DBMS_FLASHBACK.ENABLE_AT_TIME is set to a time between the 1stinsert& create function and the 2nd block.
With this setting, the content of table X1 is as expected. Also DBA_SOURCE shows the code of RETURN_SOMETHING.
But the function itself is not changed in memory and works as of it's state NOW, not at the given flashback time.
The flashback time version of RETURN_SOMETHING is even visible when you open it in SQLDeveloper (you have to believe me or test it).
Don't forget to clean up after the tests:

exec dbms_flashback.disable;

select to_char(sysdate, 'YYYY-MM-DD HH24:MI:SS') now, u, return_something(u) rrr
from x1;

NOW U RRR
------------------- ---------- -----
2018-10-01 19:39:33 2 zwoa


For this testcase no COMMIT was used. ;-)

SQL Real Time Monitoring pure HTML report (thanks to SQLDeveloper 18.3)

$
0
0
Yesterday (from the writing of this post perspective) SQLDeveloper 18.3 came out.
(it seems SQL Developer does not strict follow Oracle Database Release Number schema, otherwise it must be 18.4 already as it's released in 4th quarter of 2018)

There are many bugs fixed on 18.3 and some nice enhancements there as well.

My favorite enhancement is Real Time SQL Monitoring, HTML exports now available - no flash!

As Tanel Poder wrote about it already (Generate Oracle SQL Monitoring Reports as HTML using SQL Developer v18.3 (no Flash needed)) I'll show how you can use it without SQL Developer.
I'm not against SQL Developer, in fact I'm very happy with it!
Just the Real Time SQL Monitor tab seems kind of unresponsive to me, especially on DBs with many active sessions and high load.


So if I don't want to watch a java process drawing circles and showing blur previews, I first go to query v$sql_monitor. Beside all the columns I'll need to identify my SQL of interest, I need the columns SQL_ID, SQL_EXEC_ID, and SQL_EXEC_START.
And, of course at least once you must create a new HTML Report with SQL Developer! Save this report as a template for later use.

When you open the HTML report in an editor, there are 2 lines of special interest:
 var data_sqlId = '3v64dcg0rja6k'; 
var data_xml = '<report db_version="12.1.0.2.0...>"

You can replace them easily with the context of your SQL of interest.
To get the proper XML you can use DBMS_SQLTUNE.report_sql_monitor. But as this function provides a multiline XML, but the HTML expects the XML in one line, the call should be
SELECT XMLSerialize( DOCUMENT xmltype(DBMS_SQLTUNE.report_sql_monitor(
sql_id => '3v64dcg0rja6k',
type => 'XML',
SQL_EXEC_ID => 33554648,
SQL_EXEC_START => to_date('2018-10-10 05:36:17'),
report_level => 'ALL')) NO INDENT ) AS XML_report
FROM dual;
If you omit SQL_EXEC_ID or SQL_EXEC_START, the report will still compile bout you should understand the difference in data you get displayed.

So the minimal count you create a new HTML-Only SQL Real Time Monitoring report with SQL Developer should be once. But if you read this ( and your DBs are properly licensed), spread the good news with all the people who might benefit from these reports!


Please keep in mind that the HTML file is loading CSS and JS files from and URL at Oracles Content Delivery platform - so they are subject to any change without a new release of SQL Developer, RDBMS or anything. So expect unexpected changes every time!

preupgrade.jar - enforced recommendations

$
0
0


Recently I upgraded a database from version 12.1 to 12.2.
I take upgrades serious - similar (but not as skilled) to Mike Dietrich. So I downloaded latest preupgrade.zip and unzipped it [I made a small error by unzipping it NOT to $ORACLE_HOME/rdbms/admin but to a temporary directory - later more about this].
Beside such errors, I try to follow Oracles recommended Upgrade Process.

At some point Oracle recommends to Gather Dictionary Stats "the night before starting the upgrade".

For me this is a recommendation, not an obligation. I had (from my perspective) very good reasons NOT to follow this recommendation.

Still, during the upgrade, the preupgrade.log shows
+ (AUTOFIXUP) Gather stale data dictionary statistics prior to database
upgrade in off-peak time using:

EXECUTE DBMS_STATS.GATHER_DICTIONARY_STATS;

Dictionary statistics do not exist or are stale (not up-to-date).

Dictionary statistics help the Oracle optimizer find efficient SQL
execution plans and are essential for proper upgrade timing. Oracle
recommends gathering dictionary statistics in the last 24 hours before
database upgrade.

For information on managing optimizer statistics, refer to the 12.1.0.2
Oracle Database SQL Tuning Guide.

If they say AUTOFIXUP - treat this as a promise (or treat).

After preupgrade.jar, it's to run preupgrade_fixups.sql.
This calls (among others)
fixup_result := dbms_preup.run_fixup('dictionary_stats');
If you go through the package and other functionality, you will find function dictionary_stats_check in preupgrade_package.sql. And there it runs
BEGIN
EXECUTE IMMEDIATE
' select 1 from dual where exists(
select distinct operation
from DBA_OPTSTAT_OPERATIONS
where operation =''gather_dictionary_stats''
and start_time > systimestamp - INTERVAL ''24'' HOUR) '
INTO dictionary_stats_recent;

and if no dictionary stats were conpleted recently, this function is executed:

-- *****************************************************************
-- This fixup executes dictionary stats pre upgrade
-- *****************************************************************
FUNCTION dictionary_stats_fixup (
result_txt IN OUT VARCHAR2,
pSqlcode IN OUT NUMBER) RETURN NUMBER
IS
stats_result BOOLEAN;
sys_string varchar2(5):='SYS';
BEGIN
stats_result := run_int_proc('DBMS_STATS.GATHER_DICTIONARY_STATS', result_txt, pSqlcode);

IF (stats_result) THEN
RETURN c_success;
ELSE
RETURN c_failure;
END IF;
END dictionary_stats_fixup;

That's the moment where I recognised how serious Oracle takes gathering of dictionary stats.

A little rant on twitter about this aggressive recommendation gave me a nice reply where the recommendation is stated:

🤔

At least I know where to change the code - but it must be done before preupgrade.jar is run!

But there is a 2nd situation when DBMS_STATS.GATHER_DICTIONARY_STATS is run:
After the upgrade itself, postupgrade_fixups.sql must run.
Here the side note to my initial statement why preupgrade.zip MUST be unzipped to $ORACLE_HOME/rdbms/admin:
VARIABLE admin_preupgrade_dir VARCHAR2(512);

REM
REM point PREUPGRADE_DIR to OH/rdbms/admin
REM
DECLARE
oh VARCHAR2(4000);
BEGIN
dbms_system.get_env('ORACLE_HOME', oh);
:admin_preupgrade_dir := dbms_assert.enquote_literal(oh || '/rdbms/admin');
END;
/

DECLARE
command varchar2(4000);
BEGIN
command := 'CREATE OR REPLACE DIRECTORY PREUPGRADE_DIR AS ' || :admin_preupgrade_dir;
EXECUTE IMMEDIATE command;
END;
/

@?/rdbms/admin/dbms_registry_basic.sql
@?/rdbms/admin/dbms_registry_extended.sql


REM
REM Execute the preupgrade_package from the PREUPGRADE_DIR
REM This is needed because the preupgrade_messages.properties file
REM lives there too, and is read by preupgrade_package.sql using
REM the PREUPGRADE_DIR.
REM
COLUMN directory_path NEW_VALUE admin_preupgrade_dir NOPRINT;
select directory_path from dba_directories where directory_name='PREUPGRADE_DIR';
set concat '.';
@&admin_preupgrade_dir./preupgrade_package.sql
That's the reason why unzipping it ton another directory, after the upgrade an older version of preupgrade scripts is used - maybe not a desired thing!

Later there another fixup is called:
fixup_result := dbms_preup.run_fixup('post_dictionary');
This in that case that function is run:

-- *****************************************************************
-- POST_DICTIONARY_CHECK Section
-- This check recommends re-gathering dictionary stats post upgrade
-- The logic in the query is: Check if statistics has been taken
-- after upgrade, if not report it and generate the fixup in the
-- postupgrade fixup script, after the fixup run, it will not fail
-- and therefore it will report this check as successfull.
-- *****************************************************************
FUNCTION post_dictionary_check (result_txt OUT VARCHAR2) RETURN NUMBER
IS
dictionary_stats_recent NUMBER := 0;
correct_version boolean := TRUE;

BEGIN
IF dbms_registry_extended.compare_versions(db_version_4_dots, C_ORACLE_HIGH_VERSION_4_DOTS, 4) < 0 THEN
correct_version := FALSE;
END IF;

BEGIN
EXECUTE IMMEDIATE
' select 1 from dual where exists(
select distinct operation
from DBA_OPTSTAT_OPERATIONS
where operation =''gather_dictionary_stats''
and start_time > (select max(OPTIME) from registry$log where cid =''UPGRD_END'')) '
INTO dictionary_stats_recent;
EXCEPTION
WHEN NO_DATA_FOUND then
null;
END;

After all this, I STILL recommend to use LATEST preupgrade.jar,
unzip it to proper $ORACLE_HOME/rdbms/admin,
execute it as documented
and don't mess around with it!
Don't mess around with it unless Oracle Support says so (or you take all the unforseen side effects on your credits).


connection manager - what happens at startup & shutdown

$
0
0
Oracle Connection Manager (cman) is a great tool to create a gateway between networks which can not be connected, or filter those who can access to a service.
Most of the time, it's a very robust implementation and doesn't need a lot of attention. But if it's required to dig into it, a basic knowledge about it's components can be useful.


In this post I'll show what happens in a simple startup (& shutdown) of a mostly "default" cman.
The test was done on Oracle Linux 7.5 with cman 12.2.0.1.181016.


tl:dr

In my setup, processes are called this order.
at startup, cmctl spawns several processes. these processes communicate via sockets and has some shared memory segments

The details: 

The results in this post show some lines of interest cenerated by
strace -f -o <some_dir&gt>/cman_$$.strace -y -s 128 cman
and the 3 commands
administer cman_berx
startup
shutdown

Before each quote of the tracefile I write the line number - this is not an exact offset for your tests, just shows the sequence and sometimes big gaps which I do not show. I will name PIDs and executes together for better readability.

1:
10115 execve("<ORACLE_HOME>/bin/cmctl", ["cmctl"], [/* 118 vars */]) = 0
process 10115 is now cmctl.

200:
10115 open("<ORACLE_HOME>/env.ora", O_RDONLY) = 3
...
10115 read(3, "# Copyright (c) 2015, Oracle and/or its affiliates. All rights reserved.\n# \n# NAME\n# env.ora\n#\n# FUNCTION\n# Oracle ENVironme"..., 4096) = 852
The undocumented file <ORACLE_HOME>/env.ora is checked for TNSNAMES_ORA parameter there - my file is default, so nothing is set there.

231:
10115 access("<TNS_ADMIN>/cman.ora", F_OK) = -1 ENOENT (No such file or directory)
10115 access("/etc/cman.ora", F_OK) = -1 ENOENT (No such file or directory)
10115 access("<ORACLE_HOME>/network/admin/cman.ora", F_OK) = 0
10115 open("<ORACLE_HOME>/network/admin/cman.ora", O_RDONLY) = 3
...
10115 read(3<<ORACLE_HOME>/network/admin/cman.ora>, "######################################################################\n#\n# Copyright (c) 2001,2002, Oracle Corporation. All righ"..., 4096) = 3996
...
10115 close(3<<ORACLE_HOME>/network/admin/cman.ora>) = 0
10115 (cman) searches for cman.ora in <TNS_ADMIN> (nothing there) /etc (also nothing) and <ORACLE_HOME>/network/admin (there is an entry for cman_berx).

297:
10115 write(1</dev/pts/0>, "CMCTL> ", 7) = 7
10115 read(0</dev/pts/0>, "administer cman_berx\n", 1024) = 21
10115 (cmctl) writes a prompt (CMCTL>) and gets my command line command (administer cman_berx).

364:
10115 connect(4<socket:[1549779484]>, {sa_family=AF_INET, sin_port=htons(1522), sin_addr=inet_addr("x.y.z.78")}, 16) = -1 EINPROGRESS (Operation now in progress)
10115 (cmctl) tries to connect to the address:port provided in cman.ora - and can't connect. I assume that's the moment it realises it must start all related processes.


418:
10115 geteuid()                         = 5001
10115 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b8713915150) = 10286
10115 rt_sigprocmask(SIG_BLOCK, [PIPE], NULL, 8) = 0
10286 set_robust_list(0x2b8713915160, 24 <unfinished ...>
10115 rt_sigaction(SIGPIPE, {0x2b8717669de0, ~[ILL TRAP ABRT BUS FPE SEGV USR2 TERM XCPU XFSZ SYS RTMIN RT_1], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x2b8718e9f680}, <unfinished ...>
10286 <... set_robust_list resumed> ) = 0
10115 <... rt_sigaction resumed> {SIG_DFL, [], 0}, 8) = 0
10115 rt_sigprocmask(SIG_UNBLOCK, [PIPE], NULL, 8) = 0
10286 clone( <unfinished ...>
10115 wait4(10286, <unfinished ...>
10286 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b8713915150) = 10287
10287 set_robust_list(0x2b8713915160, 24) = 0
10287 close(6<pipe:[1549773687]>) = 0
10287 close(7<pipe:[1549773688]> <unfinished ...>
10286 exit_group(0) = ?
10287 <... close resumed> ) = 0
10287 setsid( <unfinished ...>
10286 +++ exited with 0 +++
10115 (cmctl) creates (clone) a new process (10286) - this process immediately clones another one (10287) and then terminates itself (exit_group). By this there is no parent-child relationship between 10115 (cmctl) and 10287. Still they both use the same binary (cmctl) and filehandles etc. - so 10287 closes some filehandles it doesn't require.

449:
10287 execve("/appl/oracle/product/cman_12201/bin/cmadmin", ["/appl/oracle/product/cman_12201/bin/cmadmin", "cman_berx", "-inherit"], [/* 119 vars */]) = 0
process 10287 is now cmadmin.


2602:
10287 access("/var/tmp/.oracle", F_OK)  = 0
10287 chmod("/var/tmp/.oracle", 01777) = -1 EPERM (Operation not permitted)
10287 socket(AF_LOCAL, SOCK_STREAM, 0) = 6<socket:[1549770245]>
...
10287 access("/var/tmp/.oracle_500100", F_OK) = -1 ENOENT (No such file or directory)
10287 mkdir("/var/tmp/.oracle_500100", 0700) = 0
10287 chmod("/var/tmp/.oracle_500100", 0700) = 0
10287 access("/var/tmp/.oracle_500100/s#10287.1", F_OK) = -1 ENOENT (No such file or directory)
10287 bind(6<socket:[1549770245]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10287.1"}, 110) = 0
10287 chmod("/var/tmp/.oracle_500100/s#10287.1", 0777) = 0
10287 listen(6<socket:[1549770245]>, 100) = 0
10287 getsockname(6<socket:[1549770245]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10287.1"}, [36]) = 0
10287 chmod("/var/tmp/.oracle_500100/s#10287.1", 0777) = 0
10287 (cmadmin) checks if /var/tmp/.oracle exists and if it can change it's permission. In my environment the directory exists (as Grid Infrastructure is installed) but it's permissions can't be changed. It belongs to root.  Then /var/tmp/.oracle_500100 is created and a socket is created there. For easier readability, the PID is part of the socket name.

2631:
10287 clone(child_stack=0x2b6fd20de570, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2b6fd20e99d0, tls=0x2b6fd20e9700, child_tidptr=0x2b6fd20e99d0) = 10289
10287 (cmadmin) creates another process (10289).

2701:
10287 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b6fd1f80150) = 10290
10287 (cmadmin) creates another process (10290).

2719:
10290 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b6fd1f80150) = 10291
...
10290 exit_group(0) = ?
...
10290 +++ exited with 0 +++
10290 creates 10291 and terminates itself.

2764:
10291 execve("/appl/oracle/product/cman_12201/bin/tnslsnr", ["/appl/oracle/product/cman_12201/bin/tnslsnr", "ifile=/appl/oracle/product/cman_12201/network/admin/cman.ora", "cman_berx", "-inherit", "-mode", "proxy"], [/* 119 vars */]) = 0
10291 becomes tnslsnr


3613:
10291 bind(11<socket:[1549780776]>, {sa_family=AF_INET, sin_port=htons(1522), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
10291 listen(11<socket:[1549780776]>, 128) = 0
10291 getsockname(11<socket:[1549780776]>, {sa_family=AF_INET, sin_port=htons(1522), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
10291 (tnslsnr) listens on 0.0.0.0:1522 - But in cman.ora a specific IP address is given.

3669:
10291 write(12<<DIAG_DEST>/netcman/<hostname>/cman_berx/trace/cman_berx.log>, "Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<server.domain>)(PORT=1522)))\n", 91) = 91
10291 (tnslsnr) writes something different to the logfile (?)

3674:
10291 access("/var/tmp/.oracle", F_OK)  = 0
10291 chmod("/var/tmp/.oracle", 01777) = -1 EPERM (Operation not permitted)
10291 socket(AF_LOCAL, SOCK_STREAM, 0) = 14<socket:[1549790968]>
10291 access("/var/tmp/.oracle/s#10291.1", F_OK) = -1 ENOENT (No such file or directory)
10291 bind(14<socket:[1549790968]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/s#10291.1"}, 110) = 0
10291 chmod("/var/tmp/.oracle/s#10291.1", 0777) = 0
10291 listen(14<socket:[1549790968]>, 1) = 0
10291 getsockname(14<socket:[1549790968]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/s#10291.1"}, [29]) = 0
10291 chmod("/var/tmp/.oracle/s#10291.1", 0777) = 0
10291 (tnslsnr) creates its own socket file for communication purpose.

4088:
10291 connect(16<socket:[1549778909]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sprocr_local_conn_0_PROL"}, 110) = 0

4149:
10291 connect(15<socket:[1549778910]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle/sCRSD_UI_SOCKET"}, 110) = 0
10291 (tnslsnr) checks some sockets which belong to Grid Infrastructure.

4616:
10291 clone(child_stack=0x2ac6b580b570, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2ac6b58169d0, tls=0x2ac6b5816700, child_tidptr=0x2ac6b58169d0) = 10292
10291 (tnslsnr) creates PID 10292

4754:
10292 clone( <unfinished ...>
10291 <... close resumed> ) = 0
10292 <... clone resumed> child_stack=0x2ac6b584c570, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x2ac6b58579d0, tls=0x2ac6b5857700, child_tidptr=0x2ac6b58579d0) = 10293
10293 set_robust_list(0x2ac6b58579e0, 24 <unfinished ...>
10292 creates PID 10293

4790:
10293 exit(0)                           = ?
10292 <... poll resumed> ) = 1 ([{fd=15, revents=POLLIN}])
10293 +++ exited with 0 +++
10293 exits - it did not "clone" or "execve" anything

4841:
10287 shmget(0x59248000, 2462280, IPC_CREAT|IPC_EXCL|0600 <unfinished ...>
10287 (cmadmin) allocates a shared memory segment with key 0x59248000.

4937:
10287 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b6fd1f80150) = 10294
...
10294 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b6fd1f80150) = 10295
...
10294 exit_group(0) = ?
...
10294 +++ exited with 0 +++
10287 (cmadmin) creates PID 10294, this creates PID 10295 and terminates itself.


4967:
10295 execve("/appl/oracle/product/cman_12201/bin/cmgw", ["/appl/oracle/product/cman_12201/bin/cmgw", "cmgw0", "0", "16", "cman_berx", "SNLSM:59248000"], [/* 119 vars */]) = 0
10295 becomes cmgw0 - you can see the parameter SNLSM - it gives the shared memory key from 10287 (cmadmin).

5158:
10295 shmget(0x59248000, 1, 0)          = 2099576860
10295 (cmgw0) attaches to shared memory 10287 (cmadmin).


5566:
10287 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b6fd1f80150) = 10296
...
10296 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2b6fd1f80150) = 10297
...
10296 exit_group(0) = ?
10287 (cmadmin) creates PID 10296 which creates 10297 and exits. - same as above

5625:
10297 execve("/appl/oracle/product/cman_12201/bin/cmgw", ["/appl/oracle/product/cman_12201/bin/cmgw", "cmgw1", "1", "16", "cman_berx", "SNLSM:59248000"], [/* 119 vars */] <unfinished ...>
10297 becomes cmgw1.

some (later) important sockets:
5680:
10295 connect(5<socket:[1549804554]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10287.1"}, 110 <unfinished ...>

6447:
10297 connect(5<socket:[1549798977]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10287.1"}, 110 <unfinished ...>

7448:
10295 bind(7<socket:[1549804555]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10295.1"}, 110 <unfinished ...>

7458:
10297 bind(7<socket:[1549798978]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10297.1"}, 110 <unfinished ...>

7896:
10115 connect(5<socket:[1549779550]>, {sa_family=AF_LOCAL, sun_path="/var/tmp/.oracle_500100/s#10287.1"}, 110 <unfinished ...> 

At this stage, cman is started, all processes are in memory and happy to server any requests.



The shutdown- command will be much faster:

8553:
10115 <... read resumed> "shutdown\n", 1024) = 9
10115 (cmctl) recieves "shutdown" from command line

8554
10115 write(5, "\0\0\0S\6$\0\0\0\0E\0\0\200\2\0\0\0\0\4\0\0\220_8\2\0\0\0\0\20\0#\0\0\000901A4C4F9FD8-3939-E034-0800208AB384\1\0v\0\0\0\3\0\0\0\0\0", 83) = 83
101115 (cmctl) informs 10287 (cmadmin)

8568:
10287 write(5&lt;socket:[1549778933]&gt;, "\0\0\0S\6 \0\0\0\0E\0\0\200\2\0\0\0\0\4\0\0p\344t\2\0\0\0\0\20\0#\0\0\00091986B2B10AC-3001-E034-0800208AB384\1\0l\0\0\0\1\0\0\0\0\0", 83) = 83
10287 times( &lt;unfinished ...&gt;
10295 &lt;... epoll_wait resumed&gt; [{EPOLLIN, {u32=13989488, u64=13989488}}], 1024, -1) = 1
10287 &lt;... times resumed&gt; NULL)         = 1269637186
10287 epoll_wait(7&lt;anon_inode:[eventpoll]&gt;,  &lt;unfinished ...&gt;
10295 read(5&lt;socket:[1549804554]&gt;, "\0\0\0S\6 \0\0\0\0E\0\0\200\2\0\0\0\0\4\0\0p\344t\2\0\0\0\0\20\0#\0\0\00091986B2B10AC-3001-E034-0800208AB384\1\0l\0\0\0\1\0\0\0\0\0", 8208) = 83
1
10287 (cmadmin) informs 10295 (cmgw0) (and later 10297 (cmgw1) by the same method) to shutdown

some cleanup takes place:

8585:
10295 unlink("/var/tmp/.oracle_500100/s#10295.1"<unfinished ...>

8590:
10295 rmdir("/var/tmp/.oracle_500100"<unfinished ...>

8621:
10297 unlink("/var/tmp/.oracle_500100/s#10297.1"<unfinished ...>




and all the processes terminate:

8869:
10289 exit(0)                           = ?
...
10289 +++ exited with 0 +++

8927:
10295 exit_group(0)                     = ?

8967:
10297 exit_group(0)                     = ?

9096:
10291 unlink("/var/tmp/.oracle/s#10291.1"<unfinished ...>


9293:
10291 exit_group(0)                     = ?

9305:
10287 exit_group(0)                     = ?

9321:
10115 exit_group(0)                     = ?

That's a simple startup and shutdown of connection manager - nothing fancy here, just a lot of processes and sockets which are created - and then deleted.

I hope you never need to read this article to it's end. But if you need to, it should provide some value for you!

shared resources

$
0
0
Some days ago I had one of these calls from an application colleague:
The database was slow yesterday, can you please check what's the problem?
Of course, I had some short discussion if he really means the DB or should it be called the application is slow. Also some other questions needed to be asked first, e.g. if it's a response time or throughput issue, when it was "good" last time, what "bad" and "good" means in numbers (seconds/ experience, requests / second), if it's affecting all the experiences on this DB or only a subset. Can it be reproduced or at least does it occur on a known pattern. Also the DB name and users affected I had to ask.
Some time I should create a comprehensive checklist for these questions as I'm missing some of them regularly. But that not today's topic.

As the answers are very vague (at least I got a DB & username with some rough timestamp when it might has started) I checked the DB. I'm it would be more precise to focus on user experience. But I know the developers quite well; it would cost a lot of effort & management pressure to convince them to implement proper instrumentation.
So I was going for low hanging fruits first. ASH/AWR showed this picture:
There is time of more activities in the Database, and it's quite visible where it started and ended. With some more analysis it was obvious the number of calls didn't change dramatically. Getting data from the application (yes, there was at least an activity log) showed in fact the number of requests followed a daily pattern, but this didn't look like the picture above. The pattern of demand was similar to other days; but on any other day AWR showed no such picture, only a "breathe" similar to the applications demand.
By this conversation we implicit generated a (still very blurry) "good" pattern to compare the "bad" one.


These bits of information were not perfect, but at least something I could use for further investigations.
AWR shows much higher IO times than normal. I crosschecked the most active SQL_IDs from a "good" pattern against the "bad" one and could confirm they are IO-bound in "good" as well, so they might be affected by this unusual IO.
On the DB-cluster TFA is installed, and so is oswatcher. (Again, aggregates only. But still much better information than nothing). A small awk script analysed the data there (I just don't know why oswatcher doesn't has a SQL interface similar to osquery ) and there were huge response time, but no queueing on the Linux hosts.
This would now require digging deeper through the storage layers (FC SAN and several big central storage boxes). Unfortunately I does not have access to them and their performance data is not externalized right now and the SAN admin was to busy with other things, like compiling reports for senior managers 🤦.
But at least I'm somehow lucky: At this company a nice schema is implemented which gives ASM-disks much more information than only /dev/mapper/anything. As the name of the storage box is part of the ASM-disk path, by some knowledge about the possible servers involved (e.g. only servers in this datacenter, and with some other attributes could be involved at all) and a centralized repository I got a list of all DB-servers and DBs potentially involved on that layer.
With this list and another part of our repository (in that case periodic v$sysstat snapshots [aggregates of aggregates - guess the accuracy there]) a list of other DBs with high IO numbers during that time could be compiled.
Checking these databases there was a top producer of IO:

Some additional analysis showed this load on this particular DB created sufficient IO to saturate one subsystem in the storage box. By doing so all other system which use the same subsystem also suffered. To crosscheck that, the Storage admin (who completed the report in the meantime) provided other DBs utilizing that subsystem - and they showed similar patterns at exactly that time.

Even with all the data collected, this accumulation of coincidents is still no prove what was going on. But it was sufficient likely for the management to think about remediation activities.


Here the question might show up why only the admins of this one application showed up and complained about their system? During the investigation it was visible that other DBs showed similar patterns and so other applications were likely to suffer also.
I invested some time to  get answers to these questions. This is not related to the initial goal of the activities, only to satisfy my curiosity and improve my knowledge about the whole landscape.
It showed these other applications also suffered, but the impact was much smaller for 2 reasons:

  1. The portion of DB IO time to their user experience was relatively small. Even these high response time didn't cause a lot of harm. 
  2. The requirements of the most suffering application were more strict. It can be explained by the time a person accepts to wait after a click: If this click is on a webpage, most readers of this blog might understand what's going on behind the scene and some tenth of a second up to some seconds are acceptable for an answer. But if you flip a switch to turn on the light, they expect the light to shine immediately (aka. within less time than the human senses can grasp)


After all the analysis was done and roughly explained to the manager of the most suffering application, he invited for a meeting.
A manager, some people with different roles in development and ops, a representative for the "culprit application which caused the high load" and I were there. He opened the meeting with a summary (condensed by me to):
We and they share the same storage, they did some heavy query and we suffered. 
Technically this is right, but the way it's provided to those not experienced in IT leads to the managers request:
We must ensure they do not cause such IO load anymore. 

This might seem legit, but it might not lead to the managers real goal: Make his application reasonable fast in as many situations as possible. So I had to kill this branch of the discussion for a moment and explained, why they are not the real problem:
As the whole IT is a shared system on various layers (different hosts share the same storage system, different virtual hosts share the same VM layer, different instances share the same DB-cluster, different sessions share the same instance, ...) it is an unreasonable big effort to define resource limits, quotas and similar for all components involved. Especially in a highly dynamic environment with hundreds of DBs, thousands of servers and zillions of processes.
A majority in this room showed some shock & horror and claimed they were not aware of the high level of complexity in basic infrastructure.

With some effort we can limit the IO consumption of this particular DB which caused the issue this time. But next time (and there will be a next time) we only can be sure this particular DB is not involved.
As that's not sufficient, the next request was:
Let's then monitor the storage box and alarm every time a component is saturated.
A nice idea, isn't it?
But whom to alarm? -
The SAN admin of course. 
What should he do then? He can't reduce the load and can't increase the storage capacity either (at least not short term without budget, vendor involved, ...). 
So let's inform those which are connected to this resource! 
In our example it would have been two dozen applications - most of them didn't suffer or care.
Then only the top 5.
In our case you wouldn't have been alarmed.

This also doesn't lead to anything useful.

About at that time I raised a simple question:
Who in this room is responsible for the (suffering part of the) application?
---
<silence>
(the managers face went pale for a second).
After some moments, an external (not employed) developer raised his hand and muttered something like "If no one wants to, I can take that". 
I was slightly bored of this meeting, so to speed up things my next question was:

Is every IO your application doing [during the critical activity] necessary?

The argument in short goes like that: If you are less dependent on IO timing, even bad IOs can't harm that much. I call this

make it robust against impacts out of your control

 Beside the obvious effect to improve the application, a nice side effect is a less busy total environment. This again pays back as better service for the whole company.

The meeting stopped close to that point. The new found architect & developers promised to see if they can make their application more robust, the manager wants to raise awareness in all IT departments what a shared environment is and how it can affect business processes within the company.
I have a mixed feeling: the analysis seems valid, but it was full of educated guess & shortcuts - this doesn't feel right as it contains "experience" and "luck"; there is a method which describes a more structured way, I just felt unable to follow these steps . I hope I could transfer the initial reaction (finger-pointing to another team) to something more valuable for the company: use less resources, make your application robust.
But I doubt my effort was lasting.

limit IOs by instances

$
0
0
This is a follow up to my previous post.

I had to answer if I can limit the (IO) resources a DB instance can utilize. Unfortunately, in a simple instance I can not do so. It can be done in PDBs, but right now PDBs are out of scope.
So a simpler approach was developed: limiting IOs by creating cgroups.

There are 2 steps:

  1. disks
    1. create a proper cgroup
    2. get all related devices for a given ASM diskgroup
    3. set proper limits for all the devices
  2. regularly add all matching processes to this cgroup

In the cgroups mountpoint, a new directory must be created which is used as "root" for all cgroups, so it does not collide with other cgroups implementations in the system.

This leads to a structure like
/sys/fs/cgroup/blkio/ORACLE/BX1/
where ORACLE is the "root" and "BX1" is the name of a specific cgroup.
In this group limits can be set, e.g. in blkio.throttle.read_bps_device or blkio.throttle.read_iops_device.
As the limit is per device, the total limit is divided by the number of devices.

To have any effect on processes, all processes PIDs which are under the regime of this cgroup are added to /sys/fs/cgroup/blkio/ORACLE/BX1/tasks.
For more flexibility, a list of patters (to match cmd in ps -eo pid,cmd ) are defined for each cgroup individualy. e.g. all processes, which either match a foreground process which was connected via listener BXTST02.*LOCAL=NO or any parallel process ora_p[[:digit:]].{2}_BXTST02.

In my configuration (crontab), the disks are added to the cgroup once per hour, whereas processes are added every minute.
This can lead to some delays if disks are added, and every single process can live up to one minute without any limits.

But for a longer period, it should be quite stable. (or at least, the best I can do in the sort time given)

The effect is acceptable. In a generic test with SLOB the picture is obvious:
When the processes were added to the cgroup, MB/s dropped down to the 5MB/s configured.

Of course by these limits, the average response time (or wait time) goes up.
In another test where SLOB was in the cgroup all the time, the MEAN responsetime was 0.027 sec.
But a histogram shows more than 50% of the calls finish within 10 ms (a reasonalbe value for the storage system in this test) but there is a peak between 25 and 50 ms which dominates the total response time.
RANGE {min ≤ e < max}    DURATION       %   CALLS      MEAN       MIN       MAX
--------------------- ---------- ------ ------ -------- -------- --------
1. 0ms 5ms 6.010682 2.0% 2,290 0.002625 0.000252 0.004999
2. 5ms 10ms 26.712672 9.0% 3,662 0.007295 0.005002 0.009997
3. 10ms 15ms 12.935713 4.4% 1,090 0.011868 0.010003 0.014998
4. 15ms 20ms 6.828035 2.3% 398 0.017156 0.015003 0.019980
5. 20ms 25ms 4.846490 1.6% 218 0.022232 0.020039 0.024902
6. 25ms 50ms 17.002454 5.7% 471 0.036099 0.025085 0.049976
7. 50ms 100ms 182.104408 61.6% 2,338 0.077889 0.050053 0.099991
8. 100ms 1,000ms 39.354627 13.3% 326 0.120720 0.100008 0.410570
9. 1,000ms +∞
--------------------- ---------- ------ ------ -------- -------- --------
TOTAL (9) 295.795081 100.0% 10,793 0.027406 0.000252 0.410570

This also can be seen in a graph:

The system is working and stable; probably not perfect but good enough for it's requirements.
There was a discussion if this should be achieved on storage layer. This would limit every process immediately, but would also be a much stricter setting. As an example I can exclude logwriter from any cgroup and let it work as fast as possible, whereas IO limits on storage side would put logwriter and foreground processes in same limits.

The script can be found at github, but it has some prerequisits and might not work on other than my current systems without adaption. Don't hesitate to contact me if you really want to give it a try 😉

invisible IOs

$
0
0
My team identified an interesting case of invisible IOs this week. During some checks of system statistics graphs, something unexpected ocured:
One of those "small & unimportant" test DBs had an interesting IO pattern:





This was something interesting, so a colleague decided to have a deeper look at it. The next obvious step is to see if AAS shows anything interesting. And it does:


A lot of blue (which is used for IO in all tools I'm aware of) was expected, but it's obviously dominated by green (CPU). What a surprise!

After some quick crosschecks if any of the systems show fake data, we agreed it's all correct. Still the question exists: why does system statistics show a lot of IO, whereas ASH looks very different?

More data is needed, so one question was: out of all the direct path read IOs we saw in this timespawn, how fast were they? The first source for a check is DBA_HIST_SYSTEM_EVENT:
SELECT
sysev.event_name,
sysev.wait_class,
MAX(total_waits) - MIN(total_waits) AS sum_total_waits,
MAX(time_waited_micro) - MIN(time_waited_micro) AS sum_time_waited,
( MAX(time_waited_micro) - MIN(time_waited_micro) ) / ( MAX(total_waits) - MIN(total_waits) + 1) time_per_wait
FROM
dba_hist_snapshot snap,
dba_hist_system_event sysev
WHERE snap.snap_id = sysev.snap_id
AND begin_interval_time BETWEEN to_timestamp('2019-05-03 06:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND to_timestamp('2019-05-03 20:00:00', 'YYYY-MM-DD HH24:MI:SS')
GROUP BY GROUPING SETS (
( sysev.event_name, sysev.wait_class ),
( sysev.wait_class )
)
ORDER BY 3 DESC, 2, 1;


EVENT_NAME                     WAIT_CLASS      SUM_TOTAL_WAITS SUM_TIME_WAITED  TIME_PER_WAIT
------------------------------ --------------- --------------- --------------- --------------
User I/O 156659399 87378811924 557.76
System I/O 134570949 10821645896 80.42
Idle 66603429 ############### 2432741.08
Network 65725910 5330619377 81.10
Administrative 20587618 191843235750 9318.38
direct path read User I/O 10353091 2001455155 193.32
db file sequential read User I/O 1008098 255164480 253.11
Other 947941 10721785435 11310.59
control file sequential read System I/O 472653 11995170 25.38
Application 460657 73815782 160.24
SQL*Net message from client Idle 303956 500628620721 1647037.64

The topmost User I/O event is direct path read. During the 14 hour period, it happened at about 200 times every second. Based on the ASH graph above we can assume it all happened in one session, so in this session every second about 40ms were spent in direct path read.
This now raises the question why these 40% of time are not visible in ASH? The best answer I found so far is described in ASHviz: Densities and dark matter by John Beresniewicz.
Short events are underrepresented in sampled data. And as one single direct path read lasted only 193μs on average, the observation was hit by this bias.
With this very simple test-DB it was possible to investigate the situation to a reasonable degree of understanding, But chasing the real producer of high IOs in a busy production system might be misleading when only ASH is used.



This situation is kind of a downer for me as ASH does not even close show the truth. (this is a good example, where tracing is better than sampling). So a reliable source for additional information is required.
In recent Oracle versions (10.1 or later) there is some information available: Of course v$sesstat shows statistics for every session, but the numbers are cumulative and must be processed somehow.
Oracle provides v$sessmetric for this purpose, but it only shows the last x seconds and only a small subset of statistics. A more generic solution is Tanel Põders snapper. But both approaches can only show what's going on at the time of observation.
For something more DBA_HIST_ish, DBA_HIST_SESSMETRIC_HISTORY looks promising, but it is misleading, as This view is populated only if a session metric exceeds a server metric threshold that was configured using the DBMS_SERVER_ALERT package.
There is much more about these features & their limitations in Kyle Hailys Oracle 10g Performance: chapter 04 new features.

With all that in mind, I hope for additional information about session statistic deltas. (SSD would be a perfect and unambiguous abbreviation 😅)
In short, it should write all changed session stats for all sessions at every snapshot. In addition for every closed session their statistics must be saved, otherwise their data is lost. Tanel implemented something similar already in his Session-level statspack but he decided not to store session statistics by default to save space. Also no logoff-trigger is there (yet).
Let's see if there will ever be a SSD implementation available.

deleting orphaned CommVault backups

$
0
0

In the current environment Commvault is used as a central backup solution. The setup is not very common, as CommVaults scheduler is not used for any activities. All backups are scheduled by a system which also takes some infrastructural limitations into consideration. These are (among others): shared IO resources of consolidated data­bases on a cluster, shared backup network resources of clusters, shared media agents in different datacenters, and some other constraints as well.
Also retention policies are handled only by RMAN, the retention policy of CommVaults Storage Pools is set to infinite.
Unfortunately, for some (yet not 100% clear) reasons, there are some orphaned backup pieces in CommVault which are not visible in RMAN catalog anymore. Still they need to be deleted to regain storage (and reduce number of entries in the precious Dedup DataBase).
Unfortunately, RMAN can not delete a backup piece which it doesn't know about. So we must mimic it's activities regarding the SBT_TAPE interface to send the information to delete - again.

First it's needed to get the list of backup pieces from CommVault. To do so, a query on CommVaults SQLServer DB is required:
select cbi.clientname
, cbi.instance
, cbi.startdate
, cbi.enddate
, cbi.jobid
, ora.archFileName
-- , obi.storagePolicy
-- , obi.isAged
-- , obi.Copyname
from [CommServ].[dbo].[CommCellBackupInfo] cbi
join [CommServ].[dbo].[archFileOracle] ora on cbi.jobid = ora.jobId
join [CommServ].[dbo].[CommCellOracleBackupInfo] obi on cbi.jobid= obi.jobid
where cbi.isAged = 0
and obi.isAged = 0
and cbi.enddate <= getdate()-60
-- and cbi.jobid = &jobid
order by cbi.clientname, cbi.instance, cbi.startdate

In this query some lines are commented but they might be useful for different questions. cbi.enddate is filtered to <= getdate()-60. RMAN retention is set to 40 days in this context and some spare days are used in the query.
Every archFileName must be crosschecked with RMAN catalogs RC_BACKUP_PIECE.HANDLE first. If it is still known by RMAN, there is no need to delete the corresponding CommVault Job.
(A Job in Commvault can contain several RMAN backup pieces. But the Job is/can only deleted if all BPs are send to be deleted by RMAN).

As I wrote earlier, an unknown backup piece can not be deleted by RMAN. There is no officially supported way to do so, but in Bug 1419888 : BACKUP ON ERROR DOES NOT DELETE PIECES GENERATED there is a clue how it can be done:
connect / as sysdba;
declare
dn varchar2(255);
begin
dn := dbms_backup_restore.deviceAllocate(type=>'sbt_tape', noio=>TRUE);
dbms_backup_restore.deleteBackupPiece(0,0,'04c9av2p', 0,0,1);
dbms_backup_restore.devicedeAllocate();
end;

There is also a params parameter to DBMS_BACKUP_RESTORE.DeviceAllocate which can be used to set exactly those SBT_TAPE parameters which otherwise would be used in RMAN script.


In a perfect world, this post would end here as the backup pieces would disappear.
Unfortunately it doesn't.

For any (yet not clear at all) reason, these backup pieces still exist on CommVault. All logs in $ORACLE_HOME/rdbms/log/sbtio.log looks ok:
440786 6b9d2 05/23 11:29:21 ### sbtinit: define trace file name=~~~ORACLE_HOME~~~/rdbms/log/sbtio.log
440786 6b9d2 05/23 11:29:21 ### sbtinit2: changing a trace level to: 0
440786 6b9d2 05/23 11:29:21 ### sbtinit2: API Client supports Oracle MM API Version: 2.0
440786 6b9d2 05/23 11:29:21 ### sbtinit2: Got argc=0.
440786 6b9d2 05/23 11:29:21 ### sbtinit2: exit - PID:440784, TID:440786; retVal=0
440786 6b9d2 05/23 11:29:21 ### sbtremove2: enter - PID:440784, TID:440786.
440786 6b9d2 05/23 11:29:21 ### sbtremove2: removing file: [~~~DBNAME~~~_1113069663_DF0_20181219_dbtl6i8l_1_1]...
440786 6b9d2 05/23 11:29:21 ### sbtremove2: exit - PID:440784, TID:440786; retVal=0
440786 6b9d2 05/23 11:29:21 ### sbtremove2: enter - PID:440784, TID:440786.
440786 6b9d2 05/23 11:29:21 ### sbtremove2: removing file: [~~~DBNAME~~~_1113069663_DF0_20181219_d6tl6i8g_1_1]...
440786 6b9d2 05/23 11:29:21 ### sbtremove2: exit - PID:440784, TID:440786; retVal=0
440786 6b9d2 05/23 11:29:21 ### sbtremove2: enter - PID:440784, TID:440786.
440786 6b9d2 05/23 11:29:21 ### sbtremove2: removing file: ~~~DBNAME~~~_1113069663_DF0_20181219_d8tl6i8i_1_1]...
440786 6b9d2 05/23 11:29:21 ### sbtremove2: exit - PID:440784, TID:440786; retVal=0
440786 6b9d2 05/23 11:29:21 ### sbtremove2: enter - PID:440784, TID:440786.
440786 6b9d2 05/23 11:29:21 ### sbtremove2: removing file: [~~~DBNAME~~~_1113069663_DF0_20181219_dbtl6i8l_2_1]...
440786 6b9d2 05/23 11:29:21 ### sbtremove2: exit - PID:440784, TID:440786; retVal=0
440786 6b9d2 05/23 11:29:21 ### CvHandleClean: before clean cv_handle=c2113170 ThreadNumber=1
440786 6b9d2 05/23 11:29:21 ### CvHandleClean: ThreadDeleteTidN [440786]
440786 6b9d2 05/23 11:29:21 ### sbtend: exit - PID:440784, TID:440786; retVal=0 ThreadNumber=0>
<.........
440786 6b9d2 05/23 11:29:21 ### sbtend: ***** BEGIN SBT PERFORMANCE STATISTICS(SBTPS) for Client API Session: PID:440784, TID:440786 ********
(I'm sorry for the redacted entries - easily visible by ~~~)

So back to the 1st query in this post, the one to run on CommVault SQL server. It contains a JOBID as well.

As dbms_backup_restore.deleteBackupPiece and CommVault are no friends, I still need to get rid of the orphaned backups. CommVault has a command to delete a specific backup job:
/opt/commvault/Base64/qoperation agedata -delbyjobid ‑j <JOBID> ‑delJobsOnAllCopies ‑ft Q_DATA_LOG
Unfortunately this command requires quite high privileges (the owner/creator of the job is not sufficient):

(Full) Administrative Management permission - but this is another story.

Beside these complications which makes any security manager will go crazy, the tool itself still tries to make life harder than required. When deleting a specific job, these answers must be given:
This command deletes the job on all copies, do you want to continue (y/n) ? [n] : y
To confim this action, type "Delete jobs on all copies" : Delete jobs on all copies

Obviously, (beside the typo in confim) there are no parameter to qoperation to claim "yes, I'm an adult". It needs to ask (!?)
But for every problem, there is a solution. In this case it's called expect. It literally can simulate any input-output conversation with a uncooperative program. In this case, the script for expect looks like:
#!/usr/bin/expect -f
set job_id [lindex $argv 0];

set timeout -1
spawn /opt/commvault/Base64/qoperation agedata -delbyjobid -j $job_id -delJobsOnAllCopies -ft Q_DATA_LOG
match_max 100000
expect -exact "This command deletes the job on all copies, do you want to continue (y/n) ? \[n\] : "
send -- "y\r"
expect -exact "y\r
To confim this action, type \"Delete jobs on all copies\" : "
send -- "Delete jobs on all copies\r"
expect eof
It takes the Job_ID as first parameter and steers qoperation through all its demands.

At the bottom of this post, it shows how to circumvent complications which should not exist, based on problems which should not exist also.
This says a lot about IT in 2019.

ORADEBUG DOC 19.3

$
0
0
this is the output of ORADEBUG DOC for Version 19c.

SQL> oradebug doc

Internal Documentation
**********************

EVENT Help on events (syntax, event list, ...)
COMPONENT [<comp_name>] List all components or describe <comp_name>

oradebug doc event

SQL> oradebug doc event

Event Help:
***********

Formal Event Syntax
--------------------
<event_spec> ::= '<event_id> [<event_scope>]
[<event_filter_list>]
[<event_parameters>]
[<action_list>]
[off]'

<event_id> ::= <event_name | number>[<target_parameters>]

<event_scope> ::= [<scope_name>: scope_parameters]

<event_filter> ::= {<filter_name>: filter_parameters}

<action> ::= <action_name>(action_parameters)

<action_parameters> ::= <parameter_name> = [<value>|<action>][, ]

<*_parameters> ::= <parameter_name> = <value>[, ]


Some Examples
-------------
* Set event 10235 level 1:
alter session set events '10235';

* Set events SQL_TRACE (a.k.a. 10046) level 1:
alter session set events 'sql_trace';

* Turn off event SQL_TRACE:
alter session set events 'sql_trace off';

* Set events SQL_TRACE with parameter <plan_stat> set to 'never'
and parameter <wait> set to 'true':
alter session set events 'sql_trace wait=true, plan_stat=never';

* Trace in-memory the SQL_MONITOR component (the target) and all its
sub-components at level high. Get high resolution time for each
trace:
alter session set events 'trace[sql_mon.*] memory=high,
get_time=highres';

* On-disk trace PX servers p000 and p005 for components 'sql_mon'
and 'sql_optimizer' (including sub-components) at level highest:
alter system set events 'trace[sql_mon | sql_optimizer.*]
{process: pname = p000 | process: pname=p005}';

* Same as above but only when SQL id '7ujay4u33g337' is executed:
alter system set events 'trace[sql_mon | sql_optimizer.*]
[sql: 7ujay4u33g337]
{process: pname = p000 | process: pname=p005}';

* Execute an action immediatly by using 'immediate' for the event
name:
alter session set events 'immediate eventdump(system)'

* Create an incident labeled 'table_missing' when external error
942 is signaled by process id 14534:
alter session set events '942 {process: 14534}
incident(table_missing)';


Notes
-----
* Implicit parameter level is 1 by default
e.g. '10053' is same as '10053 level 1'

* Event target (see [<target_parameters>] construct) is only
supported by specific events like the TRACE[] event

* <event_scope> and/or <event_filter> are constructs
that can be used for any event

* Same event can be set simultaneously for a different scope or
target but not for different filters.

* '|' character can be used to select multiple targets, scope or
filters.

E.g. 'sql_trace [sql: sql_id=g3yc1js3g2689 | sql_id=7ujay4u33g337]'

* '=' sign is optional in <*_parameters>

E.g. 'sql_trace level 12';

* Like PL/SQL, no need to specify the parameter name for target,
scope, filters and action. Resolution is done by position in
that case:

E.g. 'sql_trace [sql: g3yc1js3g2689 | 7ujay4u33g337]'


Help sub-topics
---------------

NAME [<event_name>] List all events or describe <event_name>
SCOPE [<scope_name>] List all scopes or describe <scope_name>
FILTER [<filter_name>] List all filters or describe <filter_name>
ACTION [<action_name>] List all actions or describe <action_name>


SQL>

oradebug doc event name


SQL> oradebug doc event name

Events in library DIAG:
------------------------------
trace[] Main event to control UTS tracing
disable_dde_action[] Event used by DDE to disable actions
ams_trace[] Event to dump ams performance trace records
ams_rowsrc_trace[] Event to dump ams row source tracing
sweep_verification Event to enable sweep file verification
enable_xml_inc_staging Event to enable xml incident staging format
dbg[] Event to hook dbgtDbg logging statements

Events in library RDBMS:
------------------------------
wait_event[] event to control wait event post-wakeup actions
alert_text event for textual alerts
trace_recursive event to force tracing recursive SQL statements
clientid_overwrite event to overwrite client_identifier when client_info is set
sql_monitor event to force monitoring SQL statements
sql_monitor_test event to test SQL monitoring
eventsync_tac Event posted from events syncing tac
sql_trace event for sql trace
pmon_startup startup of pmon process
background_startup startup of background processes
db_open_begin start of db open operation
test_gvtf test GV$() Table Tunction
fault Event used to inject fault in RDBMS kernel
gcr_systest gcr_systest
em_express EM Express debug event
emx_control event to control em express
emx_test_control event to control em express testing
awrdiag[] AWR Diagnostic Event
msgq_trace event to control msgq tracing
ipclw_trace event to control ipclw tracing
kbc_fault event to control container fault injection
asm_corruption_trace event to control ASM corruption tracing
kxdrs_sim debug event to simulate certain conditions in kxdrs layer

kcfio_debug debug event to debug kcfio based on event level

krbabrstat_fault event to control krbabrstat fault injection
periodic_dump[] event for periodically dumping
kxdrs_log_for_wbfc debug event to log skipped extents for wbfc resilvering

clean_plsql_error_stack Event to clear ORA-06512s from the error stack
plsql_coverage_flag Event to set plsql coverage flag

Events in library GENERIC:
------------------------------
kg_event[] Support old error number events (use err# for short)

Events in library CLIENT:
------------------------------
oci_trace event for oci trace

Events in library LIBCELL:
------------------------------
libcell_stat libcell statistics level specification
cellclnt_skgxp_trc_ops Controls to trace SKGXP operations
cellclnt_ossnet_trc Controls to trace IP affinity in ossnet
cellclnt_high_lat_ops Control to trace High-latency I/O operations
diskmon_sim_ops[] Diskmon simulation events
cellclnt_read_outlier_limit Control to trace read I/O outliers
cellclnt_write_outlier_limit Control to trace write I/O outliers
cellclnt_lgwrite_outlier_limit Control to trace log write I/O outliers
cellclnt_sparse_mode Mode of how to handle sparse buffers
cellclnt_submission_recon Simulation period for forcing submission reconnect

Events in library FPLIB:
------------------------------
fgq_control fine grain quarantine control

Events in library ADVCMP:
------------------------------
arch_comp_level[] arch_comp_level[<ulevel, 1-7>]
ccmp_debug columnar compression debug event
inmemory_nobasic disable KDZCF_IMC_BASIC implementation
inmemory_nohybrid disable KDZCF_IMC_HYBRID implementation
ccmp_align columnar compression enable alignment
ccmp_countstar columnar compression enable count(*) optimization
ccmp_dumpunaligned columnar compression dump dbas of unaligned CUs
ccmp_rbtree columnar compression switch back to rb tree
inmemory_force_ccl inmemory force column compression levels
columnar_cache_config flash cash configuration event
inmemory_imcu[] inmemory_imcu[<ulevel= nocomp|dml|query_low|query_high|capacity_low|capacity_high>]
inmemory_nobitpacked_gd disable bit-packed GD codes in CU. Use ub4 instead
inmemory_dicthash enable storage of dictionary hash values in IMCU
inmemory_cache_joinopt inmemory cache join optimizations
inmemory_scan_joinopt join optimizations to use during scan
inmemory_compute_cachehash compute a hash vector when there is no cached hash
ifc_dump_block block rdba to dump in IFC
ifc_disable_scan_rid disable CC2 type scan when rowid is required
ifc_sim_version_mismatch simulate IFC version mismatch error
ifc_disable_col_crypto disable column level encryption/decryption
ifc_ulevel[] ifc_ulevel
inmemory_prefixdict enable prefix compression on dictionary encoded IMCU
inmemory_disable_dsb usage of stored DSB dictionary during scans
disable_skip_pdo Disable skipping KAFPDO columns during projection
inmemory_noime disable usage of IME in queries

Events in library PLSQL:
------------------------------
plsql_event[] Support PL/SQL error number events

Events in library VPLIB:
------------------------------
key_vector_debug key vector debug parameters
key_vector_diag key vector diagnostics parameters
key_vector_test key vector testing parameters
vector_groupby_test vector group by testing parameters
vector_groupby_debug vector group by debugging parameters


SQL> spool off;

Events in library DIAG:

SQL> oradebug doc event name trace

trace: Main event to control UTS tracing

Usage
-------
trace [ component <string>[0] ]
disk < default | lowest | low | medium | high | highest | disable >,
memory < default | lowest | low | medium | high | highest | disable >,
get_time < disable | default | seq | highres | seq_highres >,
get_stack < disable | default | force >,
operation <string>[32],
function <string>[32],
file <string>[32],
line <ub4>,
conuid <ub4>


SQL> oradebug doc event name disable_dde_action

disable_dde_action: Event used by DDE to disable actions

Usage
-------
disable_dde_action [ action_name <string>[100] ]
facility <string>[20],
error <ub4>


SQL> oradebug doc event name ams_trace

ams_trace: Event to dump ams performance trace records

Usage
-------
ams_trace [ relation <string>[30] ]

SQL> oradebug doc event name ams_rowsrc_trace

ams_rowsrc_trace: Event to dump ams row source tracing

Usage
-------
ams_rowsrc_trace [ relation <string>[30] ]
level <ub4>


SQL> oradebug doc event name sweep_verification

sweep_verification: Event to enable sweep file verification

Usage
-------
sweep_verification
level <ub4>


SQL> oradebug doc event name enable_xml_inc_staging

enable_xml_inc_staging: Event to enable xml incident staging format

Usage
-------
enable_xml_inc_staging
level <ub4>


SQL> oradebug doc event name dbg

dbg: Event to hook dbgtDbg logging statements

Usage
-------
dbg [ component <string>[0] ]
operation <string>[32],
function <string>[32],
file <string>[32],
line <ub4>

Events in library RDBMS:

SQL> oradebug doc event name wait_event

wait_event: event to control wait event post-wakeup actions

Usage
-------
wait_event [ name <string>[64] ]

SQL> oradebug doc event name alert_text

alert_text: event for textual alerts

Usage
-------
alert_text
ksd_fixed_table < disable | enable >


SQL> oradebug doc event name trace_recursive

trace_recursive: event to force tracing recursive SQL statements

Usage
-------
trace_recursive

SQL> oradebug doc event name clientid_overwrite

clientid_overwrite: event to overwrite client_identifier when client_info is set

Usage
-------
clientid_overwrite

SQL> oradebug doc event name sql_monitor

sql_monitor: event to force monitoring SQL statements

Usage
-------
sql_monitor
recursive < false | true >,
force < false | true >


SQL> oradebug doc event name sql_monitor_test

sql_monitor_test: event to test SQL monitoring

Usage
-------
sql_monitor_test
level <ub4>


SQL> oradebug doc event name eventsync_tac

eventsync_tac: Event posted from events syncing tac

Usage
-------
eventsync_tac

SQL> oradebug doc event name sql_trace

sql_trace: event for sql trace

Usage
-------
sql_trace
wait < false | true >,
bind < false | true >,
plan_stat < never | first_execution | all_executions | adaptive >,
level <ub4>


SQL> oradebug doc event name pmon_startup

pmon_startup: startup of pmon process

Usage
-------
pmon_startup
delay <ub4>


SQL> oradebug doc event name background_startup

background_startup: startup of background processes

Usage
-------
background_startup
delay <ub4>


SQL> oradebug doc event name db_open_begin

db_open_begin: start of db open operation

Usage
-------
db_open_begin
delay <ub4>


SQL> oradebug doc event name test_gvtf

test_gvtf: test GV$() Table Tunction

Usage
-------
test_gvtf

SQL> oradebug doc event name fault

fault: Event used to inject fault in RDBMS kernel

Usage
-------
fault

SQL> oradebug doc event name gcr_systest

gcr_systest: gcr_systest

Usage
-------
gcr_systest
action_type <string>[32],
total_memory <ub8>,
free_memory <ub8>,
inactive_memory <ub8>,
swap_space <ub8>,
total_swap_space <ub8>


SQL> oradebug doc event name em_express

em_express: EM Express debug event

Usage
-------
em_express
level <ub4>


SQL> oradebug doc event name emx_control

emx_control: event to control em express

Usage
-------
emx_control
web_caching < false | true >,
compress_xml < none | deflate | gzip | zlib2base64 >


SQL> oradebug doc event name emx_test_control

emx_test_control: event to control em express testing

Usage
-------
emx_test_control
capture_xml < stop | start >


SQL> oradebug doc event name awrdiag

awrdiag: AWR Diagnostic Event

Usage
-------
awrdiag [ name <string>[64] ]
level <ub4>,
str1 <string>[256],
str2 <string>[256],
num1 <ub8>,
num2 <ub8>


SQL> oradebug doc event name msgq_trace

msgq_trace: event to control msgq tracing

Usage
-------
msgq_trace
level <ub4>,
flag <ub4>


SQL> oradebug doc event name ipclw_trace

ipclw_trace: event to control ipclw tracing

Usage
-------
ipclw_trace
level <ub4>,
flag <ub4>


SQL> oradebug doc event name kbc_fault

kbc_fault: event to control container fault injection

Usage
-------
kbc_fault
location < create_end | commit_end >,
error < external | internal | segv | instance >


SQL> oradebug doc event name asm_corruption_trace

asm_corruption_trace: event to control ASM corruption tracing

Usage
-------
asm_corruption_trace
level <ub4>


SQL> oradebug doc event name kxdrs_sim

kxdrs_sim: debug event to simulate certain conditions in kxdrs layer


Usage
-------
kxdrs_sim
level <ub4>


SQL>
SQL> oradebug doc event name kcfio_debug

kcfio_debug: debug event to debug kcfio based on event level


Usage
-------
kcfio_debug
level <ub4>


SQL>
SQL> oradebug doc event name krbabrstat_fault

krbabrstat_fault: event to control krbabrstat fault injection

Usage
-------
krbabrstat_fault
location < allocation | chaina | modify | unchain | chainb >


SQL> oradebug doc event name periodic_dump

periodic_dump: event for periodically dumping

Usage
-------
periodic_dump [ name <string>[64] ]
level <ub4>,
seconds <ub4>,
lifetime <ub4>


SQL> oradebug doc event name kxdrs_log_for_wbfc

kxdrs_log_for_wbfc: debug event to log skipped extents for wbfc resilvering


Usage
-------
kxdrs_log_for_wbfc
level <ub4>


SQL>
SQL> oradebug doc event name clean_plsql_error_stack

clean_plsql_error_stack: Event to clear ORA-06512s from the error stack

Usage
-------
clean_plsql_error_stack
level <ub4>


SQL> oradebug doc event name plsql_coverage_flag

plsql_coverage_flag: Event to set plsql coverage flag

Usage
-------
plsql_coverage_flag
level <ub4>,
directory <string>[4096]

Events in library GENERIC:

SQL> oradebug doc event name kg_event

kg_event: Support old error number events (use err# for short)

Usage
-------
kg_event [ errno <ub4> ]
level <ub4>,
lifetime <ub4>,
armcount <ub4>,
traceinc <ub4>,
forever <ub4>

Events in library CLIENT:

SQL> oradebug doc event name oci_trace

oci_trace: event for oci trace

Usage
-------
oci_trace
level < default | lowest | low | medium | high | highest | disable >

Events in library LIBCELL:

oradebug doc event name <libcell_events> doesn't show any documentaion.

Events in library FPLIB:


SQL> oradebug doc event name fgq_control
Error: "fgq_control" not a known event/library name
Use <event_name>, <library_name> or <library_name>.<event_name>

Events in library ADVCMP:

SQL> oradebug doc event name arch_comp_level

arch_comp_level: arch_comp_level[<ulevel, 1-7>]

Usage
-------
arch_comp_level [ ulevel <ub4> ]
ilevel <ub8>,
sortcols <ub4>,
cusize <ub4>,
analyze_amt <ub4>,
analyze_rows <ub4>,
analyze_minrows <ub4>,
mincusize <ub4>,
maxcusize <ub4>,
mincurows <ub4>,
align <ub4>,
rowlocks <ub4>,
maxcuhpctfree <ub4>,
guarantee_rll <ub4>,
cla_stride <ub4>,
dict_cla_stride <ub4>,
wt_factor <ub4>


SQL> oradebug doc event name ccmp_debug

ccmp_debug: columnar compression debug event

Usage
-------
ccmp_debug
memcheck <ub4>,
dmlclose <ub4>,
lobupdates <ub4>,
diaglevel <ub4>,
projectonly <ub4>,
colcheck <ub4>,
minmaxcheck <ub4>,
debug_hpk <ub4>,
debug_ozip <ub4>,
subcu_pmethod <ub4>,
subcu_mmarr_ppct <ub4>,
subcu_hist_ppct <ub4>,
subcu_indenc_cpct <ub4>,
subcu_loop <ub4>,
debug_pcode <ub4>,
imc_row_buffer_off <ub4>


SQL> oradebug doc event name inmemory_nobasic

inmemory_nobasic: disable KDZCF_IMC_BASIC implementation

Usage
-------
inmemory_nobasic

SQL> oradebug doc event name inmemory_nohybrid

inmemory_nohybrid: disable KDZCF_IMC_HYBRID implementation

Usage
-------
inmemory_nohybrid

SQL> oradebug doc event name ccmp_align

ccmp_align: columnar compression enable alignment

Usage
-------
ccmp_align

SQL> oradebug doc event name ccmp_countstar

ccmp_countstar: columnar compression enable count(*) optimization

Usage
-------
ccmp_countstar

SQL> oradebug doc event name ccmp_dumpunaligned

ccmp_dumpunaligned: columnar compression dump dbas of unaligned CUs

Usage
-------
ccmp_dumpunaligned

SQL> oradebug doc event name ccmp_rbtree

ccmp_rbtree: columnar compression switch back to rb tree

Usage
-------
ccmp_rbtree

SQL> oradebug doc event name inmemory_force_ccl

inmemory_force_ccl: inmemory force column compression levels

Usage
-------
inmemory_force_ccl
maxulevel <ub4>,
pctcols <ub4>


SQL> oradebug doc event name columnar_cache_config

columnar_cache_config: flash cash configuration event

Usage
-------
columnar_cache_config
dict_reuse_threshold <ub4>


SQL> oradebug doc event name inmemory_imcu

inmemory_imcu: inmemory_imcu[<ulevel= nocomp|dml|query_low|query_high|capacity_low|capacity_high>]

Usage
-------
inmemory_imcu [ ulevel < invalid | nocomp | dml | query_low | query_high | capacity_low | capacity_high > ]
target_rows <ub4>,
source_maxbytes <ub4>


SQL> oradebug doc event name inmemory_nobitpacked_gd

inmemory_nobitpacked_gd: disable bit-packed GD codes in CU. Use ub4 instead

Usage
-------
inmemory_nobitpacked_gd

SQL> oradebug doc event name inmemory_dicthash

inmemory_dicthash: enable storage of dictionary hash values in IMCU

Usage
-------
inmemory_dicthash

SQL> oradebug doc event name inmemory_cache_joinopt

inmemory_cache_joinopt: inmemory cache join optimizations

Usage
-------
inmemory_cache_joinopt
cache_hash <ub4>,
cache_numbin <ub4>


SQL> oradebug doc event name inmemory_scan_joinopt

inmemory_scan_joinopt: join optimizations to use during scan

Usage
-------
inmemory_scan_joinopt
disable_cache_hash <ub4>,
disable_cache_numbin <ub4>


SQL> oradebug doc event name inmemory_compute_cachehash

inmemory_compute_cachehash: compute a hash vector when there is no cached hash

Usage
-------
inmemory_compute_cachehash

SQL> oradebug doc event name ifc_dump_block

ifc_dump_block: block rdba to dump in IFC

Usage
-------
ifc_dump_block
ifc_dump_rdba <ub4>


SQL> oradebug doc event name ifc_disable_scan_rid

ifc_disable_scan_rid: disable CC2 type scan when rowid is required

Usage
-------
ifc_disable_scan_rid

SQL> oradebug doc event name ifc_sim_version_mismatch

ifc_sim_version_mismatch: simulate IFC version mismatch error

Usage
-------
ifc_sim_version_mismatch

SQL> oradebug doc event name ifc_disable_col_crypto

ifc_disable_col_crypto: disable column level encryption/decryption

Usage
-------
ifc_disable_col_crypto

SQL> oradebug doc event name ifc_ulevel

ifc_ulevel: ifc_ulevel

Usage
-------
ifc_ulevel [ ifc ulevel <ub4> ]
ifc_analyzer_sample <ub4>


SQL> oradebug doc event name inmemory_prefixdict

inmemory_prefixdict: enable prefix compression on dictionary encoded IMCU

Usage
-------
inmemory_prefixdict
prefixdict_enable <ub4>,
prefixdict_savings <ub4>,
dict_ratio <ub4>


SQL> oradebug doc event name inmemory_disable_dsb

inmemory_disable_dsb: usage of stored DSB dictionary during scans

Usage
-------
inmemory_disable_dsb
disable_predicate <ub4>,
disable_projection <ub4>,
disable_aggregation <ub4>,
disable_adaptivity <ub4>


SQL> oradebug doc event name disable_skip_pdo

disable_skip_pdo: Disable skipping KAFPDO columns during projection

Usage
-------
disable_skip_pdo

SQL> oradebug doc event name inmemory_noime

inmemory_noime: disable usage of IME in queries

Usage
-------
inmemory_noime

Events in library PLSQL:

SQL> oradebug doc event name  plsql_event

plsql_event: Support PL/SQL error number events

Usage
-------
plsql_event [ errno <ub4> ]

Events in library VPLIB:

oradebug doc event name does not show any documentation.



oradebug doc event scope

Event scopes in library RDBMS:


SQL[]                sql scope for RDBMS
QUEUE[] queue scope AQ


SQL> oradebug doc event scope sql

SQL: sql scope for RDBMS

Usage
-------
[SQL: sql_id <string>[20] ]


SQL> oradebug doc event scope queue

QUEUE: queue scope AQ

Usage
-------
[QUEUE: queue_name <string>[0] ]

oradebug doc event filter

Event filters in library DIAG:
------------------------------
occurence filter to implement counting for event checks
callstack filter to only fire an event when a function is on the stack.If the nofname option is used, then the event is fired only when the function is not on the stack
eq filter to only fire an event when a == b
ne filter to only fire an event when a != b
gt filter to only fire an event when a > b
lt filter to only fire an event when a < b
ge filter to only fire an event when a >= b
le filter to only fire an event when a <= b
anybit filter to only fire an event when (a & b) != 0
allbit filter to only fire an event when (a & b) == b
nobit filter to only fire an event when (a & b) == 0
bet filter to only fire an event when b <= a <= c
nbet filter to only fire an event when a < b or a > c
in filter to only fire an event when a is equal to any b .. p
nin filter to only fire an event when a is not equal to any b .. p
streq filter to only fire an event when string s1 = s2 (up to <len> characters)
strne filter to only fire an event when string s1 != s2 (up to <len> characters)
tag filter to only fire an event when a tag is set

Event filters in library RDBMS:
------------------------------
wait filter for specific wait parameters and wait duration
process filter to set events only for a specific process
px filter to check identity of the process for fault injection

Event filters in library GENERIC:
------------------------------
errarg filter to set error events only for a specific error argument

Event filters in library DIAG:

SQL> oradebug doc event filter occurence

occurence: filter to implement counting for event checks

Usage
-------
{occurence: start_after <ub4>,
end_after <ub4> }


SQL> oradebug doc event filter callstack

callstack: filter to only fire an event when a function is on the stack.If the nofname option is used, then the event is fired only when the function is not on the stack

Usage
-------
{callstack: fname <string>[64],
fprefix <string>[64],
maxdepth <ub4>,
nofname <string>[64] }


SQL> oradebug doc event filter eq

eq: filter to only fire an event when a == b

Usage
-------
{eq: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter ne

ne: filter to only fire an event when a != b

Usage
-------
{ne: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter gt

gt: filter to only fire an event when a > b

Usage
-------
{gt: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter lt

lt: filter to only fire an event when a < b

Usage
-------
{lt: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter ge

ge: filter to only fire an event when a >= b

Usage
-------
{ge: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter le

le: filter to only fire an event when a <= b

Usage
-------
{le: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter anybit

anybit: filter to only fire an event when (a & b) != 0

Usage
-------
{anybit: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter allbit

allbit: filter to only fire an event when (a & b) == b

Usage
-------
{allbit: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter nobit

nobit: filter to only fire an event when (a & b) == 0

Usage
-------
{nobit: a <ub8>,
b <ub8> }


SQL> oradebug doc event filter bet

bet: filter to only fire an event when b <= a <= c

Usage
-------
{bet: a <ub8>,
b <ub8>,
c <ub8> }


SQL> oradebug doc event filter nbet

nbet: filter to only fire an event when a < b or a > c

Usage
-------
{nbet: a <ub8>,
b <ub8>,
c <ub8> }


SQL> oradebug doc event filter in

in: filter to only fire an event when a is equal to any b .. p

Usage
-------
{in: a <ub8>,
b <ub8>,
c <ub8>,
d <ub8>,
e <ub8>,
f <ub8>,
g <ub8>,
h <ub8>,
i <ub8>,
j <ub8>,
k <ub8>,
l <ub8>,
m <ub8>,
n <ub8>,
o <ub8>,
p <ub8> }


SQL> oradebug doc event filter nin

nin: filter to only fire an event when a is not equal to any b .. p

Usage
-------
{nin: a <ub8>,
b <ub8>,
c <ub8>,
d <ub8>,
e <ub8>,
f <ub8>,
g <ub8>,
h <ub8>,
i <ub8>,
j <ub8>,
k <ub8>,
l <ub8>,
m <ub8>,
n <ub8>,
o <ub8>,
p <ub8> }


SQL> oradebug doc event filter streq

streq: filter to only fire an event when string s1 = s2 (up to <len> characters)

Usage
-------
{streq: s1 <string>[256],
s2 <string>[256],
len <ub4> }


SQL> oradebug doc event filter strne

strne: filter to only fire an event when string s1 != s2 (up to <len> characters)

Usage
-------
{strne: s1 <string>[256],
s2 <string>[256],
len <ub4> }


SQL> oradebug doc event filter tag

tag: filter to only fire an event when a tag is set

Usage
-------
{tag: tname <string>[64] }

Event filters in library RDBMS:

SQL> oradebug doc event filter wait

wait: filter for specific wait parameters and wait duration

Usage
-------
{wait: minwait <ub8>,
p1 <ub8>,
p2 <ub8>,
p3 <ub8>,
_actual_wait_time <ub8> default 'evargn(pos=1)',
_actual_wait_p1 <ub8> default 'evargn(pos=2)',
_actual_wait_p2 <ub8> default 'evargn(pos=3)',
_actual_wait_p3 <ub8> default 'evargn(pos=4)' }


SQL> oradebug doc event filter process

process: filter to set events only for a specific process

Usage
-------
{process: ospid <string>[20],
orapid <ub4>,
pname <string>[20],
con_id <ub8> }


SQL> oradebug doc event filter px

px: filter to check identity of the process for fault injection

Usage
-------
{px: slave_set <ub4>,
slave_num <ub4>,
local_slave_num <ub4>,
instance_id <ub4>,
dfo_number <ub4>,
oct <ub4>,
pxid <ub4> }

Event filters in library GENERIC:

SQL> oradebug doc event filter errarg

errarg: filter to set error events only for a specific error argument

Usage
-------
{errarg: arg1 <string>[50],
arg2 <string>[50],
arg3 <string>[50],
arg4 <string>[50],
arg5 <string>[50],
arg6 <string>[50],
arg7 <string>[50],
arg8 <string>[50] }

ORADEBUG DOC EVENT ACTION


Actions in library DIAG:

---------------------------
evfunc - Get posting function name
evfile - Get posting file name
evline - Get posting file line number as ub8
evfmt - Get trace / log format string
evargc - Get count of event check arguments as a ub8
evargn - Get event check argument value as ub8
evargp - Get event check argument value as void *
evargs
- Get event check argument as string, with optional format
errargs - Get error argument as string
errargn - Get error argument as ub8
errargp - Get error argument as pointer
errargc - Get count of error arguments as a ub8
sum
- Compute a1 + a2 + ... + a15 as ub8 (zero if all NULL)
trace
- trace to disk; apply format to string arguments
% is an argument placeholder
\n and \t are supported. Use double \ as escape
sub - Compute a1 - a2 as ub8
add - Compute a1 + a2 as ub8
mod - Compute a1 modulo a2 as ub8
div - Compute a1 / a2 as ub8
mul - Compute a1 * a2 as ub8
incr - Increment ptr by offset
decr - Decrement ptr by offset
refn
- Dereference ptr-to-number: *(ub<numsize>*)(((ub1*)<ptr>)) + <offset>)
refp
- Dereference ptr-to-ptr: *(ub1**)(((ub1*)<ptr>)) + <offset>)
refs
- Dereference ptr-to-string: *(oratext **)(((ub1*)<ptr>) + <offset>)
Length is optional; NULL-terminated string is assumed
refsl
- Dereference ptr-to-string: *(oratext **)(((ub1*)<ptr>) + <offset>)
with ptr-to-length: *(ub<lensize>*)(((ub1*)<ptr>) + <lenoffset>)
dumpFrameContext - Dump Frame Context contents
dumpBuckets
kgsfdmp
dumpDiagCtx
dumpDbgecPopLoc
dumpDbgecMarks
dumpGeneralConfiguration
dumpADRLockTable
shortstack
- get short stack (up to 256 characters)
showoffsets controls display of code offsets
skipframes can be used to overcome 256 char limit
dump_dis_action
dbgvci_action_signal_crash

Actions in library RDBMS:

---------------------------
incident - Create an Incident
sqlmon_dump - SQL Monitor Dump SGA Action
varaddr - Return address of a fixed PGA/SGA/UGA variable
username - Return user log-in name
sqlid - Return current SQL Id in character format
flashfreeze
oradebug - debug process using ORADEBUG
debugger - debug process using System Debugger
debug
- alias for 'debugger' - debug process using System Debugger
crash - crash process
kill_instance - killing RDBMS instance
controlc_signal - received 1013 signal
eventdump - list events that are set in the group
kdlut_bucketdump_action
kzxtDumpAction
dumpKernelDiagState
HMCHECK (async)
DATA_BLOCK_INTEGRITY_CHECK (async)
CF_BLOCK_INTEGRITY_CHECK (async)
DB_STRUCTURE_INTEGRITY_CHECK (async)
REDO_INTEGRITY_CHECK (async)
TRANSACTION_INTEGRITY_CHECK (async)
SQL_TESTCASE_REC (async)
SQL_TESTCASE_REC_DATA (async)
ORA_12751_DUMP
sqladv_dump_dumpctx
ORA_4030_DUMP
- dump summary of PGA memory usage, largest allocations
ORA_4036_DUMP - dump summary of PGA memory usage
HNGDET_MEM_USAGE_DUMP_NOARGS - dump hang detection memory usage
kcfis_action - kcfis actions
exadata_dump_modvers - Exadata dump module versions
QUERY_BLOCK_DUMP - Debug action for dumping a qbcdef tree
dumpADVMState - Dump contents of ADVM state
dumpASMState - Dump contents of ASM state
ASM_CHECK_DG (async) - Run check diskgroup
ASM_DUMP_KSTSS - Dump KST Trace and System State
ASM_MOUNT_FAIL_CHECK (async)
ASM_DGFDM_CHECK_NO_DG_NAME (async)
ASM_SYNC_IO_FAIL_CHECK (async)
ASM_DG_FORCE_DISMOUNT_CHECK (async)
ASM_ALLOC_FAIL_CHECK (async)
ASM_ADD_DISK_CHECK (async)
ASM_FILE_BUSY_CHECK (async)
ASM_TOOMANYOFF_FAIL_CHECK (async)
ASM_INSUFFICIENT_DISKS_CHECK (async)
ASM_INSUFFICIENT_MEM_CHECK (async)
KJZN_ASYNC_SYSTEM_STATE (async)
KSI_GET_TRACE - Get lmd0 traces for ksi issues
TRACE_BUFFER_ON - Allocate trace output buffer for ksdwrf()
TRACE_BUFFER_OFF
- Flush and deallocate trace output buffer for ksdwrf()
LATCHES - Dump Latches
XS_SESSION_STATE - Dump XS session state
PROCESSSTATE - Dump process state
SYSTEMSTATE - Dump system state
INSTANTIATIONSTATE - Dump instantiation state
CONTEXTAREA - Dump cursor context area
HEAPDUMP
- Dump memory heap (1-PGA, 2-SGA, 4-UGA, +1024-Content)
SGA_SUMMARY
- Dump SGA Summary(<=1-SGA, 2-Large Pool, 4-Streams, 8-Java, 16-Extents)
POKE_LENGTH - Set length before poking value
POKE_VALUE - Poke a value into memory
POKE_VALUE0 - Poke 0 value into memory
GLOBAL_AREA
- Dump fixed global area(s) (1=PGA/2=SGA/3=UGA, add +8 for pointer content)
REALFREEDUMP - Dump PGA real free memory allocator state
FLUSH_JAVA_POOL - Flush Java pool
PGA_DETAIL_GET
- Ask process to publish PGA detail info (level is pid)
PGA_DETAIL_DUMP
- Dump PGA detail information for process (level is pid)
PGA_DETAIL_CANCEL - Free PGA detail request (level is pid)
PGA_SUMMARY - Summary of PGA memory usage, largest allocations
MODIFIED_PARAMETERS - Dump parameters modifed by session (level unused)
ERRORSTACK
- Dump state (ksedmp). Use INCIDENT action to create incident
CALLSTACK - Dump call stack (level > 1 to dump args)
RECORD_CALLSTACK
- Record or dump call stack, level = #frames (level += 1000000 go to trc)
BG_MESSAGES - Dump routine for background messages
ENQUEUES
- Dump enqueues (level >=2 adds resources, >= 3 adds locks)
KSTDUMPCURPROC
- Dump current process trace buffer (1 for all events)
KSTDUMPALLPROCS
- Dump all processes trace buffers (1 for all events)
KSTDUMPALLPROCS_CLUSTER
- Dump all processes (cluster wide) trace buffers (1 for all events)
KSKDUMPTRACE - Dumping KSK KST tracing (no level)
DBSCHEDULER - Dump ressource manager state
LDAP_USER_DUMP - Dump LDAP user mode
LDAP_KERNEL_DUMP - Dump LDAP kernel mode
DUMP_ALL_OBJSTATS - Dump database objects statistics
DUMPGLOBALDATA - Rolling migration DUMP GLOBAL DATA
HANGANALYZE - Hang analyze
HANGANALYZE_PROC - Hang analyze current process
HANGANALYZE_GLOBAL - Hang analyze system
HNGDET_MEM_USAGE_DUMP - dump hang detection memory usage
GES_STATE - Dump DML state
RACDUMP - Dump RAC state
OCR - OCR client side tracing
CSS - CSS client side tracing
CRS - CRS client side tracing
SYSTEMSTATE_GLOBAL - Perform cluster wide system state dump (via DIAG)
DUMP_ALL_COMP_GRANULE_ADDRS
- MMAN dump all granule addresses of all components (no level)
DUMP_ALL_COMP_GRANULES
- MMAN dump all granules of all components (1 for partial list)
DUMP_ALL_REQS
- MMAN dump all pending memory requests to alert log
DUMP_TRANSFER_OPS - MMAN dump transfer and resize operations history
DUMP_ADV_SNAPSHOTS
- MMAN dump all snapshots of advisories (level unused)
CONTROLF - DuMP control file info
FLUSH_CACHE
- Flush buffer cache without shuting down the instance
SET_AFN - Set afn # for buffer flush (level = afn# )
SET_ISTEMPFILE
- Set istempfile for buffer flush (level = istempfile )
FLUSH_BUFFER - Reuse block range without flushing entire cache
BUFFERS - Dump all buffers in the buffer cache at level l
SET_TSN_P1
- Set tablespace # for buffer dump (level = ts# + 1)
BUFFER
- Dump all buffers for full relative dba <level> at lvl 10
BC_SANITY_CHECK
- Run buffer cache sanity check (level = 0xFF for full)
SET_NBLOCKS - Set number of blocks for range reuse checks
CHECK_ROREUSE_SANITY - Check range/object reuse sanity (level = ts#)
DUMP_PINNED_BUFFER_HISTORY
- kcb Dump pinned buffers history (level = # buffers)
REDOLOGS - Dump all online logs according to the level
LOGHIST
- Dump the log history (1: dump earliest/latest entries, >1: dump most recent 2**level entries)
REDOHDR - Dump redo log headers
LOCKS - Dump every lock element to the trace file
GC_ELEMENTS - Dump every lock element to the trace file
FILE_HDRS - Dump database file headers
FBINC
- Dump flashback logs of the current incarnation and all its ancestors.
FBHDR - Dump all the flashback logfile headers
FLASHBACK_GEN - Dump flashback generation state
KTPR_DEBUG
- Parallel txn recovery (1: cleanup check, 2: dump ptr reco ctx, 3: dump recent smon runs)
DUMP_TEMP - Dump temp space management state (no level)
DROP_SEGMENTS - Drop unused temporary segments
TREEDUMP
- Dump an index tree rooted at dba BLOCKDBA (<level>)
KDLIDMP - Dump 11glob inodes states (level = what to dump)
ROW_CACHE - Dump all cache objects
LIBRARY_CACHE
- Dump the library cache (level > 65535 => level = obj @)
CURSORDUMP - Dump session cursors
CURSOR_STATS - Dump all statistics information for cursors
SHARED_SERVER_STATE - Dump shared server state
LREG_STATE - Dump listener registration state
JAVAINFO - Dump Oracle Java VM
KXFPCLEARSTATS - Clear all Parallel Query messaging statistics
KXFPDUMPTRACE - Dump Parallel Query in-memory traces
KXFXSLAVESTATE
- Dump PX slave state (1: uga; 2: current cursor state; 3: all cursors)
KXFXCURSORSTATE - Dump PX slave cursor state
WORKAREATAB_DUMP - Dump SQL Memory Manager workarea table
OBJECT_CACHE - Dump the object cache
SAVEPOINTS - Dump savepoints
RULESETDUMP - Dump rule set
FAILOVER - Set condition failover immediate
OLAP_DUMP - Dump OLAP state
AWR_FLUSH_TABLE_ON
- Enable flush of table id <level> (ids in X$KEWRTB)
AWR_FLUSH_TABLE_OFF
- Disable flush of table id <level> (ids in X$KEWRTB)
AWR_DEBUG_FLUSH_TABLE_ON
- Enable debug of flushing the table with id <level> (ids in X$KEWRTB)
AWR_DEBUG_FLUSH_TABLE_OFF
- Disable debug of flushing the table with id <level> (ids in X$KEWRTB)
ASHDUMP - Dump ASH data (level = # of minutes)
ASHDUMPSECONDS - Dump ASH data (level = # of seconds)
HM_FW_TRACE - DIAG health monitor set tracing level
IR_FW_TRACE - DIAG intelligent repair set/clear trace
GWM_TRACE - Global Services Management set/clear trace
GWM_TEST - Global Services Management set/clear GDS test
GLOBAL_BUFFER_DUMP - Request global buffer dump (level 1 = TRUE)
KDFSDMP - Dump dbfs c-api states (level = what to dump)
DEAD_CLEANUP_STATE - Dump dead processes and killed sessions
IMDB_PINNED_BUFFER_HISTORY
- Dump IMDB pinned buffer history (level = (dump_level << 16 | num_buffers))
SLOCK_DUMP - Dump s-lock resources globally
STATE_OBJECT_DELETION_TIME - Dump state object deletion times
HEAPDUMP_ADDR
- Heap dump by address routine (level > 1 dump content)
POKE_ADDRESS - Poke specified address (level = value)
CURSORTRACE
- Trace cursor by hash value (hash value is address)
RULESETDUMP_ADDR - Dump rule set by address
kewmdump - Dump Metrics Metadata and Memory
con_id - Return Container Id as UB8
DBGT_SPLIT_CSTSTRING
DUMP_SWAP - dump system memory and swap information
ALERT_SWAP - issue alert message about system swap percentage
DUMP_PATCH - dump patch information
dumpBucketsRdbms
dumpKSIPCState - Dump KSIPC State
dumpKSTBuffers - Dump KST Buffers
rfrdm_dump_state - Dump DG Broker State
FLUSH_DBREPLAY_CAPTURE_BUFFER - Flush Database Replay capture buffer

Actions in library GENERIC:

---------------------------
xdb_dump_buckets
dumpKGERing - Dump contents of KGE ring buffer
dumpKGEIEParms - Dump memory around internal error parameters
dumpKGEState - Dump KGE state information for debugging

Actions in library CLIENT:

---------------------------
kpuActionDefault - dump OCI data
kpuActionSignalCrash
- crash and produce a core dump (if supported and possible)
kpudpaActionDpapi - DataPump dump action

oradebug doc component


Components in library DIAG:

--------------------------
diag_uts Unified Tracing Service (dbgt, dbga)
uts_vw UTS viewer toolkit (dbgtp, dbgtn)
diag_adr Automatic Diagnostic Repository (dbgr)
ams_comp ADR Meta-data Repository (dbgrm)
ame_comp ADR Export/Import Services (dbgre)
ami_comp ADR Incident Meta-data Services (dbgri)
diag_ads Diagnostic Directory and File Services (dbgrf, sdbgrf, sdbgrfu, sdbgrfb)
diag_hm Diagnostic Health Monitor ((null))
diag_ips Diagnostic Incident Packaging System ((null))
diag_dde Diagnostic Data Extractor (dbge)
diag_fmwk Diagnostic Framework (dbgc)
diag_ilcts Diagnostic Inter-Library Compile-time Service (dbgf)
diag_attr Diagnostic Attributes Management ((null))
diag_comp Diagnostic Components Management ((null))
diag_testp Diagnostic component test parent (dbgt)
diag_testc1 Diagnostic component test child 1 ((null))
diag_testc2 Diagnostic component test child 2 ((null))
KGSD Kernel Generic Service Debugging (kgsd)
diag_events Diagnostic Events (dbgd)
diag_adl Diagnostic ARB Alert Log (dbgrl, dbgrlr)
diag_vwk Diagnostic viewer toolkit (dbgv)
diag_vwk_parser Diagnostic viewer parser (dbgvp, dbgvl)
diag_vwk_uts Diagnostic viewer for UTS traces and files (dbgvf)
diag_vwk_ams Diagnostic viewer for AMS metadata (dbgvm)
diag_vwk_ci Diagnostic viewer for command line (dbgvci)
kghsc KGHSC Compact Stream (kghsc)
dbgxtk DBGXTK xml toolkit (dbgxtk)
KDIZOLTP ADV HIGH Index (kdizoltp)
Components in library RDBMS:
--------------------------
SQL_Compiler SQL Compiler ((null))
SQL_Parser SQL Parser (qcs)
SQL_Semantic SQL Semantic Analysis (kkm)
SQL_Optimizer SQL Optimizer ((null))
SQL_Transform SQL Transformation (kkq, vop, nso)
SQL_MVRW SQL Materialized View Rewrite ((null))
SQL_VMerge SQL View Merging (kkqvm)
SQL_Virtual SQL Virtual Column (qksvc, kkfi)
SQL_APA SQL Access Path Analysis (apa)
SQL_Costing SQL Cost-based Analysis (kko, kke)
SQL_Parallel_Optimization SQL Parallel Optimization (kkopq)
Vector_Processing Vector Processing (kkevp)
SQL_Plan_Management SQL Plan Managment (kkopm)
SQL_Plan_Directive SQL Plan Directive (qosd)
SQL_Optimizer_Stats SQL Optimizer Statistics (qos)
SQL_Optimizer_Stats_Advisor SQL Optimizer Statistics Advisor (qosadv)
SQL_Quarantine SQL Quarantine (qsfc)
SQL_Code_Generator SQL Code Generator (qka, qkn, qke, kkfd, qkx)
SQL_Parallel_Compilation SQL Parallel Compilation (kkfd)
SQL_Expression_Analysis SQL Expression Analysis (qke)
SQL_Rowsets_Setup SQL Rowsets Setup (qknr)
MPGE MPGE (qksctx)
ADS ADS (kkoads)
ICDT_Compile In Memory CDT Compilation (qks3t)
Shard_Sql Shard Sql (kkoshd)
AQP AQP (kkodp, qesdp, qersc)
PTF_Comp Polymorphic Table Functions Compilation (qksptf)
SQL_Execution SQL Execution (qer, qes, kx, qee)
Parallel_Execution Parallel Execution (qerpx, qertq, kxfr, kxfx, kxfq, kxfp)
PX_Messaging Parallel Execution Messaging (kxfp)
PX_Group Parallel Execution Slave Group (kxfp)
PX_Affinity Parallel Affinity (ksxa)
PX_Buffer Parallel Execution Buffers (kxfpb)
PX_Granule Parallel Execution Granules (kxfr)
PX_Control Parallel Execution Control (kxfx)
PX_Table_Queue Parallel Execution Table Queues (kxfq)
PX_Scheduler Parallel Execution Scheduler (qerpx)
PX_Queuing Parallel Execution Queuing (kxfxq)
PX_Blackbox Parallel Execution Blackbox (kxf)
PX_PTL Parallel Execution PTL (kxft)
PX_Expr_Eval Parallel Execution Expression Evaluation ((null))
PX_Selector Parallel Execution PX Selector (qerpsel)
PX_Overhead Parallel Execution Overhead (qerpx, kxfr, kxfx, kxfp)
Bloom_Filter Bloom Filter (qerbl, qesbl)
Time_Limit Query Execution Time Limit (opiexe, qerst)
ICDT_Exec In Memory CDT Execution (qes3t, kxtt)
EXECUTE_TEMP_TABLE Execute Temp Table (kxtt)
SQL_Trace SQL Trace (kxst, xpl)
PTF_Exec Polymorphic Table Functions Execution (qerptf)
Result_Cache Result Cache (qesrc, qerrc)
PGA_Manage PGA Memory Management ((null))
PGA_Compile PGA Memory Compilation ((null))
PGA_IMM PGA Memory Instance Manage ((null))
PGA_CMM PGA Memory Cursor Manage ((null))
PGA_ADV PGA Memory Advisor ((null))
rdbms_dde RDBMS Diagnostic Data Extractor (dbke)
VOS VOS (ks)
hang_analysis Hang Analysis (ksdhng)
background_proc Background Processes (ksb, ksbt)
system_param System Parameters (ksp, kspt)
ksu Kernel Service User (ksu)
ksutac KSU Timeout Actions ((null))
ksv_trace Kernel Services Slave Management (ksv)
file File I/O (ksfd, ksfdaf)
ksq Kernel Service Enqueues (ksq)
ksolt_trace Kernel Services Lightweight Threads (ksolt)
KSIM Kernel Service Instance Management (ksim)
KSIM_GRPOP Kernel Service Instance Management Group Operation ((null))
KSIPC VOS IPC (ksipc)
KSMSQ Message queue services (ksmsq)
KSMSQ_MQL Message Queueing Layer ((null))
KSRMA ksrma (ksrma)
KSRMF ksrmf (ksrmf)
KSIPC_AM Active Messaging ((null))
KSIPC_GRP KSIPC Group Services ((null))
KSIPC_SN KSIPC Shared Nothing ((null))
KSIPC_KV KSIPC Key Value ((null))
KSIPC_TOPO KSIPC Topology Services ((null))
KSIPC_PR KSIPC Path Record ((null))
KSIPC_IPCLW IPC LightWeight ((null))
KSIPC_IPCOR IPC Core Functionality ((null))
KSIPC_SHREG KSIPC Shared Registration ((null))
KSIPC_ASPC KSIPC Address space Map ((null))
KSIPCENV KSIPC Memory Allocations on Different Sockets (ksipcenv)
KSXP Cross Instance IPC (KSXP)
LREG Listener Registration (kml)
ksupd KSU Planned Draining (ksupd)
state_object State Objects (kss)
sess_signature Session Signature (ksusgn)
sql_mon SQL Monitor (keswx)
sql_mon_deamon SQL Monitor Deamon ((null))
sql_mon_query SQL Monitor Query ((null))
CACHE_RCV Cache Recovery (kcv, kct, kcra, kcrp, kcb)
DLF Delayed Log Force ((null))
MBR Multi-Block Read ((null))
DIRPATH_LOAD Direct Path Load (kl, kdbl, kpodp)
DIRPATH_LOAD_BIS Direct Path Kpodpbis Routine (kpodp)
RAC Real Application Clusters ((null))
GES Global Enqueue Service ((null))
KSI Kernel Service Instance locking (ksi)
RAC_ENQ Enqueue Operations ((null))
DD GES Deadlock Detection ((null))
RAC_BCAST Enqueue Broadcast Operations ((null))
RAC_FRZ DLM-Client Freeze/Unfreeze (kjfz)
KJOE DLM Omni Enqueue service (kjoe)
GCS Global Cache Service (kjb)
GCS_BSCN Broadcast SCN (kjb, kcrfw)
GCS_READMOSTLY GCS Read-mostly (kjb)
GCS_READER_BYPASS GCS Reader Bypass (kjb)
GCS_DELTAPUSH GCS Delta Push (kjb)
GCS_PCL2 Persistent cluster flash cache (kjbfp)
GSIPC Global Enqueue/Cache Service IPC ((null))
RAC_RCFG Reconfiguration ((null))
RAC_DRM Dynamic Remastering ((null))
RAC_MRDOM Multiple Recovery Domains ((null))
MRDOM_ESM Multiple/PDB Domains - Enter/Exit Server Mode (kjs, kjfz)
CGS Cluster Group Services (kjxg)
CGSIMR Instance Membership Recovery (kjxgr)
RAC_WLM Work Load Management (wlm)
RAC_MLMDS RAC Multiple LMS (kjm)
RAC_KA Kernel Accelerator (kjk)
RAC_LT RAC Latch Usage ((null))
RAC_SLOCK SLOCK Optimisation ((null))
KJSC RAC global stats (kjsc)
LMHB Heartbeat Monitoring (kjfm)
db_trace RDBMS server only tracing ((null))
kst server trace layer tracing (kst)
ddedmp RDBMS Diagnostic Data Extractor Dumper (dbked)
cursor Shared Cursor (kxs, kks)
Bind_Capture Bind Capture Tracing ((null))
KKSH Cursor Hash Table Tracing (kksh)
KSM Kernel Service Memory (ksm)
KSE Kernel Service Error Manager (kse)
explain SQL Explain Plan (xpl)
rdbms_event RDBMS Events (dbkd)
LOB_INODE Lob Inode (kdli)
rdbms_adr RDBMS ADR (dbkr)
ASM Automatic Storage Management (kf)
KFK KFK (kfk)
KFKIO KFK IO (kfkio)
KFKIO_COUNT KFK IO count ((null))
KFKIO_CANCEL KFK IO cancel ((null))
KFKIO_HARD KFK IO hard errors ((null))
KFKIO_SPLIT KFK IO split ((null))
KFKIO_MISC KFK IO miscellaneous ((null))
KFKSB KFK subs (kfksubs)
KFKSB_LIB KFK subs libraries ((null))
KFKSB_DM KFK subs disk management ((null))
KFKS KFK SAGE (kfks)
KFN ASM Networking subsystem (kfn)
KFNU ASM Umbillicus (kfnm, kfns, kfnb)
KFNS ASM Server networking (kfns)
KFNC ASM Client networking (kfnc)
KFNOR KFN orion (kfnor)
KFIS ASM Intelligent Storage interfaces (kfis)
KFM ASM Node Monitor Interface Implementation (kfm)
KFMD ASM Node Monitor Layer for Diskgroup Registration (kfmd)
KFMS ASM Node Monitor Layers Support Function Interface (kfms)
KFFB ASM Metadata Block (kffb)
KFFD ASM Metadata Directory (kffd)
KFZ ASM Zecurity subsystem (kfz)
KFC ASM Cache (kfc)
KFR ASM Recovery (kfr)
KFE ASM attributes (kfe)
KFDP ASM PST (kfdp)
KFG ASM diskgroups (kfg)
KFGB ASM diskgroups background (kfgb)
KFDS ASM staleness registry and resync (kfds)
KFIA ASM Remote (kfia)
KFIAS ASM IOServer (kfias)
KFIAC ASM IOServer client (kfiac)
KFFSCRUB ASM Scrubbing (kffscrub)
KFIO ASM translation I/O layer (kfio)
KFIOER ASM translation I/O layer (kfioer)
KFV ASM Volume subsystem (kfv)
KFVSU ASM Volume Umbilicus (kfvsu)
KFVSD ASM Volume Background (kfvsd)
KFVIOC ASM Volume Cell Storage (kfvioc)
KFDX ASM Exadata interface (kfdx)
KFZP ASM Password File Layer (kfzp)
KFA ASM Alias Operations (kfa)
KFF KFF (kff)
KFD ASM Disk (kfd)
KFDVA ASM Virtual ATB (kfdva)
KFTHA ASM Transparent High Availability (kftha)
KFFG ASM File Group (kffg)
KFFS ASM Split Mirror Operations (kffs)
KFIOS KFIO Simple (kfios)
DML DML Drivers (ins, del, upd)
Health_Monitor Health Monitor ((null))
DRA Data Repair Advisor ((null))
DIRACC Direct access to fixed tables (kqfd)
PART Partitioning (kkpo, qespc, qesma, kkpa, qergi)
PART_IntPart Interval Partitioning ((null))
PART_Dictionary Partitioning Dictionary (kkpod)
LOB_KDLW Lob kdlw (kdlw)
LOB_KDLX Lob xfm (kdlx)
LOB_KDLXDUP Lob dedup (kdlxdup)
LOB_KDLRCI Lob rci (kdlrci)
LOB_KDLA SecureFile Archive (kdla)
SQL_Manage SQL Manageability (kes)
SQL_Manage_Infra Other SQL Manageability Infrastructure (kesai, kesqs, kesatm, kesutl, kessi, keswat, keswts, keswsq)
SQL_Tune SQL Tuning Advisor (kest)
SQL_Tune_Auto SQL Tuning Advisor (auto-tune) (kestsa)
Auto_Tune_Opt Auto Tuning Optimizer (kkoat)
SQL_Tune_Index SQL Tuning Advisor (index-tune) (kestsi)
SQL_Tune_Plan SQL Tuning Advisor (plan node analysis) (kestsp)
SQL_Tune_Px SQL Tuning Advisor (parallel execution) (kestsa)
SQL_Tune_Fr SQL Tuning Advisor (fix regression) (kestsa)
SQL_Tune_Stats SQL Tuning Advisor (statistics advisor) (kestss)
SQL_Tune_Exa SQL Tuning Advisor (Exadata) (kestsaExa)
SQL_Test_Exec SQL Test-Execute Service (kestse)
SQL_Perf SQL Performance Analyzer (kesp, keswpi)
SQL_Repair SQL Repair Advisor (kesds)
SQL_trace_parser SQL trace parser (kesstp)
SQL_Analyze SQL Analyze (qksan)
SQL_DS SQL Dynamic Sampling Services (qksds)
SQL_DDL SQL DDL (atb, ctc, dtb)
KKP KKP Manager (kkp)
RAT_WCR Real Application Test: Workload Capture and Replay (kec)
Spatial Spatial (md)
Spatial_IND Spatial Indexing (mdr)
Spatial_GR Spatial GeoRaster (mdgr)
Text Text (dr)
rdbms_gc RDBMS Diagnostic Generic Configuration (dbkgc)
XS XS Fusion Security (kzx)
XSSESSION XS Session (kzxs)
XSPRINCIPAL XS Principal (kzxu)
XSSECCLASS XS Security Class (kzxc, kzxsp)
XSXDS XS Data Security (kzxd)
XSVPD XS VPD ((null))
XSXDB_DEFAULT XS XDB ((null))
XS_MIDTIER XS Midtier (kpuzxs)
XSNSTEMPLATE XS Namespace template (kzxnt)
XSACL XS ACL (kzxa)
XSADM XS Administrative operation (kzxm, kzxi)
AQ Streams Advanced Queuing (kwq, kkcn, kpon, kpoaq, kpce, kpcm, kpun, kpuaq, kws)
AQ_DEQ Streams Advanced Queuing Dequeue (kwqid, kwqdl)
AQ_BACK Streams Advanced Queueing Background (kwsbg, kwsbsm)
AQ_TM Streams Advanced Queuing Time Manager (kwqit, kwqmn)
AQ_CP Streams Advanced Queuing Cross Process (kwscp, kwsipc)
AQ_LB Streams Advanced Queuing Load Balancer (kwslb, kwslbbg)
AQ_NTFN Streams Advanced Queuing Notification (kpond, kkcne)
AQ_NTFNP12C Streams Advanced Queuing pre-12c Notification (kwqic)
AQ_TMSQ Streams Advanced Queuing Time Manager for Sharded Queue (kwsbtm, kwsbjc, kwsbit)
AQ_MC Streams Advanced Queuing Message Cache (kwsmc, kwssh, kwsmb, kwsmsg, kwssb, kwschnk, kwscb, kwsdqwm, kwssbsh)
AQ_PT Streams Advanced Queuing Partitioning (kwspt)
AQ_SUB Streams Advanced Queuing Subscription (kwssi, kwssa, kwsnsm, kwsnsme)
AQ_OPT Streams Advanced Queuing OPT (kwssub)
AQ_DLY Streams Advanced Queuing delay (kwsdly)
KSFM Kernel Service File Mapping (ksfm)
KXD Exadata specific Kernel modules (kxd)
KXDAM Exadata Disk Auto Manage (kxdam)
KCFIS Exadata Predicate Push (kcfis)
NSMTIO Trace Non Smart I/O (nsmtio)
KXDBIO Exadata Block level Intelligent Operations (kxdbio)
KXDRS Exadata Resilvering Layer (kxdrs)
KXDOFL Exadata Offload (kxdofl)
KXDMISC Exadata Misc (kxdmisc)
KXDCM Exadata Metrics Fixed Table Callbacks (kxdcm)
KXDBC Exadata Backup Compression for Backup Appliance (kxdbc)
DV Database Vault (kzv)
ASO Advanced Security Option ((null))
RADM Real-time Application-controlled Data Masking (kzradm)
SVRMAN Server Manageability (ke)
AWR Automatic Workload Repository (kew)
ASH Active Session History (kewa)
METRICS AWR metrics (kewm)
REPOSITORY AWR Repository (kewr)
FLUSH AWR Snapshot Flush (kewrf)
PURGE AWR Snapshot Purge (kewrps)
AWRUTL AWR Utilities (kewu)
AUTOTASK Automated Maintenance Tasks (ket)
MMON MMON/MMNL Infrastructure (keb)
SVRALRT Server Generated Alert Infrastructure (kel)
IUT IUT Infrastructure (keiut)
OLS Oracle Label Security (zll)
AUDITNG Database Audit Next Generation (aud, kza, kzft, aus, aop, ttp)
Configuration ANG Configuration (aud, kza, kzft, aus, aop, ttp)
QueueWrite ANG Queue Write (aud, kza, kzft, aus, aop, ttp)
FileWrite ANG File Write (aud, kza, kzft, aus, aop, ttp)
RecordCompose ANG Record Compose (aud, kza, kzft, aus, aop, ttp)
DBConsolidation ANG Database Consolidation (aud, kza, kzft, aus, aop, ttp)
SYS_Auditing ANG SYS Auditing (aud, kza, kzft, aus, aop, ttp)
KJCI KJCI Cross Instance Call (kjci)
KJZ KJZ - DIAG (kjz)
KJZC KJZC - DIAG Communication Layer (kjzc)
KJZD KJZD - DIAG Main Layer (kjzd)
KJZF KJZF - DIAG Flow Control Layer (kjzf)
KJZG KJZG - DIAG Group Services Layer (kjzg)
KJZH KJZH - DIAG API Layer (kjzh)
KJZM KJZM - DIAG Membership Layer (kjzm)
SEC Security (kz)
CBAC Code-Based Access Control (kzc)
VPD Virtual Private Database (kzr)
GDSI Generic Directory Services Integration (kzlg)
dbop DBOP monitoring (keomn)
dbop_gen DBOP generic service (keomg)
dbop_deamon DBOP monitoring Deamon (keomg)
dbop_comp DBOP composite type (keomm)
em_express EM Express (kex)
orarep orarep (ker)
Data Data Layer (kd, ka)

KDS Kernel Data Scan (kds)
KDSRID Fetch By Rowid (kdsgrp, kdsgnp)
KDSFTS Full Table Scan (kdsttgr, kdstgr)
KDSCLU Cluster Table Scan (kdsics, kdscgr)
KDWF Kernel Data Worker Framework (kdwf)
IMOLTP_KV InMemory OLTP KeyValue (kdkv)
MEMOPT_FI MEMOPTIMIZE FOR WRITE (kdfi)
RAT Real Application Testing (kec)
RAT_MASK Real Application Testing: Masking (kesm, kecprm)
BA Backup Appliance (kbrs)
KBC BA Containers (kbc)
connection_broker Connection Broker (kmp)
KRA Kernel Recovery Area Function (kra)
KRA_SQL KRA SQL Tracing ((null))
KRB Kernel Backup Restore (krb)
KRB_THREAD KRBBPC Thread Switches ((null))
KRB_IO KRB I/O ((null))
KRB_INCR KRB Incremental Restore ((null))
KRB_PERF KRB Performance Tracing ((null))
KRB_BPOUTPUT Detailed Backup Piece Output ((null))
KRB_BPVAL Detailed Block List During Restore Validate ((null))
KRB_FLWRES Details on Restore Flow ((null))
KRB_FLWCPY Details on krbydd Flow ((null))
KRB_FLWBCK Details on Backup Flow ((null))
KRB_FLWUSAGE RMAN Feature Usage ((null))
KRB_OPTIM Unused Space Compression ((null))
KRBABR Auto Block Media Recovery (krbabr)
KRC Recovery Block Change Tracking (krc)
KRC_CHANGES Recovery Block Change Tracking CHANGES ((null))
IM in-memory ((null))
IM_transaction IM transaction layer ((null))
IM_Txn_PJ IM Txn Private Journal (ktmpj)
IM_Txn_SJ IM Txn Shared Journal (ktmsj)
IM_Txn_JS IM Txn Journal Scan (ktmjs)
IM_Txn_Conc IM Txn Concurrency (ktmc)
IM_Txn_Blk IM Txn Block (ktmb)
IM_Txn_Read IM Txn Read (ktmr)
IM_Txn_ADG IM Txn ADG (ktma)
IM_space IM space layer ((null))
IM_data IM data layer (kdm)
IM_populate IM populating (kdml)
IM_background IM background (kdmr)
IM_scan IM scans ((null))
IM_journal IM journal ((null))
IM_dump IM dump ((null))
IM_FS IM faststart ((null))
IM_optimizer IM optimizer (kdmo)
IM_GD IM GD (kdmgd)
IM_ADO IM ADO (kdmado)
IM_IME IM IME (kdmime)
IM_Dictionary IM Dictionary and Cache (kdmd)
IM_Verification IM Verification (kdmv)
xdb_wallet XDB Wallet (kzs)
PROGINT Programmatic Interfaces (kp)
OCI OCI (oci, kpk, kpn)
OPI OPI (opi)
RPI RPI (rpi, kpr)
NPI NPI (npi, nco, kpfs)
Two_Task Two Task (osn, ksn)
PROGINT_PLSQL Programmatic Interfaces to/from PL/SQL (kkx, psd, pckl, plswa)
Two_Phase Two-phase commit (k2)
Conn_Pool Connection Pool (kppl)
TSM Transparent Session Migration (kpm, kps)
PROGINT_MISC Progint Miscellaneous (kpo, kpbf, kpin)
KDFS DBFS C-API (kdfs)
OLAP OLAP Analytic Workspace (xs)
OLAP_Paging OLAP Paging Manager (xspg, xsdb)
STREAMS Replication (knl)
STREAMS_CAPTURE Streams Capture (knlc)
STREAMS_PROPAGATION Streams Propagation ((null))
STREAMS_PROPAGATION_SENDER Streams Propagation Sender (knlcsv)
STREAMS_PROPAGATION_RECEIVER Streams Propagation Receiver (knanr)
STREAMS_APPLY Streams Apply (kna)
STREAMS_APPLY_COORDINATOR Streams Apply Coordinator (knac)
STREAMS_APPLY_READER Streams Apply Reader (knalf)
STREAMS_APPLY_SERVER Streams Apply Server (knas)
Space Space (ktd, kte, kts, ktf, ktt)
Search_Cache Search Cache (ktspsc)
CONTEXT Oracle Text (dr, kci)
CONTEXT_INDEX Oracle Text Index (drs, drg, drn, drl, dre)
CONTEXT_QUERY Oracle Text Query (drex, drp, dry, drw)
rdbms_uts RDBMS Unified Tracing Service (UTS) (dbkt)
MIRA_RCV MIRA Recovery (krd, krp, krr)
reg_cache Registration Caches (kssmc)
OFS Kernel Services Oracle File System ((null))
OFS_KSFS OFS File System (ksfs)
OFS_KSFSD OFS File System Dispatcher (ksfsd)
OFS_KSFSM OFS File System Metadata (ksfsm)
OFS_KSFSC1 OFS File System Callback (ksfsc1)
OFS_SKOFS OFS File System Implementation (skofs)
SQL_Compare_Plans SQL Compare Plans (keplan)

Components in library GENERIC:

--------------------------
Generic_VOS Generic VOS ((null))
VOS_Heap_Manager VOS Heap Manager ((null))
VOS_Latches VOS Latches ((null))
VOS_GST VOS Generic Stack Trace (kgds)
XML XML (qmxt, qmxq)
Generic_XDB Generic XDB ((null))
XDB_Repository XDB Repository (qme)
XDB_Protocols XDB Protocols (qmh, qmf, qmn)
XDB_Query XDB Query (qmxtra, qerxp)
XDB_XMLIndex XDB XMLIndex (qmxtri, qmixq)
XDB_Schema XDB Schema (qmxtri, qmixq)
XDB_XOB XDB XOB (qmx)
XDB_CSX XDB CSX (qmcx)
XDB_CSX_ENCODING XDB CSX ENCODING (qmcxe, qmcxm)
XDB_CSX_DECODING XDB CSX DECODING (qmcxd)
XDB_CSX_SELFCON XDB CSX SELFCON (qmcxe)
XDB_CSX_PRINT XDB CSX PRINT (qmxrs, qmxp, kolace, kolars)
XDB_CSX_TOKENMGR XDB CSX TOKENMGR (qmtm)
XDB_Default XDB Default ((null))
XDB_MEMORY XDB MEMORY (qmu)
LOB LOB (koll, kola)
LOB_Refcount LOB Refcount (kolr)
LOB_Default LOB Default (koll, kole, kokl, koxs)
LOB_DBLINK LOB DBLink (koll, kokl, kpolobr, kkxlr)
LOB_TEMPORARY LOB Temporary (kola, koklt)
LOB_API LOB API (kpolob, kkxlr, kokle, koklc)
KGH KGH Memory Allocator (kgh)
KGF ASM Automatic Storage Management (kgf)
LIBCACHE LIBCACHE (kgl, kql)
LCO_Status_DDL Change of Lib-Cache-Obj Status during DDL ((null))
OBJECTS OBJECTS ((null))
OBJECTS_DDL OBJECTS DDL (kokt)
OBJECTS_Types OBJECTS Types (kot, ko, ort)
OBJECTS_Images OBJECTS Images (koke, kot, kad)
OBJECTS_Anydata OBJECTS Anydata (kokla, kolo, kot, kad)
OBJECTS_Streams OBJECTS Streams (koxs)
OBJECTS_Dict OBJECTS Dictionary (kkdo, qcdo)
OBJECTS_Semanal OBJECTS Semantic Analysis (koks, qcso, qcto)
OBJECTS_Default OBJECTS Default ((null))
OBJECTS_Tables OBJECTS Tables (kkbo)
KGFPM PATCH repository (kgfpm)
KGFDVF Voting File Interface (kgfdvf)
Shared_Objects Shared Object Manager (pes)
SO_Loader Native Code Loader (pesld)
Direct_NFS Direct NFS Client (kgnfs, kgodm, skgnfs)
KGSC Session Cache for Cursors ((null))
KGSC_KQD Session Cache for KQD Cursors ((null))
KGSC_BUN Session Cache for KQD Bundled Cursors ((null))
KGSC_KKS Session Cache for KKS Cursors ((null))
KGSC_PLS Session Cache for PLSQL Cursors ((null))
KGSC_CONS Session Cache for Constraint Cursors ((null))
KGSC_TRIG Session Cache for Trigger Cursors ((null))
KGSC_JOX Session Cache for JOX shortname translation Cursors ((null))
KDI Index Layer (kdi)
KDIADHI Index Layer Compress Advanced High ((null))
KDIADLO Index Layer Compress Advanced Low ((null))
KDIL Index Load (kdil)
KDILADHI Index Load Compress Advanced High ((null))
KDILADLO Index Load Compress Advanced Low ((null))
KDIS Index Split (kdis)
KDISADHI Index Split Compress Advanced High ((null))
KDISADLO Index Split Compress Advanced Low ((null))
KDXT Index Reorg (kdxt)
KDXTADHI Index ADLO Compress Advanced High ((null))
KDXTADLO Index Reorg Compress Advanced Low ((null))
KDIM Index Coalesce (kdim)
KDIMADHI Index Coalesece Compress Advanced High ((null))
KDIMADLO Index Coalesece Compress Advanced Low ((null))
KDIF Index Scan (kdif)
KDIFADHI Index Scan Compress Advanced High ((null))
KDIFADLO Index Scan Compress Advanced Low ((null))
KDIZADHI Advanced High Index Codec (kdizoltp)
KDIZADLO Advanced Low Index Codec ((null))
KDIDML Index DML ((null))
KDIDMLADHI Index DML Compress Advanced High ((null))
KDIDMLADLO Index DML Compress Advanced Low ((null))
KDIGDR Index Ghost Data Removal ((null))
KDIGDRBR Index Ghost Data Removal Branch ((null))
KDIGDRLF Index Ghost Data Removal Leaf ((null))

Components in library CLIENT:

--------------------------
Client_KPU Client KPU ((null))
KPU_Memory KPU Memory ((null))
KPU_TTC KPU TTC ((null))
KPU_Relational KPU Relational ((null))
KPU_Objects KPU Objects ((null))
KPU_LOBS KPU LOBS ((null))
progint_appcont Prog Interfaces Application Continuity ((null))
progint_appcont_rdbms Prog Interfaces Application Continuity RDBMS-side ((null))
SQLLDR_Load SQLLDR Load (ul)
DPAPI_Load DPAPI Load (kpudp)
dbfs_client_profiler dbfs_client profiler ((null))

Components in library LIBCELL:

--------------------------
Client_Library Client Library ((null))
Disk_Layer Disk Layer ((null))
Network_Layer Network Layer ((null))
Connect_Layer Connect Layer ((null))
IPC_Layer IPC Layer ((null))

Components in library ORANET:

--------------------------
TNSLSNR OraNet Listener ((null))
NET_NSGR Network Service Generic Registration ((null))
NET_NSGI TNI Network Service Generic Listener User-defined class ((null))
CMAN OraNet Connection Manager ((null))
NET OraNet Services ((null))
NET_NI Network Interface Layer ((null))
NET_NS Network Session Layer ((null))
NET_NT Network Transport Layer ((null))
NET_NTM Network Transport Mailbox Layer ((null))
NET_NTP Network Transport IPC Layer ((null))
NET_NTT Network Transport TCP/IP Layer ((null))
NET_NTUS Network Transport Unix Domain Sockets Layer ((null))
NET_NL Network Library ((null))
NET_NA Network Authentication ((null))
NET_NZ Network Zecurity ((null))
NET_NTZ Network SSL ((null))
NET_NU Network Trace Route Utility ((null))
NET_NN Network Names ((null))

Components in library ADVCMP:

--------------------------
ADVCMP_MAIN Archive Compression (kdz)
ADVCMP_COMP Archive Compression: Compression (kdzc, kdzh, kdza)
ADVCMP_DECOMP Archive Compression: Decompression (kdzd, kdzs)
ADVCMP_DECOMP_HPK Archive Compression: HPK (kdzk)
ADVCMP_DECOMP_PCODE Archive Compression: Pcode (kdp)

Components in library PLSQL:

--------------------------
PLSQL_Apps PL/SQL Apps (di, pi, plitblm, scm, std, textio, wpiutil)
PLSQL_Utl_File PL/SQL Utl File (pifi)
PLSQL_Codegen PL/SQL Codegen ((null))
PLSQL_COG_IDL_Gen PL/SQL Codegen IDL Gen (pdw)
PLSQL_COG_Infrastructure PL/SQL Codegen Infrastructure (pdz)
PLSQL_COG_Native PL/SQL Codegen Native (pdn)
PLSQL_COG_Optimizer PL/SQL Codegen Optimizer (pdx)
PLSQL_COG_MCode_Gen PL/SQL Codegen MCode Gen (pdy)
PLSQL_Code_Execution PL/SQL Code Execution (pb, pd, pe, pf, plst, pri)
PLSQL_Stack_Frames PL/SQL Runtime Stack Frame (pfrstk)
PLSQL_External_Proc PL/SQL External Proc (pef, ociextp)
PLSQL_IDL PL/SQL IDL (pdt, pt)
PLSQL_ILMS PL/SQL ILMS (pgm)
PLSQL_KNLDE PL/SQL KNLDE (pbbicd, pbp3g, pbs, pbt3g, peibp)
PLSQL_KG_Interface PL/SQL KG Interface (bam, hep, hsh, lis, par, phdr, pk)
PLSQL_Infrastructure PL/SQL Infrastructure (pci, pcm, ph, pl, pncutil, pp, ps, pu, tre)
PLSQL_PSD PL/SQL PSD ((null))
PLSQL_PSD_Generic PL/SQL PSD Generic (psd, pso, psu)
PLSQL_PSD_Standalones PL/SQL PSD Standalones (p2c, pls, psx)
PLSQL_Semantics PL/SQL Semantics (ph2, pha, phn)
PLSQL_Syntax PL/SQL Syntax (ph1)

Components in library XDK:

--------------------------
XDK_Parser XDK Parser ((null))
XDK_FSM_Parser XDK FSM Parser (LpxFSM)
XDK_Schema_Validator XDK Schema Validator ((null))
XDK_Xslt_Engine XDK Xslt Engine ((null))

Components in library VPLIB:

--------------------------
Vector_Translate Vector Translate (qkaxl, qerxl, qesxl, evaxl)
Vector_Aggregate Vector Aggregate (qergv, qesgv, evavgby)
Vector_PX Vector PX (qesxlp)

ORA-21700 in data dictionary

$
0
0

This whole story started with a nice little ticket:

When I try to edit a scheduler job in <DB>  with user <USER> I am getting error:
ORA-21700: object does not exist or is marked for delete
even I can see the object and the object is not deleted or marked as delete.

At first I could not reproduce the error, so I asked for something more specific, and I got 2 screenshots. (one shown here)
Unfortunately (for me) the screenshots were from Toad and PL/SQL Developer - 2 tools I don't have install or licensed or know anything about.






So, to give it a try I connected to the DB with SQL Developer and randomly clicked around several scheduler related tabs. I was lucky to generate ORA-21700 there also. And the big value of SQL Developer in this case is the log it has for all SQLs it's sending to the DB.




It shows the statement which failed - together with it's arguments.


select job_name, argument_name, argument_position, argument_type, value, out_argument 
from SYS.Dba_SCHEDULER_JOB_ARGS
WHERE "JOB_NAME" = :OBJECT_NAME AND "OWNER" = :OBJECT_OWNER
order by argument_position

With this query it's much easier to analyse the problem. Executing the quera ys the specific user really created an ORA-21700. But where does it come from? DBMS_UTILITY.EXPAND_SQL_TEXT is very handy to get the "real" sql for a query with views. Of course it's possible to see the views text, but if it's using views again, this can be real work (which I try to avoid).

Trying all the tables in the expanded query the culprit was found easily: sys.SCHEDULER$_JOB_ARGUMENT
The table is not complicated:
NAME            DATA TYPE            NULL  DEFAULT    COMMENTS
*OID NUMBER No
*POSITION NUMBER No
NAME VARCHAR2(128 BYTE) Yes
TYPE_NUMBER NUMBER Yes
USER_TYPE_NUM NUMBER Yes
VALUE ANYDATA Yes
FLAGS NUMBER Yes

And with some more trying the problem was tracked down to 1 column in 1 row:
select VALUE from sys.SCHEDULER$_JOB_ARGUMENT where oid=2736824 

This shows a little misunderstanding in the first request, the error code and the problem. The full error description is
ORA-21700: object does not exist or is marked for delete
21700. 00000 - "object does not exist or is marked for delete"
*Cause: User attempted to perform an inappropriate operation to
an object that is non-existent or marked for delete.
Operations such as pinning, deleting and updating cannot be
applied to an object that is non-existent or marked for delete.
*Action: User needs to re-initialize the reference to reference an
existent object or the user needs to unmark the object.

The objects it is referring to are things like a TYPE, whereas in the original content it was interpreted as a SCHEDULER JOB.

But why is there a problem with an anydata column at all? couldn't it hold simply ANY data?
there is a Note at MOS which describes the situation quite good: Used Type Can Be Dropped Causing ORA-21700 When Querying A Table With SYS.ANYDATA Column ( Doc ID 1503628.1 )
When selecting an anydata column which contains atype, the given TYPE must be known. If the TYPE was deleted in the meantime, the column (and related functions like anydata.gettypename) will fail.
The block or row itself is not corrupted and it can be queried with dump().
Typ=58 Len=630: 0,1,0,0,0,0,0,1,0,0,5,189,61,90,2,96,9,0,0,0,0,0,2,80,0,0,0,0,0,...
Unfortunately I don't know the internal structure of this Type. Datatype 58 is opaque (DTYOPQ) which can hold ANYDATA, ANYTYPE, XMLTYPE, ANYDATASET (and maybe others). But how to distinguish which exactly, or how to interpred the bytes I got from dump() I couldn't find somewhere.

The problem in this particular database is this one row in sys.SCHEDULER$_JOB_ARGUMENT. To be visible in Dba_SCHEDULER_JOB_ARGS, there is a join of SCHEDULER$_JOB_ARGUMENT and SCHEDULER$_JOB on OID = OBJ#.
In my particular case there is no row anymore on SCHEDULER$_JOB. So no row is visible and there is nothing which can be dropped with DBMS_SCHEDULER.DROP_JOB. I have no idea how this happened. But it's there.

That's the situation to create a SR at MOS and ask for assistance.

Even a testcase is simply done:
CREATE OR REPLACE TYPE random_type AS OBJECT (
n NUMBER
);
/

CREATE OR REPLACE PROCEDURE do_nothing (
param1 IN random_type
) AS
BEGIN
NULL;
END do_nothing;
/

DECLARE
t random_type;
BEGIN
t := random_type(NULL);
t.n := 1;
do_nothing(t);
END;
/

BEGIN
dbms_scheduler.create_job(job_name => 'TEST_NOTHING', job_type => 'STORED_PROCEDURE', job_action => 'DO_NOTHING', number_of_arguments
=> 1, start_date => SYSDATE + 1, repeat_interval => NULL, end_date => NULL, enabled => false,
auto_drop => true,
comments => 'job to test parameter of ANYDATA'
);
END;
/

DECLARE
t random_type;
BEGIN
t := random_type(NULL);
t.n := 1;
dbms_scheduler.set_job_anydata_value(job_name => 'TEST_NOTHING', argument_position => 1, argument_value => SYS.anydata.convertobject
(t) );

END;
/

SELECT
sys.anydata.gettypename(value) d
FROM
sys.scheduler$_job_argument;

DROP TYPE random_type;

SELECT
sys.anydata.gettypename(value) d
FROM
sys.scheduler$_job_argument;

That's exactly where the ORA-21700 occurs.
In the testcase, a simple cleanup can be done by (trying to) execute this job, so it's cleared:

PAUSE "let's initiate some cleanup"

begin
dbms_scheduler.set_attribute(name => '"BERX"."TEST_NOTHING"', attribute => 'logging_level', value => dbms_scheduler.logging_full);
dbms_scheduler.set_attribute_null(name => '"BERX"."TEST_NOTHING"', attribute => 'start_date');
dbms_scheduler.enable(name => '"BERX"."TEST_NOTHING"');
end;
/

PAUSE "sleep for some time until the job is executed"

SELECT
sys.anydata.gettypename(value) d
FROM
sys.scheduler$_job_argument;

SELECT
*
FROM
dba_scheduler_job_log
WHERE
owner = user;

All I can do now is to convince an MOS to approve me deleting the problematic row.

possible changes in Oracle 20

$
0
0
As this post is written in July 2019, it's highly speculative writing about changes in Oracle version 20.
But at least it's based on some information available from Oracle.

A new autoupgrade.jar version is available at AutoUpgrade Tool (Doc ID 2485457.1). In this Note, there is also a BUGS_20190715.txt which contains (among others):
AUPG-955 Disable NONCDB validation when upgrading a DB to 20c
AUPG-975 upgrade from 12.1.0.2 to 20 is not supported

Based on that I assume, noncdb will not be officially supported anymore, but maybe it's not supported in the autoupgrade.jar, but manual steps are still allowed & supported - we will se.
Also a direct upgrade from 12.1 to 20 is out of scope of this tool.

But there is more information available.
in the autoupgrade.jar in directory resources/preupgrade/config/ there is a file parameters20.properties. It contains information about deprecated/obsolete/removed parameters.

The changes in comparison to 19 I found:

from deprecated to removed:
unified_audit_sga_queue_size

from deprecated to obsolete:
cluster_database_instances

new parameters and their status:

PARAMETERSTATUS
_arch_sim_moderemoved
_gc_async_receiveremoved
_hcs_disable_sp_jback_optremoved
_autonomous_performance_features_indexremoved
_cgs_os_level_connection_reqnodeprecated
_autonomous_performance_features_statsremoved
_px_monitor_loadobsolete
_hcs_disable_level_prune_vis_lvsremoved
_hcs_disable_level_prune_gbyremoved
_asm_enable_parity_datafile_creationremoved
_part_access_version_by_numberobsolete
_tablespace_encryption_default_algorithmremoved
_hcs_disable_cell_qry_meas_prune_optremoved
_unified_audit_flush_thresholdremoved
_optimizer_stats_on_conventional_dml_configremoved
_reset_maxcap_historyremoved
_ldr_io_size2obsolete
_hang_long_wait_time_thresholdremoved
_read_mostly_instancedeprecated
_cgs_os_level_connection_pthreadnoremoved
_max_cr_rollbacksdeprecated
_hcs_enable_multi_parent_genremoved
_disable_thread_snapshotremoved
_hcs_disable_tgt_depths_optremoved
_enable_imc_miraremoved
_log_archive_avoid_memcpyremoved
_gcs_min_slavesdeprecated
_hcs_enable_dynamic_cacheremoved
_unified_audit_flush_intervalremoved
_hcs_disable_hord_in_oby_pruneremoved
_ta_lns_wait_for_arch_logremoved
_hcs_disable_jback_opt_for_hord_in_obyremoved
_show_mgd_recovery_stateremoved
_hcs_disable_pushed_preds_in_gen_sqlremoved
_hcs_disable_cell_qry_tmplsremoved
_test_param_5deprecated
_autonomous_performance_featuresremoved
_session_percent_thresholdremoved
_ash_compression_enabledeprecated
_kebm_max_parallel_intensive_autotasksremoved


let's see if this will change in next iterations.

The biggest value of such sources (for me) is the reliability. It's not something a vendor wants the customers to know/act on (like a "what changed" document would be); it's the information required by tools, so no political filters are there.

Simple getTracefile

$
0
0
Tracefiles are a very valuable source of information in Oracle databases.
In Versions prior to 12.2 quite complicated objects (with java dependency) were required.
Fortunately, in 12.2 Oracle introduced (& documented) the View V$DIAG_TRACE_FILE_CONTENTS. Unfortunately for some (only good from an internal technical perspective) reasons it does not contain the file itself, or a LOB locator, but several chunks as VARCHAR2. So it's still worth to write some code for a simple interface to access the tracefiles.

Again it's a SQLDeveloper report. But the structure is so simple, it can be implemented in any preferred tool.



There is a simple list of tracefiles and their MODIFY_TIME as FILEDATE - based on V$DIAG_TRACE_FILE.



When a specific tracefile name is selected, the file is presented as text and BLOB. The content is identical, just BLOBs can be downloaded which is useful for huge tracefiles, whereas the CLOB representation is simple to directly read small tracefiles.


Fortunately no objects are required in the database, only proper grants on V$DIAG_TRACE_FILE and V$DIAG_TRACE_FILE_CONTENTS. These can be roles also.

As I have limited access to databases right now, I could not test a lot of settings, e.g. RAC or different character sets.

Just import this Report as an User Defined Report:

<?xml version="1.0" encoding="UTF-8" ?>
<displays>
<display id="04e5e018-014a-1000-8001-ac193119805c" type="" style="Table" enable="true">
<name><![CDATA[get tracefiles]]></name>
<description><![CDATA[create a list of tracefiles as listed in V$DIAG_TRACE_FILE and makes specific files available]]></description>
<tooltip><![CDATA[create a list of tracefiles in DIAG directory and makes specific files available ]]></tooltip>
<drillclass><![CDATA[null]]></drillclass>
<CustomValues>
<TYPE>horizontal</TYPE>
</CustomValues>
<query>
<sql><![CDATA[select TRACE_FILENAME as TRACE_NAME ,
MODIFY_TIME as FILEDATE
from V$DIAG_TRACE_FILE
Order By MODIFY_TIME desc]]></sql>
</query>
<pdf version="VERSION_1_7" compression="CONTENT">
<docproperty title="-" author="berx" subject="-" keywords="-" />
<cell toppadding="2" bottompadding="2" leftpadding="2" rightpadding="2" horizontalalign="LEFT" verticalalign="TOP" wrap="true" />
<column>
<heading font="null" size="10" style="NORMAL" color="-16777216" rowshading="-1" labeling="FIRST_PAGE" />
<footing font="null" size="10" style="NORMAL" color="-16777216" rowshading="-1" labeling="NONE" />
<blob blob="EMBED" zip="false" />
</column>
<table font="Agency FB" size="10" style="NORMAL" color="-16777216" userowshading="false" oddrowshading="-1" evenrowshading="-1" showborders="true" spacingbefore="12" spacingafter="12" horizontalalign="LEFT" />
<header enable="false" generatedate="false">
<data>

null </data>
</header>
<footer enable="false" generatedate="false">
<data value="null" />
</footer>
<security enable="false" useopenpassword="false" openpassword="null" encryption="EXCLUDE_METADATA">
<permission enable="false" permissionpassword="null" allowcopying="true" allowprinting="true" allowupdating="false" allowaccessdevices="true" />
</security>
<pagesetup papersize="LETTER" orientation="1" measurement="in" margintop="1.0" marginbottom="1.0" marginleft="1.0" marginright="1.0" />
</pdf>
<display id="null" type="" style="Table" enable="true">
<name><![CDATA[get tracefile]]></name>
<description><![CDATA[access the specific tracefile ]]></description>
<tooltip><![CDATA[Specific tracefile from list above ]]></tooltip>
<drillclass><![CDATA[null]]></drillclass>
<CustomValues>
<TYPE>horizontal</TYPE>
</CustomValues>
<query>
<sql><![CDATA[with function get_tracefile (filename varchar2) return CLOB IS
-- PRAGMA AUTONOMOUS_TRANSACTION;
-- to avoid ORA-14553: cannot perform a lob write operation inside a query
-- only required in dedicated function
--
-- in a dedicated function, this grant is required:
-- grant select on sys.V_$DIAG_TRACE_FILE_CONTENTS to xxx;
l_out CLOB := EMPTY_CLOB;
CURSOR all_payload (fn varchar2)
is
Select Payload
from V$DIAG_TRACE_FILE_CONTENTS
where trace_filename=fn
and payload is not null
order by line_number asc;
begin
NULL;

DBMS_LOB.CREATETEMPORARY(l_out, FALSE, DBMS_LOB.CALL);

for r_payload in all_payload(filename)
loop
DBMS_lob.append(l_out, r_payload.payload);
end loop;
return l_out;
DBMS_LOB.FREETEMPORARY(l_out);
end get_tracefile;
function get_tracefile_blob (filename varchar2) return BLOB IS
-- PRAGMA AUTONOMOUS_TRANSACTION;
-- to avoid ORA-14553: cannot perform a lob write operation inside a query
-- only required in dedicated function
--
-- in a dedicated function, this grant is required:
-- grant select on sys.V_$DIAG_TRACE_FILE_CONTENTS to xxx;
l_out BLOB := EMPTY_BLOB;
CURSOR all_payload (fn varchar2)
is
Select Payload
from V$DIAG_TRACE_FILE_CONTENTS
where trace_filename=fn
and payload is not null
order by line_number asc;
begin
NULL;
DBMS_LOB.CREATETEMPORARY(l_out, FALSE, DBMS_LOB.CALL);
for r_payload in all_payload(filename)
loop
DBMS_lob.append(l_out, utl_raw.cast_to_raw(r_payload.payload));
end loop;
return l_out;
DBMS_LOB.FREETEMPORARY(l_out);
end get_tracefile_blob ;
select get_tracefile(:TRACE_NAME) as " Tracefile", get_tracefile_blob(:TRACE_NAME) as " Binary Tracefile " from dual]]></sql>
<binds>
<bind id="TRACE_NAME">
<prompt><![CDATA[TRACE_NAME]]></prompt>
<tooltip><![CDATA[TRACE_NAME]]></tooltip>
<value><![CDATA[NULL_VALUE]]></value>
<bracket><![CDATA[null]]></bracket>
</bind>
</binds>
</query>
<pdf version="VERSION_1_7" compression="CONTENT">
<docproperty title="-" author="-" subject="-" keywords="-" />
<cell toppadding="2" bottompadding="2" leftpadding="2" rightpadding="2" horizontalalign="LEFT" verticalalign="TOP" wrap="true" />
<column>
<heading font="null" size="10" style="NORMAL" color="-16777216" rowshading="-1" labeling="FIRST_PAGE" />
<footing font="null" size="10" style="NORMAL" color="-16777216" rowshading="-1" labeling="NONE" />
<blob blob="EMBED" zip="false" />
</column>
<table font="null" size="10" style="NORMAL" color="-16777216" userowshading="false" oddrowshading="-1" evenrowshading="-1" showborders="true" spacingbefore="12" spacingafter="12" horizontalalign="LEFT" />
<header enable="false" generatedate="false">
<data>

null </data>
</header>
<footer enable="false" generatedate="false">
<data value="null" />
</footer>
<security enable="false" useopenpassword="false" openpassword="null" encryption="EXCLUDE_METADATA">
<permission enable="false" permissionpassword="null" allowcopying="true" allowprinting="true" allowupdating="false" allowaccessdevices="false" />
</security>
<pagesetup papersize="LETTER" orientation="1" measurement="in" margintop="1.0" marginbottom="1.0" marginleft="1.0" marginright="1.0" />
</pdf>
</display>
</display>
</displays>

Any comments, hints or recommendations are highly welcome!

OGB Appreciation Day : manipulating execution plans

$
0
0
Timscall for OGB Appreciation Day 2019 (#ThanksOGB) comes in time this year.

Even unemployed at the moment, I had a chance to help a friend with some problematic SQL statements.

The statements are not that important at all but he possibilities we had to address the problems were great!


First of all an existing execution plan, together with session stats, ASH/AWR and in one case SQLTRACE. None of these sources of information shows all details we required, but all together provided sufficient information to understand the problem.
The probably most comprehensive way to get all these information (except tracing) is sqld360.

Next of course is the system of hints.
There are many discussions if hints are a good and bad thing. It's not me to argue it here.
I appreciate hints exist in Oracle RDBMS as they can be used if arguments in their favor are strong.

Third the possibilities to apply hints to a SQL are huge. Even the statement itself can not be changed, there are stored outlines, SQL Profile, SQL Plan Baselines in SQL Plan Management, SQL Patch, and if some really dirty tricks are required, SQL Translation Framework and other mean tools can be used.

So there are sufficient tools available to improve a statements execution.


The sheer amount of possibilities might be overwhelming or even deterrent. In fact it's not that complicated and results can be achieved in short time.
Writing this, it's always easy to deliver shiny fancy solutions when standing on giants shoulders.
To give some examples, here is a random selection of articles I used during the latest investigations:

Occurence - Jonathan Lewis
Fixing SQL Plans: The hard way – Part 1 - Advait Deo
SQL Profiles - Kerry Osborne
buffer sorts - Jonathan Lewis
Oracle’s OPT_ESTIMATE Hint: Usage Guide - Christo Kutrovsky
Visual SQL Tuning (VST) - Kyle Hailey

I'm grateful for the possibilities we have to fix SQL executions, and I'm also grateful for all those people who share their knowledge, so I can learn from them!

Oracle patches - some basics and good-to-know information

$
0
0
Oracle Patches can occur very complicated and confusing. Most of the time they are zip files with strange and sometimes inconsistent naming convention.
As these patches are processed by a program called opatch (which also can be obtained as a patch - but it does not follow the same structure as regular patches).

Back to basics: In patches words like product, platform or language are used. They are all IDs and their "data dictionary" can easily be obtained. There are these links:
productshttps://updates.oracle.com/Orion/Services/metadata?table=aru_products
releaseshttps://updates.oracle.com/Orion/Services/metadata?table=aru_releases
platformshttps://updates.oracle.com/Orion/Services/metadata?table=aru_platforms
languageshttps://updates.oracle.com/Orion/Services/metadata?table=aru_languages
product_groupshttps://updates.oracle.com/Orion/Services/metadata?table=aru_product_groups
product_releaseshttps://updates.oracle.com/Orion/Services/metadata?table=aru_product_releases
component_releaseshttps://updates.oracle.com/Orion/Services/metadata?table=aru_component_releases
aru targetshttps://updates.oracle.com/Orion/Services/metadata?table=aru_targets
componentshttps://updates.oracle.com/Orion/Services/components?ctype=product,release,platform,platform_version,wildcard-release,config-group,release-component,code_list
certificationshttps://updates.oracle.com/Orion/Services/components?ctype=certification
patch recommendationshttps://updates.oracle.com/Orion/Services/recommended_patches
ARU stands for Automated Release Update.

All these files can also obtained in a single zip file: https://updates.oracle.com/download/em_catalog.zip
The name of the zip file also shows it's origin purpose: provide Enterprise Managers recommended patches, download etc. with sufficient information. Some parts like platforms and languages are quite stable, others like releases and  recommendations, change regularly.

recommendations brings uns closer to real patches. it starts with
<results md5_sum="212e2bc3336d1aa3e41074e796201aa9">
<generated_date in_epoch_ms="1571568490000">2019-10-20 10:48:10</generated_date>
<patches md5_sum="d58c2ef61a596dd7bdef85b61cd06861">

followed by entries for each patch like
<patch uid="3d4a1929407b7e05430e2fd5bbc18481" has_prereqs="n" has_postreqs="n" is_system_patch="n">
<bug>
<number>30049606</number>
<abstract><![CDATA[WINDOWS DB BUNDLE PATCH 12.1.0.2.191015]]></abstract>
</bug>
<name>30049606</name>
<type>Patch</type>
<psu_bundle>Oracle 12.1.0.2.191015 for Windows</psu_bundle>
<status>Available</status>
<access id="m">Open access</access>
<url>
<patch_readme host="https://updates.oracle.com"><![CDATA[/Orion/Services/download?type=readme&aru=23166562]]></patch_readme>
<patch_details><![CDATA[/download/30049606.html]]></patch_details>
</url>
<request_id>23166562</request_id>
<urm_components>
<qparts ctype_id="201">
<qpart cid="32615" version="Q5"><![CDATA[Oracle Database - Enterprise Edition]]></qpart>
<qpart cid="42353" version="Q790"><![CDATA[Oracle Server - Standard Edition]]></qpart>
</qparts>
<urm_releases ctype_id="5">
<urm_release cid="86176" version="12.1.0.2.0"><![CDATA[Oracle Database]]></urm_release>
</urm_releases>
</urm_components>
<product id="9480" bugdb_id="5"><![CDATA[Oracle Database Family]]></product>
<release id="600000000009300" name="12.1.0.2.0" platform_patch_not_required="Y" cc="Y"><![CDATA[Oracle 12.1.0.2.0]]></release>
<platform id="912" bugdb_id="912"><![CDATA[Microsoft Windows (32-bit)]]></platform>
<language id="0" iso_code="EN"><![CDATA[American English]]></language>
<translations_available>No</translations_available>
<classification id="185">Security</classification>
<patch_classification id="185">Security</patch_classification>
<life_cycle id="175">Recommended</life_cycle>
<support_level id="E">Extended Support</support_level>
<entitlements>
<entitlement code="EXS"/>
</entitlements>
<target_types>
<target_type>
<aru_target><![CDATA[All Database]]></aru_target>
<em_tag></em_tag>
</target_type>
</target_types>
<fixes_bugs truncated="no">
<bug>
<number>12681464</number>
<abstract><![CDATA[UPDATE STATEMENT FAILED - ORA7445 [KGGSLDESTROY] ERROR]]></abstract>
</bug>
<bug>
<number>12799826</number>
<abstract><![CDATA[LOCK TABLE FOR INDEX FAILS WITH ORA-600 [KKBKAUXBLL0]]]></abstract>
</bug>
... many more
</fixes_bugs>
<size>317964815</size>
<files>
<file>
<name>p30049606_121020_WINNT.zip</name>
<size>317964815</size>
<download_url host="https://updates.oracle.com"><![CDATA[/Orion/Services/download/p30049606_121020_WINNT.zip?aru=23166562&patch_file=p30049606_121020_WINNT.zip]]></download_url>
<digest type="SHA-1">784196EE41085E633F1C10C5F12B47717232C3C5</digest>
<digest type="SHA-256">849D03DCBB4EDCD2F08474371309470E9D3AE172E4BD7296453F31AEAE40E9D7</digest>
</file>
</files>
<updated_date in_epoch_ms="1571299461000">2019-10-17 08:04:21</updated_date>
<released_date in_epoch_ms="1570628127000">2019-10-09 13:35:27</released_date>
</patch>

Many entries in this XML can be interpreted by the information provided in em_catalog.zip. For each patch there is a download link: https://updates.oracle.com/Orion/Services/download/p30049606_121020_WINNT.zip?aru=23166562&patch_file=p30049606_121020_WINNT.zip


As there are many patches which are not mentioned in recommendations, and also for them there is a link to get an equal XML regarding this patch:
https://updates.oracle.com/Orion/Services/search?bug=<patch_number>

Even the URL suggests it is a search for a BUG, it's a search for a PATCH (as a PATCH can contain fixes for multiple BUGs).

If the same information is required in a HTML form, the URL is
https://updates.oracle.com/Orion/PatchDetails/process_form?patch_num=<patch_number>

Oracle patches - a patchfiles anatomy

$
0
0
In my previous post I explained some basics about IDs of platforms, products and similar and where these patches can be obtained.

Today my focus is on the content of some patchfiles.

As I'm basically a DBA, my patchfiles are regarding database binaries. Patchfiles for other software products might differ, but I hope their main structure is similar.

A very simple patch is p26007010_12102180417ProactiveBP_Linux-x86-64.zip
it contains these files:
.
├── 26007010
│   ├── etc
│   │   └── config
│   │   ├── actions.xml
│   │   └── inventory.xml
│   ├── files
│   │   └── lib
│   │   └── libserver12.a
│   │   ├── ktsla.o
│   │   └── ktsld.o
│   └── README.txt
└── PatchSearch.xml

Let's start with PatchSearch.xml. This file is outside of the directory with patch number.

<!-- This file contain patch Metadata -->
<results md5_sum="911913c83e833a5b6cd893365066141c">
<generated_date in_epoch_ms="1538413221000">2018-10-01 17:00:21</generated_date>
<patch has_prereqs="n" has_postreqs="n" is_system_patch="n">
<bug>
<number>26007010</number>
<abstract><![CDATA[DATABASE GREW EVEN THOUGH RESOURCES DELETED]]></abstract>
</bug>
<name>26007010</name>
<type>Patch</type>
<status>Available</status>
<access id="m">Open access</access>
<url>
<patch_readme host="https://updates.oracle.com"><![CDATA[/Orion/Services/download?type=readme&aru=22425588]]></patch_readme>
<patch_details><![CDATA[/download/26007010.html]]></patch_details>
</url>
<request_id>22425588</request_id>
<product id="9480" bugdb_id="5"><![CDATA[Oracle Database Family]]></product>
<release id="6000000000093018041701" name="12.1.0.2.180417ProactiveBP" platform_patch_not_required="Y" psu="Y" cc="Y"><![CDATA[Oracle 12.1.0.2.180417 Proactive BP]]></release>
<platform id="226" bugdb_id="226"><![CDATA[Linux x86-64]]></platform>
<language id="0" iso_code="EN"><![CDATA[American English]]></language>
<translations_available>No</translations_available>
<classification id="174">General</classification>
<patch_classification id="174">General</patch_classification>
<support_level id="G">General Support</support_level>
<entitlements>
<entitlement code="SW"/>
</entitlements>
<patch_relationships>
<related_patch>
<bug>
<number>27338029</number>
<abstract><![CDATA[DATABASE BUNDLE PATCH 12.1.0.2.180417]]></abstract>
</bug>
<relation_types>
<relation_type>Prereq</relation_type>
</relation_types>
<name>27338029</name>
<type>Patch</type>
<patch_details><![CDATA[/Orion/Services/search?bug=27338029&release=600000000009300]]></patch_details>
</related_patch>
</patch_relationships>
<fixes_bugs truncated="no">
<bug>
<number>26007010</number>
<abstract><![CDATA[DATABASE GREW EVEN THOUGH RESOURCES DELETED]]></abstract>
</bug>
</fixes_bugs>
<size>210076</size>
<files>
<file>
<name>p26007010_12102180417ProactiveBP_Linux-x86-64.zip</name>
<size>210076</size>
<download_url host="https://updates.oracle.com"><![CDATA[/Orion/Services/download/p26007010_12102180417ProactiveBP_Linux-x86-64.zip?aru=22425588&patch_file=p26007010_12102180417ProactiveBP_Linux-x86-64.zip]]></download_url>
<digest type="SHA-256">ED9AE319DD9A6B149BFA66842C8AF03F853908B16B694781F78D0BC1D0E8071E</digest>
<digest type="SHA-1">F91E6F7525A09E8AEAC93FED6DC39DDA4382648B</digest>
</file>
</files>
<downloads_count>0</downloads_count>
<updated_date in_epoch_ms="1538413182000">2018-10-01 16:59:42</updated_date>
<released_date in_epoch_ms="1521586681000">2018-03-20 22:58:01</released_date>
</patch>
</results>

At the beginning this xml shows some generic information about the patch like it's product, release, platform. It also contains the bug it fixes and download filename. With previous post most of it's content should be easily readable.


The README.txt (and sometimes README.html) is this particular file admins should read, understand and follow. It contains directives how to check, if the patch is applicable, and how to apply it.


The next part bottom up is a directory structure files - lib - libserver12.a - ktsl*.o
As a rule of thumb these files will be somehow merged into the respective directories in the binaries home directory (ORACLE_HOME in the case of RDBMS).

More details can be found in etc - config directory:

the inventory.xml again contains some additional details about the patch:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<oneoff_inventory>
<opack_version version="14.1.0.1.9"/>
<patch_id number="26007010"/>
<minimum_opatch_version>12.1.0.1.4</minimum_opatch_version>
<unique_patch_id>22425588</unique_patch_id>
<cannot_autorollback>false</cannot_autorollback>
<date_of_patch day="1" month="Oct" time="09:59:37 hrs" year="2018" zone="PST8PDT"/>
<base_bugs>
<bug description="DATABASE GREW EVEN THOUGH RESOURCES DELETED" number="26007010"/>
</base_bugs>
<required_components>
<component internal_name="oracle.rdbms" opt_req="O" version="12.1.0.2.0"/>
</required_components>
<os_platforms>
<platform id="226" name="Linux x86-64"/>
</os_platforms>
<executables>
<executable path="%ORACLE_HOME%/bin/oracle"/>
</executables>
<instance_shutdown>true</instance_shutdown>
<instance_shutdown_message/>
<online_rac_installable>true</online_rac_installable>
<run_as_root>false</run_as_root>
<sql_migrate>false</sql_migrate>
<wls_prereq_oneoffs/>
<prereq_oneoffs>
<prereq oneoff_id="27338029"/>
</prereq_oneoffs>
<overlay_oneoffs>
<overlay oneoff_id="27338029"/>
</overlay_oneoffs>
<patch_type sub_type="update" value="singleton"/>
<patch_language value="en"/>
<product_family value="db"/>
<patching_model value="one-off"/>
<auto>false</auto>
<translatable>false</translatable>
<application_dirs/>
<patch_characteristics>
<content_type value="generic"/>
</patch_characteristics>
</oneoff_inventory>

whereas actions.xml describes what to do with the content of the patch:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<oneoff_actions>
<oracle.rdbms opt_req="O" version="12.1.0.2.0">
<archive backup_in_zip="false" name="libserver12.a" object_name="lib/libserver12.a/ktsla.o" path="%ORACLE_HOME%/lib" shaolue="6D6BA5355DB887694E210E53856F2CE8F779BB9E"/>
<archive backup_in_zip="false" name="libserver12.a" object_name="lib/libserver12.a/ktsld.o" path="%ORACLE_HOME%/lib" shaolue="842C83B27C0517C0D938865C819F2090163A926F"/>
<make change_dir="%ORACLE_HOME%/rdbms/lib" make_file="ins_rdbms.mk" make_target="ioracle"/>
</oracle.rdbms>
</oneoff_actions>

In this case the 2 files in lib/libserver12.a  are copied to %ORACLE_HOME%/lib and a new libserver12.a (and the oracle binary) is newly generated by make -f ins_rdbms.mk ioracle.

Of course other patches can contain other actions. Their basic meaning should be guessable with some common sense.



When comparing previous patch to p26439748_12102180116ProactiveBP_Linux-x86-64.zip , we can see some important details:
.
├── 26439748
│   ├── etc
│   │   └── config
│   │   ├── actions.xml
│   │   └── inventory.xml
│   ├── files
│   │   └── lib
│   │   └── libserver12.a
│   │   └── ktsla.o
│   └── README.txt
└── PatchSearch.xml

actions.xml:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<oneoff_actions>
<oracle.rdbms opt_req="O" version="12.1.0.2.0">
<archive name="libserver12.a" object_name="lib/libserver12.a/ktsla.o" path="%ORACLE_HOME%/lib"/>
<make change_dir="%ORACLE_HOME%/rdbms/lib" make_file="ins_rdbms.mk" make_target="ioracle"/>
</oracle.rdbms>
</oneoff_actions>

We can easily see both patches touch ktsla.o. That's where they conflict and
opatch prereq CheckConflictAgainstOHWithDetail -ph ./
shows
Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
...


To fix this conflict, let's have a look at p27860850_12102180417ProactiveBP_Linux-x86-64.zip:
.
├── 27860850
│   ├── etc
│   │   └── config
│   │   ├── actions.xml
│   │   └── inventory.xml
│   ├── files
│   │   └── lib
│   │   └── libserver12.a
│   │   ├── ktsla.o
│   │   └── ktsld.o
│   └── README.txt
└── PatchSearch.xml

PatchSearch.xml now contains
...
<bug>
<number>27860850</number>
<abstract><![CDATA[MERGE REQUEST ON TOP OF DATABASE BP 12.1.0.2.180417 FOR BUGS 26007010 26439748]]></abstract>
</bug>
<name>27860850</name>
<type>Patch</type>
...
<fixes_bugs truncated="no">
<bug>
<number>26007010</number>
<abstract><![CDATA[DATABASE GREW EVEN THOUGH RESOURCES DELETED]]></abstract>
</bug>
<bug>
<number>26439748</number>
<abstract><![CDATA[ENQ: TX - CONTENTION AND GC BUFFER BUSY ACQUIRE WAITS ON SECUREFILES IN 12.1]]></abstract>
</bug>
<bug>
<number>26513067</number>
<abstract><![CDATA[ASSIGNMENT OF INSTANCE AFFINITY FLAWED IN KTSLA_HBB_UPDATE_CACHE()]]></abstract>
</bug>
<bug>
<number>27255377</number>
<abstract><![CDATA[ORA-00600: INTERNAL ERROR CODE, ARGUMENTS: [KTEUPROPAGATETIME:CLSVIOL_KCBGCUR_9]]]></abstract>
</bug>
</fixes_bugs>
...

This particular merge patch contains fixes for 26007010 and 26439748 (and some others).


There are also bigger patches like p27486326_121020_Linux-x86-64.zip:
.
├── 27486326
│   ├── 26983807
│   │   ├── README.txt
│   │   ├── etc
│   │   │   ├── config
│   │   │   │   ├── actions.xml
│   │   │   │   └── inventory.xml
│   │   │   └── xml
│   │   │   ├── GenericActions.xml
│   │   │   └── ShiphomeDirectoryStructure.xml
│   │   └── files
│   │   ├── oc4j
│   │   │   └── ...
│   │   └── wlm
│   │   └── ...
│   ├── 27338013
│   │   ├── README.html
│   │   ├── README.txt
│   │   ├── automation
│   │   │   ├── ...
│   │   ├── custom
│   │   │   └── scripts
│   │   │   ├── post.txt
│   │   │   ├── pre.txt
│   │   │   └── prepatchverify.sh
│   │   ├── etc
│   │   │   ├── config
│   │   │   │   ├── actions.xml
│   │   │   │   └── inventory.xml
│   │   │   └── xml
│   │   │   ├── GenericActions.xml
│   │   │   └── ShiphomeDirectoryStructure.xml
│   │   └── files
│   │   ├── ...
│   ├── 27338020
│   │   ├── README.html
│   │   ├── README.txt
│   │   ├── automation
│   │   │   ├── bp1-auto-inplace-non-rolling-automation.xml
│   │   │   ├── bp1-auto-inplace-rolling-automation.xml
│   │   │   ├── bp1-auto-rollback-inplace-automation.xml
│   │   │   ├── bp1-auto-rollback-inplace-non-rolling-automation.xml
│   │   │   ├── bp1-inplace-automation.xml
│   │   │   ├── bp1-inplace-non-rolling-automation.xml
│   │   │   ├── bp1-out-of-place-automation.xml
│   │   │   ├── bp1-out-of-place-non-rolling-automation.xml
│   │   │   ├── bp1-out-of-place-non-rolling-switchback.xml
│   │   │   ├── bp1-out-of-place-switchback.xml
│   │   │   ├── bp1-rollback-inplace-automation.xml
│   │   │   ├── bp1-rollback-inplace-non-rolling-automation.xml
│   │   │   └── messages.properties
│   │   ├── custom
│   │   │   └── scripts
│   │   │   ├── postpatch.sh
│   │   │   └── prepatch.sh
│   │   ├── etc
│   │   │   ├── config
│   │   │   │   ├── actions.xml
│   │   │   │   └── inventory.xml
│   │   │   └── xml
│   │   │   ├── GenericActions.xml
│   │   │   └── ShiphomeDirectoryStructure.xml
│   │   └── files
│   │   ├── ...
│   ├── 27338029
│   │   ├── 20243804
│   │   │   ├── etc
│   │   │   │   └── config
│   │   │   │   ├── actions.xml
│   │   │   │   └── inventory.xml
│   │   │   └── files
│   │   │   ├── ...
│   │   ├── 20415006
│   │   │   ├── etc
│   │   │   │   └── config
│   │   │   │   ├── actions.xml
│   │   │   │   └── inventory.xml
│   │   │   └── files
│   │   ├── ...
│   ├── 20594149
│   │   │   ├── etc
│   │   │   │   └── config
│   │   │   │   ├── actions.xml
│   │   │   │   └── inventory.xml
│   │   │   └── files
│   │   │   ├── ...
│   │   ├── README.html
│   │   └── README.txt
│   │   ├── 27338029
│   │   │   ├── etc
│   │   │   │   └── config
│   │   │   │   ├── actions.xml
│   │   │   │   ├── composite.xml
│   │   │   │   └── inventory.xml
│   │   │   └── files
│   │   │   ├── ...
│   ├── README.html
│   ├── README.txt
│   ├── automation
│   │   ├── bp1-auto-inplace-non-rolling-automation.xml
│   │   ├── ...
│   │   └── messages.properties
│   └── bundle.xml
└── PatchSearch.xml

979 directories, 7578 files

The first obvious diference are the patch numbers inside of 27486326, and these patch number contain their own well known directory structure. And 27338029 again has it's sub patches.
But let's start with known PatchSearch.xml:

<!-- This file contain patch Metadata -->
<results md5_sum="0e8f74b3cdd19ff9af42650d53fbfafd">
<generated_date in_epoch_ms="1523969284000">2018-04-17 12:48:04</generated_date>
<patch has_prereqs="n" has_postreqs="n" is_system_patch="y">
<bug>
<number>27486326</number>
<abstract><![CDATA[DATABASE PROACTIVE BUNDLE PATCH 12.1.0.2.180417]]></abstract>
</bug>
<name>27486326</name>
<type>Patch</type>
<status>Available</status>
<access id="m">Open access</access>
<url>
<patch_readme host="https://updates.oracle.com"><![CDATA[/Orion/Services/download?type=readme&aru=22118389]]></patch_readme>
<patch_details><![CDATA[/download/27486326.html]]></patch_details>
</url>
<request_id>22118389</request_id>
<product id="9480" bugdb_id="5"><![CDATA[Oracle Database Family]]></product>
<release id="600000000009300" name="12.1.0.2.0" platform_patch_not_required="Y" cc="Y"><![CDATA[Oracle 12.1.0.2.0]]></release>
<platform id="226" bugdb_id="226"><![CDATA[Linux x86-64]]></platform>
<language id="0" iso_code="EN"><![CDATA[American English]]></language>
<translations_available>No</translations_available>
<classification id="174">General</classification>
<patch_classification id="174">General</patch_classification>
<support_level id="G">General Support</support_level>
<entitlements>
<entitlement code="SW"/>
</entitlements>
<fixes_bugs truncated="no">
<bug>
<number>13498243</number>
<abstract><![CDATA[CALCULATE AVAILABLE SCN HEADROOM]]></abstract>
</bug>
<bug>
<number>13542050</number>
<abstract><![CDATA[USE OF KGL MUTEXES MIGHT BLOCK ON BOGUS MUTEX HOLDER]]></abstract>
</bug>
<bug>
<number>13640676</number>
<abstract><![CDATA[CANNOT EXCHANGE TABLE PARTITION WITH TEXT INDEX CREATED BEFORE 11.2.0.2 UPGRADE]]></abstract>
</bug>
...
</fixes_bugs>
<size>2531696443</size>
<files>
<file>
<name>p27486326_121020_Linux-x86-64.zip</name>
<size>2531696443</size>
<download_url host="https://updates.oracle.com"><![CDATA[/Orion/Services/download/p27486326_121020_Linux-x86-64.zip?aru=22118389&patch_file=p27486326_121020_Linux-x86-64.zip]]></download_url>
<digest type="MD5">86424FF991F46750CBFA9DBBDC27E208</digest>
<digest type="SHA-256">F16D2BEF1B02E0132C75ADB3EF176D3DD30C32FBC8D40F35EA68457E918648B7</digest>
<digest type="SHA-1">7DD310BA943EED216E905D44FF49F5FD9C952DBF</digest>
</file>
</files>
<downloads_count>0</downloads_count>
<updated_date in_epoch_ms="1523969071000">2018-04-17 12:44:31</updated_date>
<released_date in_epoch_ms="1518611278000">2018-02-14 12:27:58</released_date>
</patch>
</results>

So it looks exactly the same and shows no evidence of the more complex structure inside of 27486326.

The next file is 27486326/bundle.xml:
<system_patch_bundle_xml type_version="2.0" bundle_type="ENGSYSTEM" patch_abstract="sample EsysPatch description" patch_id="27486326" unique_patch_id="22118389">
<system urm_id="9999" type="oracle_exadata_dbsys">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="18073" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</system>
<subpatches>
<subpatch patch_type="SINGLETON" patching_tool="OPATCH" location="26983807" platform="226" patch_id="26983807" unique_patch_id="21704746">
<target_types>
<target_type version="12.1.0.2.0" type="has">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="18086" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</target_type>
<target_type version="12.1.0.2.0" type="cluster">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="21227" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</target_type>
</target_types>
</subpatch>
...
<subpatch patch_type="SINGLETON" patching_tool="OPATCH" location="27338020" platform="226" patch_id="27338020" unique_patch_id="21965858">
<target_types>
<target_type version="12.1.0.2.0" type="has">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="18086" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</target_type>
<target_type version="12.1.0.2.0" type="oracle_database">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="18073" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</target_type>
<target_type version="12.1.0.2.0" type="rac_database">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="18083" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</target_type>
<target_type version="12.1.0.2.0" type="cluster">
<product qpart_id="5" aru_id="9480">Oracle Database</product>
<release urm_id="21227" aru_id="600000000009300">Oracle 12.1.0.2.0</release>
</target_type>
</target_types>
</subpatch>
...
</subpatches>
<automation>
<hash>dd55bb3aef5c9dbe72052b3109725ebac5db922c</hash>
<oplan_version>11.2.0.2.5</oplan_version>
</automation>
<fvalue>
<min_opatch_version>12.1.0.1.4</min_opatch_version>
<value>a93be317becb06265c84897fa024a3481f9c3bd5</value>
<function>3</function>
</fvalue>
</system_patch_bundle_xml>

It explains the subpatches and their targets.

While 26983807 has only 2 targets has and cluster, 27338020 also has targets oracle_database and rac_database. In comparison the screenshot from corresponding README.html.

The directory 27486326/27338029 is slightly different from others: It contains additional patches, but the directory structure etc/config is missing.
Instead there is a sub-patch with the same name and this one has a little gem in 27338029.

composite.xml:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<composite>
<sequence>
<patch id="20243804"/>
<patch id="20415006"/>
<patch id="20594149"/>
<patch id="20788771"/>
<patch id="20950328"/>
<patch id="21125181"/>
<patch id="21359749"/>
<patch id="21527488"/>
<patch id="21694919"/>
<patch id="21949015"/>
<patch id="22806133"/>
<patch id="23144544"/>
<patch id="24340679"/>
<patch id="24732088"/>
<patch id="25397136"/>
<patch id="25869760"/>
<patch id="26609798"/>
<patch id="26717470"/>
<patch id="26925263"/>
<patch id="27338029"/>
</sequence>
<replaces_upis>
<upi>18350083</upi>
<upi>18703022</upi>
<upi>18977826</upi>
<upi>19194568</upi>
<upi>19553095</upi>
<upi>19694308</upi>
<upi>20464632</upi>
<upi>20683584</upi>
<upi>20904347</upi>
<upi>21099266</upi>
<upi>21385018</upi>
<upi>21481899</upi>
<upi>21602269</upi>
<upi>21850549</upi>
<upi>22036385</upi>
</replaces_upis>
<bundle series="DBBP" id="180417"/>
<replaces_bundle series="PSU" id="180417"/>
</composite>

The Patch IDs mentioned here are all the other directories in 27486326/27338029, with their own well known structure. The upi (Unique Patch ID) is identical to aru from previous post and
by using one of this arus to get the corresponding readme ( https://updates.oracle.com/Orion/Services/download?type=readme&aru=18350083 ) this uip/aru leads to Patch 19769480 - Database Patch Set Update 12.1.0.2.2 (Includes CPUJan2015).
As so often, when the details are clear, most things are obvious: this Bundle Patch replaces all previous BPs.


That should be sufficient information for one post, for some additional information please read Frits Hooglands opatch investigations.
Viewing all 172 articles
Browse latest View live