Low Cardinality != Bitmap Index

Sorry but this post is a bit of a rant. I was called into a performance issue yesterday. The users were complaining of slow performance. I issued extended SQL tracing on the session and found the SQL statement was a simple SINGLE ROW insert statement using bind variables. No triggers on the table.

What I found were hundreds of thousands of db file sequential read wait events to insert a single row. I checked out the data dictionary for any supporting indexes and found 10 indexes on the table, 4 of which were bitmap indexes. Fortunately, this was a 10g database, so the object number associated with the sequential reads were easily plucked using a simple AWK script.

wait #22: nam='db file sequential read' ela= 377 file#=17 block#=20988904 blocks=1 obj#=725386 tim=2691112678912

I found that nearly 99.99% of these wait events were owed to this object, a bitmap index.This application is not your standard OLTP as the underlying table gets loaded with thousands of rows each day with SINGLE ROW inserts. The dreaded concurrency and deadlocking did not come into play, well, because the load process is single threaded. However, all queries against this table need to perform very quickly. So, in that sense it has an OLTP face. Here is the rub. First, I asked if these indexes (in particular the bitmap indexes) could be dropped prior to their "load" and recreated after. The answer I received was essentially, "no, that is the way the application works." I then asked them to tell me why this index was a bitmap index. The developer stated the rationale was the fact that the data was uniformly distributed over 6 distinct values. I suppose that seems reasonable. I then asked the developer if this column was used in join conditions for other queries. The answer was a resounding NO.

Not to my surprise the index built as a standard b*tree index was just as efficient and lacked the horrific index maintenance overhead associated with SINGLE ROW inserts. The only reason the index was defined as a bitmap index was its cardinality and nothing more. I had them drop the index. The load that was taking 20+ hours to complete finished in under a minute. The lesson here is: Know your data, know your code and then evaluate the use of bitmap indexes to support your table access. The simple fact of low cardinality does not alone justify the use of a bitmap index. As a matter of fact, this bitmap index was so chubby that after it was re-created post load, it had been reduced in size by 99%. I suppose that is another point: Bitmap indexes aren't necessarily space savers either if used in an improper context.

BTW, the hundreds of thousands of blocks reads were not what you might have thought: locks against rows with the same bitmap as the inserted value for the bitmap column. Oracle was ranging over the index nonsensically looking for the proper place to dump the row. As the hundreds of thousands of sequential reads rolled by not a single TM lock was obtained and ZERO db block changes had accumulated. It was only when the row finally inserted that a few blocks changes showed up. This is just another example of a peculiarity with bitmap indexes that can crop up if used unlawfully.

Database Continuity

Ever just have a burning desire to do something that never seems to go away? For me, that desire has been to write a book; more specifically an Oracle technology-based book. (Okay, maybe a novel later on in life...) I thoroughly enjoy researching Oracle technology and finding solutions to puzzling questions. I am, however, pragmatic and seek to understand that which I can put to good use in the future.

I was recently discussing this desire with a colleague. I told him that I felt there was a need for a really good backup and recovery book. Actually, I expounded a bit and said that there is a need for a good database continuity book. It just feels that backup and recovery is an overused phrase for a dramatically underutilized and uncultivated set of skills. After all, how frequently are we involved in backup and recovery exercises? I would guess that backup and recovery activities comprise less than 5% of the time spent by a DBA during the course of any given year. That would be less than 100 hours in a full work year. I suspect it could be much less for some.

Isn't spending little or no time on backup and recovery a good thing? That does imply our systems are resilient and few faults surface that require us to exercise our recovery plan. And, in the age of RMAN we simply don't have to worry about the nuances of recovery, right? RMAN knows exactly what is needed for restoration, and all the DBA needs to do is execute a few commands to restore and recover the database. What technology has afforded us with regard to ease of backup configurations and redundant infrastructure, it has equally reduced our ability to confidently take control when up against a critical database recovery scenario. In short, we become complacent and our knowledge of backup and recovery diminishes over time. How confident are we that our backup strategy meets the recovery point (RPO) and recovery time (RTO) objective of our business?

I digress. Let’s get back to the conversation with my colleague and this notion of database continuity. I defined for him database continuity as follows: Database continuity is a superset of knowledge, processes and tools that fulfill the data protection requirements of an organization. By consequence, backup and recovery become processes in the continuity methodology. Database continuity is a broadened perspective of Oracle database recovery and is intended to include: disaster recovery, standby databases, archive log management, user-managed backups, RMAN, RPO and RTO, etc. Each of these aspects of database continuity requires the DBA to have a firm understanding of Oracle database recovery. If we truly understand recovery these different continuity dimensions converge rapidly. You can plug in your knowledge of recovery to assist with any dimension. So, while the notion of database continuity has greater breadth at face value, it can be reduced to recovery mechanics, constructs and objectives.

That being said, I have many ideas about a book on Oracle database continuity. However, I want to hear from you. What do you find lacking in the backup and recovery books on the market? Maybe one text speaks to an aspect for which you wish the author had given more detail. Or, maybe there is an overindulgence of certain topics that you wish had been left out. What material would help you retain and reuse your recovery knowledge? I am not out to write a book on RMAN or Data Guard; thousands of pages have already been devoted to the treatment of these technologies. I view guides on such topics as utilities to affect my recovery objectives and mobilize my recovery knowledge.