Gray Matter Boundaries

Ironically, after yesterday’s post about Oracle’s caviler attitude toward dealing with communities built around open software, a piece of died-in-the-wool proprietary software stepped out of Oracle’s (ever growing) stable of software, neighed a couple of times, stamped it’s feet and kicked me in the head.  Specifically, one of our 11g database servers shot up to one minute load averages a little above 2,000.  Yes, two zero zero zero.

Despite the comically high load, our M5000 server running Solaris 10 withstood the onslaught and the system was still quite usable.  A testament to the engineering teams at Sun if I ever saw one.  A quick jump to old performance standbys, mpstat, iostat, prstat and vmstat revealed many involuntary context switches and a run queue in the thousands.  My DBA colleague and I went about the business of diagnosing the problem and he promptly ran an AWR report.  Below was what we found.

"Wait, is that billions?"

“Wait, is that billions?” I asked naively.

“Yeah… something’s wrong.” my DBA colleague replied in his usual calm, understated way.

Oracle Support confirmed that, yes, four billion mutex waits in a span of an hour appeared to be the cause of our pain.  Luckily for us, this is not undiscovered country.  A quick Google search later revealed that 11g is notorious for this particular type of pain.  The fix, of course, was a patch.  Specifically, 10411618: Add different wait schemes for mutex waits.

Annoyingly, had Oracle been gracious enough to add DTrace probes to their enterprise products, we could’ve saved a lot of heartache with a one-liner.

Also, here’s a good intro to Oracle DB mutexes and latches.

For the past year, I have suffered from what I’ve churlishly dubbed as Oracle-ish Anxiety.  It’s the feeling that at any moment, Oracle may greedily yank a technology that I depend upon, kicking and screaming, from the clutches of the public domain.

OpenSolaris was the first to go to the chopping block.  No, chopping block is the wrong metaphor.  Void is more like it.  No more OSUG meetings, no more t-shirts with OpenSolaris logos, no more following online wars on topics such as governance bodies and community participation (online wars aren’t necessarily bad, they occur because people care).  The hole OpenSolaris left is raw and only Oracle remains.  A hard pill to swallow, to say the least.

Honestly, it makes me a little sad.  Whether they meant to or not, Oracle killed a vibrant community that, while smaller than the Linux cowboys, was growing a following of rabid devotees.  Strangely, OpenSolaris’ followers devotion was driven by rather boring technologies that stood in stark contrast to the whims of a fanboy (I refuse to use an i, I absolutely refuse) nation.  DTrace just isn’t sexy compared to jiggly desktops.

Then the page at midnight comes.  No one in your house is safe from an angry pager.  Partners threaten to launch the pagers into walls, babies wake up and cry, and even the dogs regard you with disdain (until you give them a treat).  After surviving withering stares from tired family members, you hobble half awake to a terminal, turn off the jiggly windows, and SSH into work.  To your dismay, an important database server has crashed and important folks all over the company want to know what you’re going to do about it.

Suddenly, in a moment of blurry eyed recognition, DTrace, ZFS, and SMF become very sexy.  The moment these technologies save your bacon is the moment you become a devotee.  You realize that had you been dealing with Windows, Linux, or another *NIX, your goose would be proverbially cooked.  That just puts icing on the cake.

As an Oracle customer responsible for large systems that rely heavily on technologies in Solaris 10 (especially ZFS), I have a nagging feeling that as much as Oracle knows how to make a dollar, they don’t know how to capitalize on the moment that turns people into devotees.  And that, my friends, is what sets my Oracle-ish Anxiety a rock’n.