Understanding and tuning the JVM is quite important to get the best performances out of OpenDJ. We do provide some high level guidance in our documentation and I’ve been talking about Java performances in the last few years at various Java User Groups in France and Switzerland (you can find presentations in French here or here) as well as at a major conference in Brazil : FISL in 2009. On this later occasion, I was asked to cover the presentation for 2 prestigious names in the Sun Hotspot JVM team : Charly Hunt and Tony Printezis. I’ve spent a few hours with them and have learnt a great deal about the internals of the Hotspot JVM and memory management, and all magic parameters, in order to deliver that presentation. At that time, our directory team was interacting a lot with the Hotspot team as we were testing a new and promising garbage collector: Garbage First aka G1. OpenDS was even wrapped and used in one of the largest collection of tests for the Sun JVM.

During the acquisition of Sun by Oracle, the future of G1 and the Hotspot JVM were unsure and our interactions with the Hotspot team diminished seriously.

At ForgeRock, we continued to pay attention to Garbage First and for a long time, we noticed that it wasn’t moving along. Most of the issues that were raised after tests with OpenDS and that were addressed in some development version of the JVM were not integrated in official JVM releases. It only with the Oracle JVM 1.7 update 2 that we noticed the large list of issues fixed with G1. We’ve then resumed testing OpenDJ with G1 to see that while the promise of no full GC seems to be addressed, the performance impact of G1 is still significantly high. With our limited tests of JVM under 4GB of heap size, we noticed a 10% performance degradation over CMS, corresponding with an approximate 10% increase of CPU load (on a quad core machine with hyperthreading on), but with better overall response times for OpenDJ as the maximum response time decreased from 200ms to 80ms, as illustrated below.

LDAP Modrate with Garbage First
-------------------------------------------------------------------------------
 Throughput     Response Time 
 (ops/second)   (milliseconds) 
recent average  recent average 99.9% 99.99% 99.999% err/sec Entries/Srch
-------------------------------------------------------------------------------
16196.7 16374.1  1.972 1.951  18.886 28.129 66.933  0.0
16468.8 16374.9  1.941 1.951  18.883 28.087 66.521  0.0

LDAP Modrate with CMS
-------------------------------------------------------------------------------
 Throughput     Response Time 
 (ops/second)   (milliseconds) 
recent average  recent average 99.9% 99.99% 99.999% err/sec Entries/Srch
-------------------------------------------------------------------------------
17937.1 17487.7  1.780 1.827  18.175 30.521 116.990 0.0
17783.7 17494.3  1.796 1.826  18.145 30.320 117.017 0.0

We need to run more tests with OpenDJ and G1, especially with very large heaps (from 4 to 32GB), but we’re not sure whether G1 will be able to deliver the performances it promised.

And today I noticed on LinkedIn that both Charly Hunt and Tony Printezis, the 2 main engineers behind the HotSpot JVM and Garbage First, had left Oracle for new adventures. Charly’s gone to  SalesForce and Tony to Adobe. This is certainly a good move for both of them, but it leaves me worried about the future of the Hotspot JVM and its ability to deliver innovation in GCs.

[Update on May 6th]

It appears that more engineers of the Sun JVM team have actually left in the last couple of months : John Pampuch, Igor Veresov, Paul Hohensee..