Month: April 2012

An Optimized solution for directory services ?

Ludo4 Comments

I was recently pointed to a white paper published by Oracle called : Oracle Optimized Solution for Oracle Unified Directory — Implementation Guide.

Because Oracle Unified Directory and OpenDJ have a common root (both derive from the Sun initiated OpenDS project), I was curious about that optimized solution, and if there was anything that might be applicable for our customers. And after reading the 45 pages white-paper, honestly, the Oracle Optimized Solution is not something I would recommend to any of our customers (*).

The white paper describes the hardware used for the solution : 3 SPARC T4-1 systems with 128GB of RAM and 6 300GB internal disks each. A Sparc T4-1 machine is an 8 core machine with each core supporting up to 8 threads. Each T4-1 system has 10GbE add-on network card. And each T4-1 machine is attached to a Sun Storage 2500-M2 array (with 2540 controllers) through two fiber channel cards, and each storage has 12 disks.

Let’s see the average price for this solution : The SPARC T4-1 with 128 GB of RAM has an estimated public price of $24,344, but with only 2 internal disks. So add another $1,660 for the additional 4 disks. The lowest price for a 10GbE card for that system is $2,000 and the cheapest storage array with the same amount of disks roughly $27,000. A total cost per system over $55,000, not including the cost of the Operating System, and a total cost for the “Optimized Solution” of approximately $165,000 (estimated public price).

So what do we get in performance for this price ? Well the white paper will not tell you what the solution is optimized for. The only number that appears, is the time it took to import the 15 Million entries on one of the systems :

[07/Jan/2012:12:19:29 +0000] category=JEB severity=NOTICE msgID=8847569 msg=Total import time was 3790 seconds. Phase one processing completed in 2868 seconds, phase two processing completed in 922 seconds
[07/Jan/2012:12:19:29 +0000] category=JEB severity=NOTICE msgID=8847454 msg=Processed 15000001 entries, imported 15000001, skipped 0, rejected 0 and migrated 0 in 3790 seconds (average rate 3957.0/sec)
[07/Jan/2012:12:19:29 +0000] category=JEB severity=NOTICE msgID=8847536 msg=Import LDIF environment close took 0 seconds

Last week, I was in Mexico with a partner of ours, demonstrating the capabilities of OpenDJ with the customer’s data (exported from a week ago, and which also contains several hundreds of very large static groups). We used x.86 based machines, with 96GB of memory, although we only used 16GB for the instance of OpenDJ.

And here’s the output of the import command :

[25/Apr/2012:20:10:44 +0200] category=JEB severity=NOTICE msgID=8847538 msg=DN phase two processing completed. Processed 21654508 DNs
[25/Apr/2012:20:10:45 +0200] category=JEB severity=NOTICE msgID=8847569 msg=Total import time was 2002 seconds. Phase one processing completed in 1137 seconds, phase two processing completed in 865 seconds
[25/Apr/2012:20:10:45 +0200] category=JEB severity=NOTICE msgID=8847454 msg=Processed 21654508 entries, imported 21654508, skipped 0, rejected 0 and migrated 0 in 2002 seconds (average rate 10815.3/sec)
[25/Apr/2012:20:10:45 +0200] category=JEB severity=NOTICE msgID=8847536 msg=Import LDIF environment close took 0 seconds

I don’t have the price for the servers we used (but our partner can get in touch with you if you’re interested in the solution), but I doubt that it tops half of the price of the Oracle optimized solution !

So before you drink the Oracle cool-aid, think twice about what an optimized solution should be, and how much it should cost. Oh, by the way, there is no cost in license for OpenDJ, it’s open source, it’s available now and you can try it free of charge. Of course, we do appreciate if you subscribe to one of our support offering to protect your investment and ensure some Service Level Agreement.

(*) I would not recommend a directory solution on SPARC Tx machine ever. While the machines have a good capacity for load, the performance for any write activity is really bad, especially as soon as access controls are in use. Most of our partners who have been deploying directory services on these machines will agree with me. As a matter of fact, I don’t recall any recent customer mentioning SPARC nor Solaris when renewing their directory service infrastructure.

Mexico, Mexiiiiico !

Ludo2 Comments

I’m just back from a week of business trip to Mexico City. This was my first time in Mexico and I’ve heard all the rumors of it being a very dangerous city. I must say that I’ve seen a very very big city, vibrant, busy, with a lot of car trafic, but at no point I had any fear of being robbed or molested.

Two things have marked me during my stay. First, the city is very green. There are lots of trees, plants, flowers everywhere. All main avenues are borded by trees. It’s like mother nature is trying to tell us that she still exists despite the concrete and buildings.

Trees in AvenuesTree in flower

The other thing is that at any time of the day or the night, there are people in the street, trying to earn a little bit of money, selling water, tissues or balloons.

Globero


The food was amazing. I enjoyed tacos, fresh fruits, some argentinian bife, jalapeños… Spicy, but not “mucho picante”. As well as beers like Victoria, Bohemia, Dox Equis, Modelo… And tequila of course !

Other photos from my trip are on Google+

By the way, we did work this week in Mexico.

Below is a photo of the screen as we’ve finished importing the customers’ data in OpenDJ (the data includes a few hundreds of groups, each averaging 40 000 members). I like this kind of performance number ! And I will probably say more about the hardware and settings to achieve that in a future post.

I shall say a big thank you to our partner in Mexico and Latin America : NoLogin. They’ve made everything to make my stay safe and comfortable, including with jalapeños and tequila !

I hope the few companies I visited will turn into customers. I’d like to come back again in Mexico. These 5 days have just gone to0 fast. And I’ve just started to get into lutta libre 😉

Mexican Wrestler

Devoxx France, 1st Edition

LudoLeave a comment

DecoxxI was in Paris end of last week, attending the first edition of Devoxx France, a Java developers’ conference.

Devoxx is a well known and highly attended Java developers conference that takes place late fall in Antwerp. The French version has been initiated by the Paris Java User Group and has a similar structure but with 75% of the talks in French.
For a first edition, Devoxx France 2012 is a real success. Sold out 2 weeks before the event, over 1200 persons attended the 3 days’ conference. Yet, the conference felt human.

I was there only for the second and third days, as an attendee and as a co-speaker in the BOF session about Open Source Software in France.It was also the opportunity to meet and discuss with other developers, open source project leaders and potential customers.

photo by Pierre-Antoine Gregoire, alias @zepag

Out of the talks that I’ve attended, I preferred the ones that were quite low level. Among them, 2 around hacking the JVM and the bytecode:  

I also enjoyed 2 presentations by Alex Snaps, one around concurrency and the CompareAndSet method. The other one about SizeOf or the difficulty to compute the exact size of Java objects, in order to improve large cache efficiency and management. I shall look at the ehcache project code, to see if we can leverage some of it for OpenDJ caches.

Also worth mentioning, 2 greats keynotes on the Friday morning with Pat Chanezon and Neal Ford.

I’ve taken a few photos during Devoxx France 2012, feel free to use or share them (under Creative Commons). And if you want to see more photos of the event, you can check Arnaud Heritier’s collections : Day 1, Day 2, Day 3.

See you next year at Devoxx France 2013, and may be in Antwerp in November for Devoxx (WorldWide)

A timeline of LDAP directory services…

Ludo1 Comment

Bill Nelson,  has published the “The Most Complete History of Directory Services You Will Ever Find” (until the next one comes along), a detailed history of LDAP based directory services and products. Expect a few updates as people find about this and ask for adding new data points. But this is the most complete summary I’m aware of. I had a timeline of Sun directory products a few years ago, but Bill’s has more details.

His post includes a visual timeline of the directory service products and their heritage, linked here under, for your convenience.

Click on the picture for a full size image.

Personally, I’ve been involved with the Sun and derived lines since 1996, and now drive the ForgeRock one: OpenDJ !

OpenAM 10.0.0 is now available…

LudoLeave a comment

This is a big milestone for ForgeRock and the OpenAM project, an open source WebSSO, Authentication, Authorization, Federation and Entitlements solution. After months of development (a few more than we anticipated), we’ve finally released OpenAM 10.0.0, a major version of the product.

OpenAM 10 brings a set of new features, including support for OAuth 2.0 client authentication, the ForgeRock Identity Gateway (built out of project OpenIG), enhanced SAML 2 identity provider capabilities, a new Risk Based Authentication module, …  It also now relies on OpenDJ 2.4.5, the latest stable release of OpenDJ the open source LDAP directory server, and supports the internet-draft based LDAP password policy. You can find more details  in the press announcement, or the product release notes. The documentation of the OpenAM 10 release can be read at http://docs.forgerock.org/en/index.html?product=openam&version=10.0.0.

The OpenAM 10 release owes a lot to the OpenAM community, for the issues raised : a total of 41 issues fixed in OpenAM 10 were raised by 26 different persons, and for the generous patches offered to fix over a dozen of these issues.
To each and every contributor : THANK YOU !

Tab Sweep for Friday April 13th

LudoLeave a comment

Another week goes by, and it’s time for another tab sweep.

Syntegrity Networks, one of our major partners in the US, has launched a campaign to encourage their customers to migrate from Sun Directory Server to OpenDJ.

Silverpeas, a Collaborative Platform, built as open source under the GNU Affero license by the eponym company, has been supporting LDAP for authentication and authorization for some time. The documentation for setting up the LDAP domain has been updated using OpenDJ as the recommended server.

ForgeRock OpenIDM capabilities are growing. After getting OpenIDM to work with Activity to provide workflows, the team posted a experimental tutorial to integrate Jasper with OpenIDM to produce nice reports. You can find more of these tutorials in the OpenIDM How To Collection.

Tips: resource limits in OpenDJ

Ludo4 Comments
Photo by Scallop Holden http://www.flickr.com/photos/scallop_holden/
Photo by Scallop Holden http://www.flickr.com/photos/scallop_holden/

OpenDJ, the open source LDAP directory services in Java, defines a few global resource limits to prevent client connections or operations from abusing the server’s resources. These limits are

  • the maximum number of entries returned to a search request (size-limit, default is 1000),
  • the maximum amount of time to spend returning results to a client (time-limit, default is 60 seconds),
  • the maximum number of entries to look through while processing a search request (lookthrough-limit, default is 5000),
  • the maximum amount of time a connection can sit idle before the server disconnect it (idle-time-limit, default is unlimited).

There are default values for all of these limits in the Global configuration, but they can also be set on a per user basis. The global limits are read or set using dsconfig :

$ bin/dsconfig get-global-configuration-prop -p 4444 -X -n -h localhost \
 -D cn=directory\ manager -w secret12
Property : Value(s)
--------------------------------------:------------------------
bind-with-dn-requires-password : true
default-password-policy : Default Password Policy
disabled-privilege : -
entry-cache-preload : false
etime-resolution : milliseconds
idle-time-limit : 0
lookthrough-limit : 5000
max-allowed-client-connections : 0
max-psearches : unlimited
proxied-authorization-identity-mapper : Exact Match
reject-unauthenticated-requests : false
return-bind-error-messages : false
save-config-on-successful-startup : true
size-limit : 1000
smtp-server : -
time-limit : 60 s
writability-mode : enabled

The per user limits have a different LDAP attribute name and can be found or set directly in users’ entry, or through Collective Attributes. The Directory Manager entry has such specific limits set, so that everything is unlimited.

$ bin/ldapsearch -D "cn=directory manager" -w secret12 -p 1389 -X -b "cn=config" \
  '(objectClass=inetOrgPerson)' ds-rlim-time-limit ds-rlim-size-limit \
  ds-rlim-lookthrough-limit ds-rlim-idle-time-limit
dn: cn=Directory Manager,cn=Root DNs,cn=config
ds-rlim-lookthrough-limit: 0
ds-rlim-time-limit: 0
ds-rlim-idle-time-limit: 0
ds-rlim-size-limit: 0

If you decide to change the default global settings, for example the idle-time-limit, to force idle connections to be closed by the server after some time (often a smaller time than the settings of the load-balancer in between your applications and the OpenDJ servers), please remember that you might also want to change the limit for “cn=Directory Manager”, especially if your client applications are connecting with Directory Manager credentials.

Tab sweep, Easter edition, and upcoming events

LudoLeave a comment
cc by http://www.flickr.com/photos/noukorama/

Articles and links

Action Identity has posted a couple of articles about ForgeRock products:

Our friends at ProfiQ have posted an article describing how to use OpenDJ with Red-Hat Certificate System.

While talking about using OpenDJ with LDAP enabled applications, we try to maintain a page on OpenDJ documentation wiki with different tutorials on how to configure OpenDJ client applications.

Upcoming Events

ForgeRock will be present at the European Identity and Cloud Conference (EIC), April 17-20 in Munich.

We will also be participating to Devoxx France, April 18 to 20 in Paris. I will be co-speaking on Thursday 19, 7pm about Open Source in France, and will be available for individual meetings from Thursday morning to Friday end of afternoon. So, if you want to discuss about ForgeRock products or job opportunities, send me a mail, or leave a comment.

Cache strategy for OpenDJ LDAP directory server

Ludo3 Comments

System administrators that are familiar with legacy LDAP directory servers know that one of the key for the best performance is caching the data. With Sun Directory Server or OpenLDAP, there are 3 levels of caching that could be done : the filesystem level, the database level and the entries level. The filesystem level cache is managed by the OS and cannot be controlled by the application. Using the filesystem cache is good when the directory server is the only process on the machine, and/or for initial performance. The database level cache allows faster read or write operations, and also includes the indexes. The later cache is the higher level cache, and usually the one that provides the best performances as it requires the least processing from the server to return entries to the applications, and it has the least contention.

OpenDJ has a different design for its database and core server, and thus the caching strategy needs to be different.

By default, OpenDJ does have a database cache enabled, and 3 different kind of entry caches, all disabled. The reason for the 3 entry caches is that they are implementing for different needs and access patterns. But all have in common a specific filter to select which entries to cache, and some settings as to how much memory to use. During our stress and performance tests, we noticed that using an entry cache for all accessed entries added a lot of pressure on the garbage collector, and also caused more garbage collection from the old generation, often leading to either fragmentation of the memory, or more frequent full GC (also known as “Stop the world GC”). This resulted in an overall lower consistent average response time and throughput.

So, we recommend that you favor the database cache, and do not setup an entry cache, except for specific needs (and do not try to activate all 3 entry caches, this may lead to some really strange behavior).

The default settings with OpenDJ 2.4 is that 10 % of the JVM heap space will be used for the database cache. With OpenDJ 2.5 (soon to be released), we have bumped the default to 50% of the heap space. If you’re tuning the heap size and make it larger than 2GB, we recommend that you keep that 50% ratio or even increase it if the heap size exceeds the 3GB.

If you do have a few very specific entries that are very often accessed, like large static groups that are constantly used for ACI or group membership by application, then the entry cache becomes handy, and then you want to set a filter so only these specific entries are cached.

For example, if you want to cache at most 5 entries, that are groupOfNames, you can use the following dsconfig command:

bin/dsconfig set-entry-cache-prop --cache-name FIFO
 --set include-filter:\(objectclass=GroupOfNames\)
 --set max-entries:5 --set max-memory-percent:90 --set enabled:true
 -h localhost -p 4444 -D "cn=Directory Manager" -w secret12 -X -n

Otherwise, you’d better of running with no entry cache. OpenDJ read performance are such that the directory server can respond to tens of thousands if not hundred of thousands searches per second with average response time in the order of a milli-second. This should be good enough for most applications !