OpenDJ 3.0.0 has been released…

FR_plogo_org_FC_openDJ-300x86As part of the release of the ForgeRock Identity Platform that we did last week, we’ve released a major version of our Directory Services product : OpenDJ 3.0.0.

The main and most important change in OpenDJ 3.0 is the work on the backend layer, with the introduction of a new backend database, supported by a new low level key-value store. When installing a new instance of OpenDJ, administrators now have the choice of creating a JE Backend (which is based on Berkeley DB Java Edition, as with previous releases of OpenDJ), or a PDB Backend (which is based on the new PersistIt library). When upgrading, the existing local backends will be transparently upgraded in JE Backends, but indexes will need to be rebuilt (and can be rebuilt automatically during the upgrade process).

Both backends have the same capabilities, and very similar performances. Most importantly, both backends benefit from a number of improvements compared with previous releases : the size of databases and index records are smaller, some indexes have been reworked to deliver better performances both for updates and reads. Overall, we’ve been increasing the throughput of Adding/Deleting entries in OpenDJ by more than 15 %.

But the 2 backends are different, especially in the way they deal with database compression. Because of the way it’s dealing with journals and compression, the new PDB backend may deliver better overall throughput, but may increase its disk occupancy significantly under heavy load (it favours updates over compression). Once the throughput is reduced under a certain threshold, compression will be highly effective and the overall disk occupancy will be optimised.

A question I often get is “Which backend should I use? “. And I don’t have a definitive answer. If you have an OpenDJ instance and you’re upgrading to 3.0, keep the JE Backend. This is a simple and automated upgrade. If you’re installing a new instance of OpenDJ, then I would say it’s a matter of risks. We don’t have the same wide experience with the PDB backend than we have had with the JE backend over the last 10 years. So, if you want to be really safe, chose the JE Backend. If you have time to test, stage your directory service before putting it in production, you might want to go with the PDB Backend. As, moving forward, we will focus our performance testing and improvements on the PDB backend essentially.

That’s all for now. In a followup post, I will continue to review the changes in OpenDJ 3.0…

Meanwhile, you can download OpenDJ 3.0 from ForgeRock’s BackStage and start playing with it. And check the Release Notes for more information.

PS: The followup posts have been published:

LDAPCon 2015

22494196563_56cdbd5a6c_zTime flies… LDAPCon 2015 has happened and we all have returned from Edinburgh to our daily lives.

As for the previous editions, this year’s conference was well attended, very friendly, with plenty of time to socialize around a (few) pint(s) of beer.

23126811911_71434b0311_mDavid Goodman started the conference with a keynote presentation that illustrated the long path followed by LDAP, but also how important it still is in the major industries, especially in the Telco world.

My 2 presentations were given on the first day of the conference. The first one was about “LDAP Asynchronous Programming” and the Promises API we’ve added in the OpenDJ LDAP SDK.

The second presentation was an update on the OpenDJ project with a highlight on what is in the OpenDJ 3.0 release due mid December.

All of the presentations are already available through the web site, either in the Programme section, or directly in the Downloads section.

Thanks and kudos to this year’s organisers : Andrew Findlay and Stephen Quinney.

As usual, you can get a glimpse of the conference and people on my photo album.

LDAPCon 2015 photo album by Ludovic Poitou
LDAPCon 2015

LDAPCon is this week…

Starting Wednesday with tutorials, and the main conference on Thursday and Friday, the 5th International LDAP Conference happens in Edinburg, this week.

I will be there during the 3 days, along with several members of the OpenDJ team. I hope to see you there.

ForgeRock is a platinium sponsor of the conference. We are offering a free pass to the conference. If you can be in Edinburg at the end of the week and you are interested, please reach out to me.

OpenDJ Nightly Builds…

For the last few months, there’s been a lot of changes in the OpenDJ project in order to prepare the next major release : OpenDJ 3.0.0. While doing so, we’ve tried to keep options opened and continued to make most of the changes in the trunk/opends part, keeping the possibility to release a 2.8 version. And we’ve made tons of work in branches as well as in trunk/opendj. As part of the move to the trunk, we’ve changed the factory to now build with Maven. Finally, at the end of last week, we’ve made the switch on the nightly builds and are now building what will be OpenDJ 3, from the trunk.

For those who are regularly checking the nightly builds, the biggest change is going to be the version number. The new build is now showing a development version of 3.0.

$ start-ds -V
OpenDJ 3.0.0-SNAPSHOT
Build 20150506012828
--
 Name Build number Revision number
Extension: snmp-mib2605 3.0.0-SNAPSHOT 12206

We are still missing the MSI package (sorry to the Windows users, we are trying to find the Maven plugin that will allow us to build the package in a similar way as previously with ant), and we are also looking at restoring the JNLP based installer, but otherwise OpenDJ 3 nightly builds are available for testing, in different forms : Zip, RPM and Debian packages.

OpenDJ Nightly Builds at ForgeRock.org

We have also changed the minimal version of Java required to run the OpenDJ LDAP directory server. Java 7 or higher is required.

We’re looking forward to getting your feedback.

Linux AD Integration with OpenDJ – by Pieter Baele

This week I stumbled upon this presentation done by Pieter Baele, about the integration of Linux, Microsoft AD and OpenDJ, to build a secure efficient naming and security enterprise service.

The presentation covers the different solutions to provide integrated authentication and naming services for Linux and Windows, and described more in depth one built with OpenDJ. Overall, it has very good information for the system administrators that need to address this kind of integration between the Linux and the Windows world.

Screen Shot 2015-04-03 at 00.21.10

About auditing LDAP operations…

OpenDJ LogoMany years ago, when I’ve started working on LDAP directory services, we needed to have some auditing of the operations occurring on the server. So, the server had a “Access” log which contained a message when an operation was received, and one when it was returned to the client, which included the processing time on the server side (the etime parameter). On Netscape and Sun directory servers, the etime was measured in seconds. This format allowed us to detect requests that were taking a long time, or were started but not finished.

In OpenDJ, we switched the etime resolution to milliseconds, but there’s an option to set it to nano-seconds. Yet, with millisecond resolution, there are still a number of log entries with an etime value of 0. The truth is that the server is faster, but so are the machines and processors.

At a rate of 50 000 operations per seconds (which can easily be sustained on my laptop), having two messages per operation does generate a lot of data to write to disk. That’s why we have introduced a new audit log format, not well advertised I must say, in OpenDJ 2.6.0. To enable the new format, use the following dsconfig command:

dsconfig set-log-publisher-prop -h localhost -p 4444 -X -n \
 -D "cn=directory manager" -w password \
 --publisher-name File-Based\ Access\ Logger  --set log-format:combined

And now instead of having 2 lines per operations, there is a single one.

Before:

[23/Feb/2015:08:56:31 +0100] SEARCH REQ conn=0 op=4 msgID=5 base="cn=File-Based Access Logger,cn=Loggers,cn=config" scope=baseObject filter="(objectClass=*)" attrs="1.1"
[23/Feb/2015:08:56:31 +0100] SEARCH RES conn=0 op=4 msgID=5 result=0 nentries=1 etime=0
[23/Feb/2015:08:56:31 +0100] SEARCH REQ conn=0 op=5 msgID=6 base="cn=File-Based Access Logger,cn=Loggers,cn=config" scope=baseObject filter="(objectClass=*)" attrs="objectclass"
[23/Feb/2015:08:56:31 +0100] SEARCH RES conn=0 op=5 msgID=6 result=0 nentries=1 etime=0

After, in combined mode:

[23/Feb/2015:13:00:28 +0100] SEARCH conn=48 op=8215 msgID=8216 base="dc=example,dc=com" scope=wholeSubtree filter="(uid=user.1)" attrs="ALL" result=0 nentries=1 etime=0
[23/Feb/2015:13:00:28 +0100] SEARCH conn=60 op=10096 msgID=10097 base="dc=example,dc=com" scope=wholeSubtree filter="(uid=user.6)" attrs="ALL" result=0 nentries=1 etime=0

The benefits of enabling the combined log format are multiple. Less data is written to disk for each operation, less I/O operations are involved, resulting in overall better throughput for the server. And it allows to keep more history of operations with the same volume of log files.

Do you think that OpenDJ 3.0 access log files should use the combined format by default ?

OpenDJ on Windows…

OpenDJ LogoOpenDJ, the LDAP directory services in Java, is supported on multiple platforms and has been for many years. We’re testing on Linux, Windows, Solaris, Mac OS X, but also different JVMs: Oracle JRE, OpenJDK, Azul Zulu, IBM JVM…

With OpenDJ 2.6, we’ve made it easier for people to install it on Linux machines by providing RPM and Debian packages.

We are now also providing a MSI package to ease the installation and removal on Windows machines. The MSI package is available for nightly builds here.

OpenDJ MSI InstallerScreen Shot 2015-01-28 at 09.14.01

Availability in OpenDJ Training in London, Week of June 23rd.

TrainingThe ForgeRock University department has scheduled an in person, instructor led training for the OpenDJ Administration, Maintenance and Tuning module, in London from June 23rd to June 26th 2014.

The 4 days training provides the perfect opportunity to learn in details everything you’ve ever wanted to know about the OpenDJ directory service and how to get the best of it.

The training is firmly confirmed but still have a few seats available. If you’re interested, you can register here.

Ansible roles for OpenDJ

My colleague Warren, who I had the pleasure to work with at Sun and again with ForgeRock, has been playing with Ansible and has produced 2 roles to install OpenDJ and to configure replication. Check Warren’s blog post for the details, or go directly to the Ansible Galaxy.

About LDAP Syntaxes and backward compatibility…

In the LDAP information model, a syntax constrains the structure and format of attribute values. OpenDJ defines and implements a large number of syntaxes (you can discover them by reading the ldapSyntaxes attribute from the cn=Schema entry).

But infrequently, we receive enquiries on an obscure and non standard syntax, often in the form of “I’m having an error importing schema from this or that legacy directory server”, with an error message that ends with “No such syntax is configured for use in the Directory Server”.

As syntaxes are constraining the structure and format of attribute values, they are implemented as code, specifically Java code in OpenDJ. It’s possible to implement new syntaxes by implementing the org.opends.server.api.AttributeSyntax abstract class, and installing the classes or the JAR in OpenDJ classpath. But often, it’s easier and more convenient to define a syntax by configuration, and OpenDJ offers 3 possibilities to define new syntaxes. In term of backward compatibility, I will only focus on the 2 main ones, by substitution and by pattern (the 3rd one allows to define enumeration of values).

With OpenDJ, you can define a new syntax by configuration and delegating the contraints to an already implemented syntax. A simple example is the URI syntax (which was defined is some very old schema with the OID  1.3.6.1.4.1.4401.1.1.1). A URI is really an ASCII string, and it might be sufficient to accept attributes with URI syntax to verify that all characters are pure ASCII. The standard syntax for ASCII strings is IA5String aka 1.3.6.1.4.1.1466.115.121.1.15.

ldapSyntaxes: ( 1.3.6.1.4.1.4401.1.1.1 DESC ‘URI’ X-SUBST ‘1.3.6.1.4.1.1466.115.121.1.15’ )

Insert the above line in the schema LDIF file before the attributeTypes, and you’re done.

The other option is to define the syntax as a pattern, using regular expressions. This could be better when willing to enforce additional constraints on an URI, for example, verifying that the URI is an LDAP one.

ldapSyntaxes: ( 999.999.999.1 DESC 'LDAP URI Syntax' X-PATTERN '^ldap://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]' )

So the next time you are trying to import some legacy schema to the OpenDJ directory server, and you have an error due to missing syntaxes, you know what to do to quickly resolve the problem.

OpenDJ Contact Manager for Android

With OpenDJ 2.6.0, we’ve introduced a new way to access your directory data, using HTTP, REST and JSon. The REST to LDAP service, available either embedded in the OpenDJ server or as a standalone web application, is designed to facilitate the work of application developers. And to demonstrate the interest and the ease of use of that service, we’ve built a sample application for Android : the OpenDJ Contact Manager

OpenDJ Contact Manager Android AppAbout screen of the OpenDJ Contact Manager Android App

The OpenDJ Contact Manager is an open source Android application that was built by Violette, one of the ForgeRock engineer working in the OpenDJ team. You can get the source code from the SVN repository : https://svn.forgerock.org/commons/mobile/contact-manager/trunk. Mark wrote some quite complete documentation for the project, with details on how to get and build the application. He published it at http://commons.forgerock.org/mobile/contact-manager/.

The whole application is just about 4000 lines of code, and most of it is dealing with the display itself. But you can find code that deals with asynchronous calls to the OpenDJ rest interface, with paging through results, and parsing the resulting JSON stream to populate the Contacts, including photos. Et voila :

OpenDJ Contact Manager displaying a Contact

The application is just a sample but it clearly is usable in its current form and will allow once a contact was retrieved from the OpenDJ directory, to add it to the Contacts standard application, call the person, locate its address on maps, send the person an email, navigate through the management chain…

In future versions, we are planning to add support for OAuth 2.0, removing the need to store credentials in the application settings.

As it’s open source, feel free to play with it, hack and contribute back your changes.

LDAPCon 2013 – a summary…

ldapcon_2013_logo_line_dateLast Monday and Tuesday (Nov 18-19), I was in Paris attending the 4th International LDAP Conference, an event I help to organize with LDAPGTF, a network of French actors in the LDAP and Identity space. ForgeRock was also one of the 3 gold sponsors of the conference along with Symas and Linagora.

LDAPCon 2013The conference happens every other year and is usually organized by volunteers from the community. This year, the French guys were the most motivated, especially Clément Oudot from Linagora, leader of the LDAP Tool Box and lemonLDAP projects, and Emmanuel Lecharny one of the most active developers on Apache Directory Server.

I was honored to be the keynote and first speaker of the conference and presented “The Shift to Identity Relationship Management“, which was well received and raised a lot of interest from the audience.

The first day was focusing more on the users of LDAP and directory services technologies, and several presentations were made about REST interfaces to directory services, including the standard in progress: SCIM.

Kirian Ayyagari, from the Apache Directory project, presented his work on SCIM and the eSCIMo project. Present for the first time at LDAPCon, Microsoft’s  Philippe Beraud spoke about Windows Azure Active Directory and its Graph API. And I talked about and demoed the REST to LDAP service that we’ve built in OpenDJ. For the demo, I used PostMan, a test client for HTTP and APIs, but also our newly open sourced sample application for Android : OpenDJ contact manager. In the afternoon, Peter Gietz talked about the work he did around SPML and SCIM leveraging OpenLDAP access log.

After many talks about REST, we had a series of talk around RBAC. Shawn McKinney presented the Fortress open source IAM project and more specifically the new work being done around RBAC. Then Peter, Shawn and Markus Widmer talked about the effort to build a common LDAP schema for RBAC. And Matthew Hardin talked about the OpenLDAP RBAC overlay bringing policy decisions within the directory  when deploying Fortress.

Then followed presentations about local directory proxy services for security based on OpenLDAP, about Red Hat FreeIPA (another first appearance at LDAPCon) and about OpenLDAP configuration management with Apache Directory Studio. Also Stefan Fabel came all the way from Hawaii ( Aloha ! ) to present a directory based application for managing and reporting publications by a university: an interesting story about building directory schema and data model.

The day ended with a presentation from Clement Oudot about OpenLDAP and the password policy overlay. As usual, talking about the LDAP password policy internet-draft raises the question of when it will be finally published as an RFC. While there is a consensus that it’s important to have a standard reference document for it, I’m failing to see how we can dedicate resources to achieve that goal. Let’s see if someone will stand up and take the leadership on that project.

After such a long day of talks and discussion, most of the attendees converged to a nearby pub where we enjoyed beers and food while winding down the day through endless discussions.

The second day of LDAPCon 2013 was more focused on developers and the development of directory services. It was a mix of status and presentations of open source directory projects like OpenDJ, OpenLDAP or LSC, some discussions about backend services, performance design considerations and benchmarks, a talk about Spring LDAP… As usual, we had a little bit of a musical introduction to Howard Chu‘s presentation.

LP0_1068I enjoyed the Benchmark presentation by Jillian Kozyra, which was lively, rational and outlining the major difference between open source based products and closed source ones (although all closed source products were anonymized due to license restrictions). It’s worth noting that Jillian is pretty new in the directory space and she seems to have tried to be as fair as possible with her tests, but she did say that the best documented product and the easiest one to install and deploy is OpenDJ. Yeah !!! 🙂

Another interesting talk was Christian Hollstein‘s about his “Distributed Virtual Transaction Directory Server“, a telco grade project he’s working on to serve the needs of the 4G network services (such as HSS, HLR…). It’s clear to me that telco operators and network equipment providers are now all converging to LDAP technologies for the network and this drives a lot of requirements on the products (something I knew since we started the OpenDS project at Sun, kept in mind while developing OpenDJ, even though right now our focus has mainly been on the large enterprises and consumer facing directory services).

All the slides of the conference have been made available online through the LDAPCon.org website and the Lanyrd event page. Audio has also been recorded and will be made available once processed. And as usual, all the photos that I took during the conference are publicly available in my Flickr LDAPCon 2013 Set. Feel free to copy for personal use.

It’s been a great edition of the LDAPCon and I’m looking forward to the next one, in 2 years !

Meanwhile I’d like to thanks the sponsors, all 75 attendees, the 19th speakers and the 2 organizers I had not mentioned yet : M.C. Jonathan Clarke and Benoit Mortier.

OpenDJ is Java 8 ready…

Java 8 CompatibleA few weeks ago, I came upon the Adopt openjdk program, launched by the London Java User Group. I’ve decided to give it a try and for that to leverage the Dev@Cloud and FOSS program from CloudBees.

While building OpenDJ I hit a roadblock : XJC is defective with OpenJDK 8 and prevents us from building the DSML gateway that is part of the OpenDJ project. It’s only recently when the openjdk bug database was made publicly available that I found it was a known P1 issue, yet still not resolved.

Recently someone filed a bug against OpenDJ for failing with openjdk 8, so we paid more attention, found the cause of the failure and fixed it. And now OpenDJ directory server is working fine on openjdk 8. We are keeping an automatic build and test with openjdk8 (*) to make sure things will work when Java 8 is released.

Next steps will be to verify that OpenDJ still works with the beta version of jdk8 of IBM Java virtual machine.


* If you’re not familiar with OpenDJ nightly tests, do not try to interpret the results out there. Some of those complex replication tests are unfortunately sensitive to processor speed, thread timing and synchronization.So they tend to fail often on single CPU virtual machines where resources are unknown. They are fully passing on 2 or more CPU machines.

OpenDJ : Visualizing the Replication Topology

My coworker Chris Ridd has spent a little bit of spare time writing a small utility that can parse the output of OpenDJ monitoring information to extract the details of the replication topology. Give the output to some graphical tool and here’s the result (based on one of our biggest customer -anonymized- data) :

ReplTopo

This is a worldwide deployment with many directory services in 4 regions and 8 replication services fully connected. Each directory service is connected to a single replication server, but can failover in matter of seconds, by priority in the same region.

If you want to give it a try on your own replication topology, it’s simple. The tool is open source and part of the OpenDJ utilities that Chris has pushed to GitHub. Just feed it with the output of ldapsearch on cn=monitor.

The 4th LDAP international conference will be in Paris…

ldapcon_2013_logo_line_dateLDAPCon, the LDAP international conference is taking place every other year. So, after 2007, 2009 and 2011, the 4th LDAP international conference has been announced and will take place on November 18-19, in Paris France.

LDAPCon brings together vendors, developers, active LDAP practitioners, system administrators to share their experiences about service operations, interoperability, application development and discuss LDAP at large, in a friendly and passionated atmosphere. It’s a unique occasion to discuss with the developers of most LDAP related projects, seed them with new ideas, learn the under-documented tips and tricks about your favorite server or library, or exchange with other users and system administrators about the best practices around LDAP directory services and applications.

Call for Papers have been raised. You have up to September 8th 2013 to submit your talk. You can find all details, important dates or topic ideas on the LDAPCon CFP page.

You can also track the event, the availability of tickets, the attendees, or indicate your participation on the LDAPCon page on  Lanyrd.

I hope to see you in Paris in November.

LDAPCon 2007 attendeesLDAPCon 2009 attendeesLDAPCon 2011 attendees