Sun Directory compresses data for better performance !

Sun Directory Server Enterprise Edition 7.0 was released last November, and in the December timeframe Brad Diggs and Wajih Ahmed, both Principal Field Technologists and big experts in Directory Services, backed with engineers from the Directory engineering team and Mr Benchmark, put the product on the test bench to evaluate its performance and scalability with Sun new hardware and especially the new F-20 PCIe flash drives (see also what Mr Benchmark says about the F-20).

Brad’s first article describes how much Directory Server 7 entry compression rocks, "extending search performance by more than 50% through increased caching potential". Brad provides details of his findings and gives the commands to run to get the benefits of DSEE 7 in your deployment.

The entry compression feature is also available in the technology that will power future versions of Sun Directory Enterprise Edition: the OpenDS project. In OpenDS, there are 2 options to reduce the size of entries stored in the database. The first one is called entry compaction, and it’s enabled by default. The entry compaction feature removes all references to attribute names and replace them with small identifiers. The second option is actually entry compression which will use the popular ZLib algorithm. This option is not activated by default, but it’s just a command away :

<OPENDS_HOME>/bin/dsconfig -X -p 4444 -h localhost -D cn=Directory\ manager\
 -w password -n set-backend-prop \

 –backend-name userRoot –set entries-compressed:true

Below is the dsconfig usage for disabling entry compaction with OpenDS:

<OPENDS_HOME>/bin/dsconfig -X -p 4444 -h localhost -D cn=directory\ manager\
 -w password -n set-backend-prop \

 –backend-name backend –set compact-encoding:false

Here’s a table that compares the size of the databases of OpenDS 2.2.0 with no compat encoding, with it (default settings) and with compression enabled. The table compares the size of the entry record within the database as well as the overall size of the database which also includes indexes (default OpenDS settings).

Entry Count LDIF Entry Size Uncompacted Entry Size Compacted Entry Size Compressed Entry Size Uncompacted DB Size Compacted DB Size Compressed DB Size
100K 599 b 645 b 481 b 361 b 178.8 MB 163.20 MB 151.65 MB
-34% 25% -9.6% 7.1%
1M 603 b 649 b 485 b 364 b 1,515 MB 1,358 MB 1,243 MB
-34% 25% -11.5% 8.5%
10M 607 b 653 b 490 b 363 b 13,973 MB 12,416 MB 11,188 MB
-33% 26% -12.5% 9.9%

The percentages are computed from the reference value which is the default i.e. compacted. A negative value means an increased size, a positive one means a reduced size.

The second table compares the import times for the 3 different modes for storing entries, for the 3 sample data files.

Entry Count Uncompacted Compacted Compressed
100K 21 s 21 s 22 s
1.1% -3.5%
1M 106 s 107 s 112 s
0.5% -4.9%
10M 1006 s 1009 s 1101 s
0.2% -8.9%

Note: in this table, negative numbers represent increase in time required to import compared to the default settings.

Enabling compression does result in a smaller disk use with that sample data (fully random values), but does come with a performance penalty at least at import time, less than 10% but the penalty increases with the amount of entries.
If you’ve read Brad’s article on DSEE entry compression, you understand that the smaller the entries in the database, the more can be potentially cached in the Database Cache and the better the overall performances are. So if your entries are quite large, contain values that are strings, you should consider enabling the entry compression with OpenDS.

Changing from the default mode (compacted) to uncompacted mode does not give any real advantage in performance, but does increase the disk space usage, so I do not see the value of changing these settings in OpenDS.

Anyway, the benefits of having compact entries in the database are available today with Sun Directory Server Enterprise Edition 7 and Sun OpenDS Standard Edition 2.2, and are helping customers to reduce the overall cost of ownership of the directory services.

Technorati Tags: , , , ,

OpenDS 2.3.0-build001 is now available…

In December, the team had released OpenDS 2.2.0, a stable release of the LDAP directory server written for the Java platform. While the quality assurance team was testing and making sure the level of quality and reliability was there, the developers continued to add features on the trunk.

So today, I’m happy to announce the promotion of OpenDS 2.3.0-build001, the first development build on the path to OpenDS 2.4, a stable release currently planned for Summer 2010.

OpenDS 2.3.0-build001 is built from revision 6353 of our source tree and contains 2 major features :

  • The import code has been refined to improve performances, especially with very large data set. Tests have been run with 1,000,000,000 entries and the file imported in about 40 hours.
  • This version of OpenDS provides support for Subentries in LDAP [RFC 3672] and Collective Attributes for LDAP [RFC 3671]. In addition to the support of the Collective attribute standard, OpenDS supports a more user friendly notation, allowing to make use of Collective attribute definitions with standard attributes. More on this subject in a future article, although if you want to understand how to use it, Anton already wrote a description on the OpenDS Wiki.

These 2 features are just started to be seriously tested by the OpenDS team, so they cannot be considered as fully stable yet. There are already some issues being investigated, with the rebuild-index and verify-index commands. If you do find an issue with this OpenDS build, please report it in Issue Tracker.

The direct link to download the core server is:

The direct link to download the DSML gateway is:

We have also updated the archive that may be used to install OpenDS via Java Web Start. You may launch that using the URL, or visit for more information.

Detailed information about this build is available at, including the detailed change log

Major changes since OpenDS 2.2.0 include :

  • Multiple improvements on the Control Panel and the Setup including display of equivalent command-line commands, cosmetic aspects, messages, referral handling…
  • Multiple incremental improvements with the External Changelog, including reliability and performance
  • Revision 6190 – Upgraded the version of the Berkeley DB Java Edition to 4.0.73
  • Revision 6192 (Issue #4360) – Improves responsiveness of the OpenDS server when hammered with large searches and slow clients
  • Revisions 6198, 6209 (Issue #4371) – Solves an issue with PKCS12 certificates that do not have alias
  • Revision 6208 (Issue #4373) – Improves time to start the replication service
  • Revision 6292 (Issue #3601) – Adds the ability to update or delete schema attribute types and object classes definitions in the Control Panel
  • Revision 6332 (Issue #4472) – Fixes the way a Workflow element is checked to be a parent of another Workflow element
  • Revision 6334 (Issue #4464) – Solves an issue where reading the RootDSE could take too long when External Changelog is enabled
  • Revision 6336 (Issue #4477) – Increases the maximum size of DB log files from 10MB to 100MB
  • Revision 6351 – Support for localization in Catalan (ES_CA)
  • Revision 6353 (Issue #4489) – Resolve an issue where Java would not be detected even though JAVA_HOME variable was set

Technorati Tags: , , , , , , ,

Happy New Year !

Hello again after this too long break. Well, I wasn’t on vacation the whole time, but I find it hard to go back into writing mood. I also got distracted by the amount of snow that we received in Grenoble’s valley: yesterday there was still 30 cm of snow everywhere around my house.

Under the snow

Anyway, I’m back to this blogosphere, with the same hope that I will be posting more regularly than last year.

Opends Logo Tag

Looking back at 2009, it’s been an amazing year for the OpenDS project team.

In a year, we’ve released 3 important versions of OpenDS, with many features and innovation:

  • In January, OpenDS 1.2 added the Control Panel, SASL security, Support for JCEKS, enhancements in Access Controls and several Solaris / OpenSolaris specific features such as IPS packages, support for SMF and RBAC, …
  • In July, OpenDS 2.0 brought many performance improvements, several new features including Assured Replication, Recurring Tasks, Locale specific matching rules, and enhancements of the monitoring, the indexing, the ease of use, …
  • In December, OpenDS 2.2 added support for Fractional Replication, External Changelog, some date and time based matching rules, syntaxes extensions for Enumerations and Regular Expressions, up to 8 Masters in Replication, …

During the same period, the OpenDS Community has more than doubled, and so has the number of downloads of the OpenDS builds.

The OpenDS development continues. We have planned the release of OpenDS 2.4 in the middle of 2010. You can check the OpenDS RoadMap to see the features that are being worked on. If you’re using OpenDS in production, or if you’re building solutions that use the OpenDS LDAP directory server, please share your experience with the community. Send us details of your experiences or deployments. We will post them on the OpenDS wiki or Sun Adoption Stories blog.

While we’re still in January, let me wish all of you a happy and prosperous New Year 2010, and a long life to the OpenDS project.

Technorati Tags: , , , , ,