DSEE 6.1 qualified with Sun Cluster 3.2…

SC3.2 Console Sun Java System Directory Server Enterprise Edition 6.0 delivered a Sun Cluster HA agent for Directory Server that was qualified with Sun Cluster 3.1 (which is the version in Java Enterprise System 5).

We’ve done a quick round of sanity tests of Sun Java System Directory Server 6.1 with Sun Cluster 3.2, and we can now claim the combination officially supported.

There is still a defect with Directory Server 6.1 that prevents the server from starting when SC 3.2 has been installed on a machine ( defect #6563445: Directory Server fails to start if rlimit is set to infinity). This defect has been fixed in the coming 6.2 version. Meanwhile Sun support organization will be happy to deliver a hot-fix for this on demand.

Technorati Tags: , , ,

A guided tour of the DSEE documentation set…

0595352170M

Mark, my office mate and long time technical writer in the Directory Services team, has just posted a detailed description of the Directory Server Enterprise Edition 6 documentation set. If you’re looking for information about DSEE and don’t know where to start from, I suggest that you go there and save a reference to it.

Technorati Tags: , , , ,

Saving Harry Potter…

I was out this week-end and when I came back yesterday evening, the expected brown box was waiting for me in the mailbox. So I opened it religiously and took a long look at the cover. And I put down the book on the pile of books that I will be taking with me for my vacations, begining of August. Oh yes, I am curious to see what will happen to Harry and how the story is ending, but I’m not that desperate that I can’t wait for a couple of weeks. At least then I will have the time to read chapters after chapters with limited distraction and interruptions. A good way to forget about the office and dive into vacation mode… I’m looking forward to it.

Img 0318

Technorati Tags: ,

OpenDS presentation at LDAPcon 2007

I’ve just received the confirmation that my proposal for an OpenDS presentation at the 1st International LDAPv3 Conference has been accepted.

The list of all presentations has been published. There will be a good representation of the LDAP community with most open source projects, commercial products and Ph.D. candidates.

I hope to see many of you in Cologne (Köln) early September.

Technorati Tags: , , , , ,

Over 80 bugs killed in over a week…

Two weeks ago, I announced the OpenDS Summer 2007 Bugfest.

The bug hunt ended last Friday and it’s been a quite successful week with exactly 84 bugs being fixed during these 10 days (including the US and French public holiday), by 14 different committers. This effort led to the first weekly build of what will be the 1.0 version of OpenDS.

While the Sun team was strongly encouraged by management to participate, we’ve seen little involvement of the community.

This is certainly a sign that we still have a lot of work to do to broaden the community, get it involved, and for us to be more transparent and open.

As a small step in that direction, I’ve started to daily join the #opends IRC chat room on Freenode.net, and to encourage participation from my co-workers. Time will tell if this helps…

Technorati Tags: , ,

A long awaited feature…

Directory Server Enterprise Edition 6.x is built on SleepyCat Berkeley DB for the storage of the LDAP entries and indexes…

For many years, customers have requested that we provide a way to shrink the database files, reclaiming unused pages. Well, the version of the SleepyCat BDB that we are using with DS 6.1 now has a public API that offer this capability and we’ve added the feature in DSEE 6.1. There is now a new subcommand for dsadm the offline DS management CLI: repack.

The usage is the following:

ludo:dsee63 > ds6/bin/dsadm repack --help
Usage: dsadm repack [ -b ] INSTANCE_PATH SUFFIX_DN [SUFFIX_DN ...]
Repacks existing suffix
The accepted value for OPTIONS is:
-b, --backend
Enables to specify backend name instead of SUFFIX_DN
For global options, use dsadm --help.
INSTANCE_PATH  Path of the Directory Server instance
SUFFIX_DN      Suffix DN (Distinguished Name) to repack
For more information, see dsadm(1M).

It operates on a Suffix and all DB files for this suffix are compacted. This includes the main data file (id2entry), all index files but also the replication changelog file if the suffix is Replicated and configured as a Master or Hub Replica. The compaction will process up to 8 files in parallel, each in its own thread.

Because database compaction is very IO intensive and requires exclusive access to the database, the Directory Server must be stopped to run this.

The time to do the compaction varies a lot depending on the overall size of the database, the number of updates done on the data since the creation of the suffix or last compaction, and mostly the performance of the disk subsystem. Or course, the larger the DB and the more changes, the longer it’ll take but the more it’ll regain disk space. In our experience with a database of approximately 10 million entries and many changes done since creation, the compaction process took approximately a couple of hours (on a v20 z).

Definitely, database compaction is not something that you want to run in your weekly maintenance routine. The downtime of Directory Server may be too long. But if the available disk’s space is getting close to the low watermark, it is worth stopping the server and reclaiming some of the space, before thinking of expanding the disk partition.

Below is an example of the dc=example,dc=com DB files before and after compaction, and the commands used to do the compaction: stop the server, compact, start the server.

ludo:dsee63 > ll instances/ds1/db/example/
total 3640
drwx------   2 lpoitou  icnc        1536 Jul  3 10:45 .
drwxr-xr-x   3 lpoitou  icnc         512 Jul  9 17:39 ..
-rw-------   1 lpoitou  icnc          38 Jul  3 10:45 DBVERSION
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_aci.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 17:39 example_ancestorid.db3
-rw-------   1 lpoitou  icnc      147456 Jul  9 17:39 example_cn.db3
-rw-------   1 lpoitou  icnc      122880 Jul  9 17:39 example_entrydn.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_givenName.db3
-rw-------   1 lpoitou  icnc      835584 Jul  9 17:39 example_id2entry.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_mail.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_mailAlternateAddress.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_mailHost.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_member.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsCalXItemId.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nscpEntryDN.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsds5ReplConflict.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsLIProfileName.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nsRoleDN.db3
-rw-------   1 lpoitou  icnc      131072 Jul  9 17:39 example_nsuniqueid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_nswcalCALID.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_numsubordinates.db3
-rw-------   1 lpoitou  icnc       32768 Jul  9 17:39 example_objectclass.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_owner.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 17:39 example_parentid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pipstatus.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pipuid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdaccountlockedtime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdfailuretime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdgraceusetime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_seeAlso.db3
-rw-------   1 lpoitou  icnc      139264 Jul  9 17:39 example_sn.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_telephoneNumber.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_uid.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_uniquemember.db3
ludo:dsee63 > ds6/bin/dsadm stop /local/demo/dsee63/instances/ds1
Server stopped
ludo:dsee63 > ds6/bin/dsadm repack /local/demo/dsee63/instances/ds1 dc=example,dc=com
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example'
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', LDAP entries, size 827392 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index aci, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index ancestorid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index ancestorid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index aci finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index cn, size 139264 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', LDAP entries finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index cn finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailHost, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailHost finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index member, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index member finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsCalXItemId, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsCalXItemId finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nscpEntryDN, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nscpEntryDN finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsds5ReplConflict, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsds5ReplConflict finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsLIProfileName, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsLIProfileName finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsRoleDN, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsRoleDN finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsuniqueid, size 122880 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nswcalCALID, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nswcalCALID finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index nsuniqueid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index entrydn, size 122880 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index numsubordinates, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index numsubordinates finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index entrydn finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index givenName, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index givenName finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index parentid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index parentid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipstatus, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipstatus finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipuid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pipuid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdaccountlockedtime, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdaccountlockedtime finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdfailuretime, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdfailuretime finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdgraceusetime, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index pwdgraceusetime finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index seeAlso, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index seeAlso finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index sn, size 131072 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uid, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uid finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index sn finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index owner, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index owner finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index telephoneNumber, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index telephoneNumber finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uniquemember, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index uniquemember finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mail, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mail finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailAlternateAddress, size 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index mailAlternateAddress finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index objectclass, size 32768 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example', index objectclass finished, size now 16384 bytes.
[10/Jul/2007:14:11:46 +0200] - Repacking backend 'example' ended.
[10/Jul/2007:14:11:46 +0200] - Repack finished.
[10/Jul/2007:14:11:46 +0200] - Waiting for 6 database threads to stop
[10/Jul/2007:14:11:47 +0200] - All database threads now stopped
ludo:dsee63 > ll instances/ds1/db/example/
total 1000
drwx------   2 lpoitou  icnc        1536 Jul  3 10:45 .
drwxr-xr-x   3 lpoitou  icnc         512 Jul 10 14:11 ..
-rw-------   1 lpoitou  icnc          38 Jul  3 10:45 DBVERSION
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_aci.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_ancestorid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_cn.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_entrydn.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_givenName.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_id2entry.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_mail.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_mailAlternateAddress.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_mailHost.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_member.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsCalXItemId.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nscpEntryDN.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsds5ReplConflict.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsLIProfileName.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsRoleDN.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nsuniqueid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_nswcalCALID.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_numsubordinates.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_objectclass.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_owner.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_parentid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pipstatus.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pipuid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pwdaccountlockedtime.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_pwdfailuretime.db3
-rw-------   1 lpoitou  icnc       16384 Jul  9 16:38 example_pwdgraceusetime.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_seeAlso.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_sn.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_telephoneNumber.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_uid.db3
-rw-------   1 lpoitou  icnc       16384 Jul 10 14:11 example_uniquemember.db3
ludo:dsee63 > ds6/bin/dsadm start /local/demo/dsee63/instances/ds1
Server started: pid=15983
ludo:dsee63 >

Technorati Tags: , ,