OpenDS 1.3.0-Build002 is now available

Opends Logo TagWe have just uploaded OpenDS 1.3.0-build002, built from revision 5124 of our source tree, to our promoted builds folder.

The direct link to download the core server is: http://www.opends.org/promoted-builds/1.3.0-build002/OpenDS-1.3.0-build002.zip

The direct link to download the DSML gateway is: http://www.opends.org/promoted-builds/1.3.0-build002/OpenDS-1.3.0-build002-DSML.war

We have also updated the archive that may be used to install OpenDS via Java Web Start. You may launch that using the URL http://www.opends.org/promoted-builds/1.3.0-build002/install/QuickSetup.jnlp, or visit https://www.opends.org/wiki/page/OverviewOfTheQuickSetupTool for more i

nformation.

Detailed information about this build is available at http://www.opends.org/promoted-builds/1.3.0-build002.

Upgrade from 1.0 or 1.2 is still broken, but upgrade from 1.3.0-build001 should work.

Major changes that have been incorporated since the last promoted build (OpenDS 1.3.0-build001) include:

  • Revision 5052 (Issue #3823) – Correct an issue that caused the ds-privilege-name to cause an unchecked exception.
  • Revision 5054 (Issue #3832) – Provide ergonomics style settings for common server properties.
  • Revision 5055 (Issue #3831) – Add dsconfig support for a NullBackend.
  • Revision 5058 (Issue #3824) – Fix a null pointer exception that occurred if an offline import was done on an NDB back end.
  • Revision 5059 (Issue #3821) – NDB back end – do not normalize user provided DN forms when storing.
  • Revision 5060 (Issue #3816) – Fix a problem with replication when uppercase characters were included in DNs.
  • Revision 5061 (Issue #3851) – Force the monitor back end to return parent monitoring entries before their children.
  • Revision 5062 (Issue #3804) – Improve replication monitoring.
  • Revision 5064 (Issue #3678) – Prevent dsconfig from displaying the corresponding command when the –displayCommand option is not specified.
  • Revision 5066 (Issue #3860) – Fix a problem that prevented the server from finding a certificate in a JKS keystore if the alias contained uppercase characters.
  • Revision 5067 (Issue #2993) – Print the server’s runtime information in the logs when the server is started by the setup command.
  • Revision 5070 (Issue #3861) – Update the control panel to handle the new monitor attributes.
  • Revision 5071 (Issue #3764) – Fix a problem that caused the server to hang when the replication configuration was changed during a total update.
  • Revision 5072 (Issue #839) – Add a specific code for LDAPException handling.
  • Revision 5074 (Issue #3863) – Fix a dsconfig exception when trying to set the subject attribute in a Subject DN To User Attribute Certificate Mapper.
  • Revision 5079 – Fix indexed search scoping.
  • Revision 5083 (Issue #3525) – Ensure that attribute modifications are replicated for modDN operations.
  • Revision 5085 (Issue #3837) – Fix a problem that caused a subtree ldapsearch operation to return the same result twice.
  • Revision 5086 – Simplify index configuration and add configuration placeholders for future use.
  • Revision 5087 (Issue #3402) – Fix a replication conflict resolution issue that caused a failure to resolve a double mod_rdn of the same entry.
  • Revision 5088 (Issue #3714) – Ensure the ldapsearch returns exit code 10 when it receives a referral.
  • Revision 505091 (Issue #3873) – Fix a problem that caused certain commands to use the java.properties of other commands.
  • Revision 5095 (Issue #3300) – Fix a problem that prevented the server from starting if the back end of a replicated suffix was disabled.
  • Revision 5099 (Issue #3315) – Fix a problem that caused dsreplication initialize-all to return before the peer had completed initialization.
  • Revision 5104 (Issue #3853) – NDB back-end : make sure that blob handles are wired to their result sets to prevent out of order retrieval when iterating.
  • Revision 5105 (Issue #3886) – Fix an issue that prevented online import-ldif from using the skipfile.
  • Revision 5107 (Issue #3877) – Fix an unexpected error that occurred when enabling safe_data mode.
  • Revision 5110 (Issue #3854) – Fix various issues that caused uninstall to fail on Windows, due to log files.
  • Revision 5112 (Issue #3887) – Fix a problem that caused changes in the dynamic assured replication configuration to cause timeouts.
  • Revision 5113 (Issue #3829) – Fix a problem that occurred when dsreplication enable was used with “localhost” as the host name for –host1 and –host2.
  • Revision 5115 (Issue #3833) – Fix a corrupt index in a replication topology.
  • Revision 5117 (Issue #3827) – NDB back end : fix a problem that prevented the NDB back end from being enabled dynamically with dsconfig.
  • Revision 5119 (Issue #3889) – Allow replicas to know about the state of the other replicas in the topology.
  • Revision 5122 (Issues #3884 and 3867) – Fix two issues that prevented GSSAPI SASL authentication from working as expected.

Technorati Tags: , , , , ,

OpenDS, Sun Labs and Java teams collaborate on performances

Opends Logo TagIn the past months, Matthew Swift, OpenDS core server lead developer, has been focusing most of his work on improving OpenDS server’s performances, trying to leverage as many tools as he could.

DukewithhelmetSometime, tools are not enough, and Matt started to discuss with Laurent Daynes, researcher on multitasking virtual machines and also based in the Sun Grenoble Engineering Center, about some strange behaviors of the JVM when trying to benchmark OpenDS with huge databases and caches. A few weeks after, Tony Printezis, expert in the Garbage Collector, came to Grenoble and got interested in our project and experiences. For the last few months, they’ve been working together, exchanging instrumented JVM, logs, results and ideas… The exchanges are valuable for both the HotSpot JVM team and the OpenDS team, as together we’re really pushing Java and OpenDS to the limits, with a real case scenario.

For the OpenDS team, being able to tap in directly into the brains behind Java or ZFS, is a huge advantage.

You can find more details on Matt’s blog.

Technorati Tags: , , , , ,

Alive and Kicking…

Opends Logo Tag
The OpenDS project was launched almost 3 years ago, and while it took us some time to release a first stable release and we had a hiccup, the project has been constantly active.

An illustration of the activity can be found on Ohloh.net but unfortunately those statistics are not refreshed on a regular basis and the OpenDS ones haven’t been updated since August 2008.

Another illustration is this colorful visualization of the history of commits in the OpenDS source code repository, also known as CodeSwarm, that I’ve put together. Simply enjoy !


Technorati Tags: , , , , ,

LDAP Directory Services performance: optimize the filesystem cache !

Getting the best performance of an LDAP directory server is a difficult task as there are many parameters to take into account, especially with the hardware itself. The CPU clock matters for processing thousands of requests per second as fast as possible. The amount of memory controls how much caching can be done for optimization. The storage subsystem has has some importance on both read and write operations, both in term of throughput and response time. The network interfaces sets how fast to read and respond to client applications.

But the filesystem and how it is configured is also proven to be part of the equation.

Brad Diggs, aka “The Zone Manager” and Senior Directory Architect,has posted a long, detailed and extremely well written article on Filesystem cache optimization strategies, comparing UFS and ZFS, providing tuning tips for both of them.

To quote one of my co-worker:

“after I applied some of those changes, we went from 850 modifications per second to 1100 modifications per second, but more importantly the variation was drastically reduced, providing more constant performance”.

If you’re looking to optimize the performances of Sun Directory Server, or even OpenDS, this article is a must read, and make sure you keep a bookmark on it.

Technorati Tags: , , , , ,

OpenDS Tab Sweep March 17

I’m not doing regular tab sweep but there has been several interesting things said on or about OpenDS in the last few days.

Technorati Tags: , , ,

Apprendre LDAP en école d’ingenieur…

Ces dernières semaines, je suis allé donner des cours sur LDAP aux étudiants Français.

Le 24 Février, j’étais à l’INSA à Lyon pour un cours en amphi de 2 heures pour une introduction sur LDAP et les Serveurs d’Annuaire, devant une promotion d’environ 120 étudiants. Le lendemain, Sylvain et 4 autres ingénieurs du Sun Grenoble Engineering Center ont poursuivi la formation par 4 sessions de 2 heures de travaux pratiques, mettant en oeuvre OpenDS et GlassFish. La présentation en Francais est disponible.

Le 5 Mars, à l’invitation de Julien Ponge, promoteur du projet IzPack, j’etais à l’ISIMA, l’école d’ingénieur de Clermont-Ferrand, pour une conférence sur LDAP et le projet OpenDS. Une présentation suivie par une trentaine d’étudiants. La présentation en Français est ici.

Technorati Tags: , , , , ,

Learning LDAP in Universities and Engineering Schools…

In the last few weeks, I’ve been involved with giving lectures on LDAP to French students (in French).

On February 24th, I was at INSA Lyon for a 2 hours lecture introducing LDAP and Directory Services to an audience of approximately 120 students. The next day, Sylvain and 4 other engineers from the Sun Grenoble Engineering Center conducted 4 sessions of a 2 hours workshop on LDAP, Directory Services with OpenDS. Slides in French are available (and the English version as well).

On March 5th, invited by Julien Ponge, I was at ISIMA, the Engineering School from Clermont-Ferrand for a conference about LDAP and OpenDS. The talk was attended by about 30 students. Slides in French are here.

Technorati Tags: , , , , ,

OpenDS 1.3.0-Build001 is now available

Opends Logo TagWe have just uploaded OpenDS 1.3.0-build001, built from revision 5050 of our source tree, to our promoted builds folder.

OpenDS 1.3.0-build001 is the first promoted build from the trunk past the 1.2.0 stable release, and the first of a series leading to OpenDS 2.0 sometime in June.

There are many updates in this build, some internal code refactoring (like the ASN1 library and support), some performance improvements, some new features. Code for a new backend has been committed, but is not built by default. This backend provides a remote access to the NDB database used by MySQL Cluster.

Happy testing…

The direct link to download the core server is: http://www.opends.org/promoted-builds/1.3.0-build001/OpenDS-1.3.0-build001.zip

The direct link to download the DSML gateway is: http://www.opends.org/promoted-builds/1.3.0-build001/OpenDS-1.3.0-build001-DSML.war

We have also updated the archive that may be used to install OpenDS via Java Web Start. You may launch that using the URL http://www.opends.org/promoted-builds/1.3.0-build001/install/QuickSetup.jnlp, or visit https://www.opends.org/wiki/page/OverviewOfTheQuickSetupTool for more i

nformation.

Detailed information about this build is available at http://www.opends.org/promoted-builds/1.3.0-build001.

Major changes incorporated since OpenDS 1.2.0 include:

  • Revision 4714 – L10n localization of generated files.
  • Revision 4715 – Fix double extension loading when instance and install are in the same directory.
  • Revision 4717 (Issue #3621) – Ensure that the import-ldif command uses the correct default port (4444).
  • Revision 4718 – Upgrade the je.jar to verson 3.3.75.
  • Revision 4719 (Issue #3644) – Fix an issue that caused dsconfig to fail without an error message.
  • Revision 4722 – Change the TDES key size to use an effective key size of 112.
  • Revision 4723 & 4776 (Issues #3640 and 497) – Refactor the replication code to make it more generic and provide the assured replication feature.
  • Revision 4728 – Make the generic menu bar class public so that potential extensions of the control panel can use it.
  • Revision (Issue #3639) – Fix an issue that prevented a restore from being applied to the correct backend when more than one backend was defined.
  • Revision 4735 (Issue #3653) – Ensure that the status command displays the LDIF Connection Handler in its list of connection handlers.
  • Revision 4748 – Provide the ability to create a monitor provider with hierarchical naming, modify the Network Monitor provider names to include hierarchical naming, provide basic monitoring objects, add monitoring instrumentation for connection handlers, and add the ability to do a subtree search in the monitor backend.
  • Revision 4749 – Add two monitor objects.
  • Revision 4753 (Issue #3641) – Register service tags from SVR4 pkg installs.
  • Revision 4759 (Issue #274) – Provide support for recurring tasks.
  • Revision 4764 (Issue #3668) – Ensure that the Control Panel displays connection handler listen addresses correctly.
  • Revision 4765 – Improvements to the assured replication feature.
  • Revision 4768 (Issue #3657) – Correct a problem with the “Save” button in the “Manage Entries” panel.
  • Revision 4769 – Extend GenericDialog so that its extending classes can use a customized message for the title.
  • Revision 4770 (Issue #262) – Provide a plug-in for Collation/Internationalization.
  • Revision 4772 (Issue #3671) – Provide a configuration completed listener.
  • Revision 4773 (Issue #3645) – Generate aggregation constraints correctly.
  • Revision 4775 (Issue #3667) – Make dsconfig usage messages consistent.
  • Revision 4779 – Improvements to the assured replication feature.
  • Revision 4781 – Provide support for dynamic domain group id reconfiguration in replication servers.
  • Revision 4782 (Issue #3674) Provide support for backup and export in the tasks back end.
  • Revision 4783 – Allow for removal of backups in the schema back end.
  • Revision 4788 – Replace Latin1 characters with valid escaped UTF-8 characters in DNs/RDNs.
  • Revision 4791 – Implement support for click-through license approval in quicksetup.
  • Revision 4792 (Issue #3676) – Fix a problem that prevented ldapmodify from processing the ;binary transfer option.
  • Revision 4795 – Provide support for NULL back ends.
  • Revision 4800 (Issue #3694) – Fix the BER encoding/decoding for negative integers.
  • Revision 4803 (Issue #3689) – Fix an issue in which the Control Panel displayed the incorrect hostname.
  • Revision 4804 (Issue #3688) – Remove unexpected html tags in the output of the status command and provide support for connection handlers with multiple listen addresses.
  • Revision 4805 – Allow the extension of dsreplication userdata objects, extend the server descriptor used by the internal administrator, allow back ends to be disabled if the server is configured in manual mode.
  • Revision 4807 (Issue #3685) – Correct a Swing repainting problem in the control panel.
  • Revision 4808 (Issue #3695) – Improve the way in which the example plugin ant file handles message creation on Windows.
  • Revision 4809 (Issue #3640) – Improvements to the generic replication service.
  • Revisions 4810, 4811, 4817, 4818, 4819 & 4843 – Add support for client connection affinity.
  • Revision 4814 – Improve the license acceptance mechanism.
  • Revision 4815 – Enable OpenDS servers to load data from Directory Server Enterprise Edition servers.
  • Revision 4820 (Issue #3700) – Correct the handling of failure of a workflow element creation.
  • Revision 4822, 4842 – Fixes to the assured replication mechanism.
  • Revision 4823 (Issue #3699) – Correct the way in which the server handled the password Expired Control during a BIND operation, if the password had been reset.
  • Revision 4827 (Issue #3698) – Fix a problem that prevented changing the Directory Manager password with the Control Panel.
  • Revision 4829 – Allow import-ldif to load VLV indexes.
  • Revision 4830 (Issue #3701) – Correct the way in which setup manages back ends when replication involves multiple base-dns.
  • Revision 4831 (Issue #3709) – In the Control Panel, change the value of the “Backup Path” to the instance path (rather than the installation path)
  • Revision 4833 (Issue #2829) – Fix a problems that occurred when configuring and unconfiguring replication servers.
  • Revision 4835 (Issue #3710) – Fix a Control Panel error that occurred when creating a new base DN with automatically generated data.
  • Revision 4840 (Issue #3711) – Allow remote server debugging.
  • Revision 4844 – Reduce replication overhead.
  • Revision 4855 (Issue #3579) – Ensure that import-ldif countRejects works as expected.
  • Revision 4856 (Issue #3640) – Refactor replication code to make it more generic.
  • Revision 4858 & 4859 (Issue #3683) – Fix a problem that caused the replication conflict resolution code to assumes a conflict when replication replayed a DELETE on an entry with child entries.
  • Revision 4861 – Enable assured replication monitoring.
  • Revision 4862 (Issue #3716) – Fix a problem that caused start-ds.bat to use the wrong environment variable for passing arguments.
  • Revision 4863 (Issue #3717) – Allow command line output and error stream to be changed.
  • Revision 4870 & 4902 (Issue #3724) – Instead of using a hardcoded trust manager provider and algorithm, takes the default algorithm of the JVM.
  • Revision 4872 (Issue #3723) – Fix the ACI SSF bind rule != operator.
  • Revision 4874 (Issue #3718) – Correct a problem that caused the -A, –typesOnly option to be ignored by ldapsearch.
  • Revision 4887 (Issue #3131) – Fix a problem that caused upgrades using the webinstaller to hang on Windows.
  • Revision 4896 (Issue #3731) – Remove the status-panel command (which has been replaced by control-panel.
  • Revision 4898 (Issue #3733) – When performing a search on the root DSE to retrieve the list of namingContexts, display only the public naming contexts visible through the current network group.
  • Revision 4905 (Issue #3736) – Correct a problem that prevented replication dynamic purge delay changes from being taken into account.
  • Revision 4909 (Issue #3750) – Improve the behaviour when forcing a password change after admin reset.
  • Revision 4912 – Extend the directory server so that other tools launching it can impose their own usage message.
  • Revision 4921 – Allow overwriting classes to specify whether the schema should be read.
  • Revision 4922 (Issue #3726) – Fix a problem that prevented the RealAttributesOnly Control from working when the types-only search option was enabled.
  • Revision 4923 – Add 508 compliance to the Browse Schema panels on the Control Panel.
  • Revision 4925 – Update the ServiceTag data configuration with an optional configuration directory.
  • Revision 4926 & 4928 (Issue #3760 & 3761) – Fix a problem that caused dsconfig to exit abruptly when creating a component with a missing parent or when creating a component with the same name.
  • Revision 4931 (Issues #3446 & 3726) – Introduce comprehensive unit tests for checking attribute filtering in search operations, add improvements to the virtual attribute provider API, add improvements to virtual attribute processing during Entry duplication.
  • Revision 4932 (Issues #3682 & 3643) – Addition of localized resource files with new translations.
  • Revision 4937 (Issue #3561) – Fix an issue that prevented aliased attributes from being returned properly.
  • Revision 4941 (Issue #3763) – Provide a dsconfig error message if a type of object does not exist.
  • Revision 4942 – Fix a problem that prevented entry locks from being released.
  • Revision 4944 – Update the SVR4 factory to avoid conflict with the IPS factory.
  • Revision 4946 – Provide support for collation indexing.
  • Revision 4949 – Fix a problem with IPv4 wild card pattern matching that prevented address masks of the form “*.*.*.*” from ever matching an IPv4 address.
  • Revision 4950 (Issue #3734) – Make network group policies extensible.
  • Revision 4956, 4957 & 4961 – ASN1 refactoring.
  • Revision 4958 – Make it possible for users to configure the Control Panel refresh period.
  • Revision 4962 (Issue #3391) – Fix a problem that prevented custom DIT Structure Rules from being added to 99-user.ldif.
  • Revision 4963 (Issue #3606) – Fix a problem that prevented the isMemberOf attribute from working for dynamic groups.
  • Revision 4964 – Add a user interface to display global monitoring information and connection handler monitoring information.
  • Revision 4969 (Issue #3775) – Fix an intermittent unit test failure in the NetworkGroupTest and add support for finalizing the NetworkGroupConfigManager.
  • Revision 4972 – Add the displayCommand and commandFilePath options to the dsreplication command. Also, fix an issue that prevented some passwords passed in the command-line as file arguments from being taken into account in interactive mode.
  • Revision 4981 – Add a check for a null SASL Context and fix an error message in the EXTERNAL Digest Handler.
  • Revision 4986 – First phase of support for an NDB backend.
  • Revision 4991 (Issue #3795) – Fix a problem that caused TLS to fail when adding or modifying a large attribute.
  • Revision 4993 (Issue #3797) – Ensure that the directory server logs it’s instance path at startup.
  • Revision 4998 (Issue #3774) – Fix a problem that prevente the sort control from working with collation matching rules.
  • Revision 4999 (Issue #3801) – If a connection handler has no address defined in its configuration, only show the port of the connection handler, rather than a fictitiuos address.
  • Revision 5004 (Issue #3805) – Support for SASL Connection Security (Phase2).
  • Revision 5006 (Issue #3806) – Fix a problem that caused an ldapsearch operation on the rootDSE to return error 255 if no back end was defined.
  • Revision 5008 (Issue #3800) – Redesign the Monitoring General Information panel to handle the display of a large number of operations.
  • Revision 5009 (Issue #3806) – Prevent schema elements in 06-compat.ldif from being deleted by a user.
  • Revision 5010 (Issue #3803) – Correct ASN.1 encoding of VLVResponseControl so it sends the result code as a BER Enumerated instead of Integer.
  • Revision 5011 (Issue #3809) – Add a flush() call to the ASN1Writer to make sure that all bytes are sent out.
  • Revision 5017 (Issue #3812) – Display a warning if the user provides the wrong replication port for an existing replication server.
  • Revision 5018 (Issue #3804) – Improve replication monitoring.
  • Revision 5023 (Issue #3815) – Declare NS password expire and expiring in the SupportControl in the RootDSE.
  • Revision 5024 (Issue #3820) – Fix an issue that prevented the removal of spaces at the end of string with Non-ASCII characters.
  • Revision 5025 (Issues #3687 & 3690) – Fix a problem that caused incorrect SNMP values to be displayed.
  • Revision 5027 (Issue #3817) – Fix a problem that caused a the configuration to break if a 2nd workflow with cn=config baseDn can break the configuration.
  • Revision 5029 & 5030 (Issue #3802) – Fix incorrect script launcher return codes.
  • Revision 5031 & 5035 (Issue #3826) – ix a problem that prevented ldapsearch from prompting for a bind password if the option -w was not provided.
  • Revision 5032 (Issue #3798) – Fix a problem that prevented the server from using more than 50% of machine memory.
  • Revision 5033 (Issues #3808 & 3810) – Fix the hasSubordinates attribute under cn=monitor and prevent a Parent DN entry from being returned when using a child entry as the search base DN.
  • Revision 5036 (Issue #3765) – Correct the way in which export-ldif handles relative paths.
  • Revision 5038 (Issue #3834) – Fix a problem that caused an exception when dsconfig was used to configure a Network Group QOS Policy in interactive mode.
  • Revision 5040 (Issue #3840) – Fix a problem in the license file that broke the Java Web Start installer.
  • Revision 5042 (Issue #3849) – Allow read-only properties to be modified at component creation time.
  • Revision 5043 (Issue #3846) – Provide routines in the WorkflowElement class to retrieve child workflow elements.
  • Revision 5044 (Issue #3841) – Fix performance degradations observed when using LDAPWriter.
  • Revision 5045 (Issue #3852) – Fix a problem that caused dsconfig to exit when parent component did not exist, if run in interactive mode.
  • Revision 5046 (Issue #3844) – Ensure that replication changes are not lost when ReplicationDomain.publish is called by several threads.
  • Revision 5050 (Issue #3845) – Fix an IllegalStateException that occurred during Schema Backend initialization.

Technorati Tags: , , , , ,

OpenDS tips for the developer: One click debug and profiling with NetBeans

If you are using the NetBeans IDE, you can check out OpenDS code from the SVN repository and create immediately a free form project. Debugging OpenDS or Profiling it is then immediately available in a single click as all the necessary hooks are provided in the OpenDS build.xml file.

Simply click on the Debug Project icon in the NetBeans IDE toolbar to start a debugging session of the OpenDS server, or click on the “Profile Project ICon” for a profiling session.

Nbopends

If you want more advanced integration of OpenDS with the NetBeans IDE, you can download the sample nbproject.zip fiile from the OpenDS Documentation wiki and follow the instructions from this page.

Technorati Tags: , , , , , , , ,

OpenDS Tips: Adding schema from OpenLDAP

Opends Logo TagThe OpenDS schema is slightly different from the OpenLDAP one, but it’s quite simple to convert schema files from one format to another.

OpenDS, like Sun Directory Server Enterprise Edition and Fedora DS, uses a strict RFC 4512 and LDIF format.

In OpenLDAP, the actual text of the schema definition is similar and described using the RFC 4512 notation but uses the printer friendly notation, similar to the textual description in RFC documents.

So when converting schema files from OpenLDAP, for use in OpenDS, there are mainly 4 differences to take care of:

  • In OpenLDAP, an attribute definition begins with “attributetype” while in OpenDS it begins with “attributetypes: “
  • Similarly, in OpenLDAP, an object class definition has an “objectclass” prefix while it is “objectclasses: “
  • OpenDS follows the LDIF conventions that the continuation line begins with a single space character, and that an empty line is an entry separator
  • Finally, OpenDS schema files have a .ldif extension and only this extension is considered when loading schema from the config/schema directory.

The following python script can be used to convert an OpenLDAP schema file to a format usable by OpenDS (as well as Sun Directory Enterprise Edition). The script also recursively expands the OID macro format used in OpenLDAP schema files.

For now, syntax definitions are currently ignored as they cannot be loaded in OpenDS as they require associated code.

Usage is quite simple: schema-convert.py -o result.ldif openldap-schema-file

Enjoy and don’t hesitate to send feedback, suggestions for improvements.

Update on March 15: I’ve added support for name prefixed OIDs substitution as suggested by Martin Gwerder.

Update on April 9: OpenDS schema files uses the .ldif extension, and only files with this extension are loaded by the server from the config/schema directory.

Update on July 31: Now checking and removing quotes around Sup or Syntaxes values.

 


#!/usr/bin/env python
# encoding: utf-8
"""
schema-convert.py
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at
# trunk/opends/resource/legal-notices/OpenDS.LICENSE
# or https://OpenDS.dev.java.net/OpenDS.LICENSE.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at
# trunk/opends/resource/legal-notices/OpenDS.LICENSE.  If applicable,
# add the following below this CDDL HEADER, with the fields enclosed
# by brackets "[]" replaced with your own identifying information:
#      Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
#
#      Copyright 2009 Sun Microsystems, Inc.
Created by Ludovic Poitou on 2009-01-28.
This program converts an OpenLDAP schema file to the OpenDS schema file format.
"""
import sys
import getopt
import re
import string
help_message = '''
Usage: schema-convert.py [options] <openldap-schema-file>
options:
\t -o output : specifies the output file, otherwise stdout is used
\t -v : verbose mode
'''
class Usage(Exception):
def __init__(self, msg):
self.msg = msg
def main(argv=None):
output = ""
seclineoid = 0
IDs = {}
if argv is None:
argv = sys.argv
try:
try:
opts, args = getopt.getopt(argv[1:], "ho:v", ["help", "output="])
except getopt.error, msg:
raise Usage(msg)
# option processing
for option, value in opts:
if option == "-v":
verbose = True
if option in ("-h", "--help"):
raise Usage(help_message)
if option in ("-o", "--output"):
output = value
except Usage, err:
print >> sys.stderr, sys.argv[0].split("/")[-1] + ": " + str(err.msg)
print >> sys.stderr, "\t for help use --help"
return 2
try:
infile = open(args[0], "r")
except Usage, err:
print >> sys.stderr, "Can't open file: " + str(err.msg)
if output != "":
try:
outfile = open(output, "w")
except Usage, err:
print >> sys.stderr, "Can't open output file: " + str(err.msg)
else:
outfile = sys.stdout
outfile.write("dn: cn=schema\n")
outfile.write("objectclass: top\n")
outfile.write("")
for i in infile:
newline = ""
if not i.strip():
continue
#if i.startswith("#"):
#	continue
if re.match("objectidentifier", i, re.IGNORECASE):
# Need to fill in an array of identifiers
oid = i.split()
if not re.match ("[0-9.]+", oid[2]):
suboid = oid[2].split(':')
IDs[oid[1]] = IDs[suboid[0]] + "." + suboid[1]
else:
IDs[oid[1]] = oid[2]
continue
if seclineoid == 1:
subattr = i.split()
if not re.match("[0-9.]+", subattr[0]):
if re.match (".*:", subattr[0]):
# The OID is an name prefixed OID. Replace string with the OID
suboid = subattr[0].split(":")
repl = IDs[suboid[0]] + "." + suboid[1]
else:
# The OID is a name. Replace string with the OID
repl = IDs[subattr[0]]
newline = string.replace(i, subattr[0], repl, 1)
seclineoid = 0
if re.match("attributetype ", i, re.IGNORECASE):
newline = re.sub("attribute[tT]ype", "attributeTypes:", i)
# replace OID string with real OID if necessary
subattr = newline.split()
if len(subattr) < 3:
seclineoid = 1
else:
if not re.match("[0-9.]+", subattr[2]):
if re.match (".*:", subattr[2]):
# The OID is an name prefixed OID. Replace string with the OID
suboid = subattr[2].split(":")
repl = IDs[suboid[0]] + "." + suboid[1]
else:
# The OID is a name. Replace string with the OID
repl = IDs[subattr[2]]
newline = string.replace(newline, subattr[2], repl, 1)
if re.match("objectclass ", i, re.IGNORECASE):
newline = re.sub("object[cC]lass", "objectClasses:", i)
# replace OID String with real OID
subattr = newline.split()
if len(subattr) < 3:
seclineoid = 1
else:
if not re.match("[0-9.]+", subattr[2]):
if re.match (".*:", subattr[2]):
# The OID is an name prefixed OID. Replace string with the OID
suboid = subattr[2].split(":")
repl = IDs[suboid[0]] + "." + suboid[1]
else:
# The OID is a name. Replace string with the OID
repl = IDs[subattr[2]]
newline = string.replace(newline, subattr[2], repl, 1)
# Remove quoted syntax.
if re.search("SYNTAX\s'[\d.]+'", newline):
# Found a quoted syntax in an already updated line
newline = re.sub("SYNTAX '([\d.]+)'", "SYNTAX \g<1>", newline)
else:
if re.search("SYNTAX\s'[\d.]+'", i):
# Found a quoted syntax in the original line
newline = re.sub("SYNTAX '([\d.]+)'", "SYNTAX \g<1>", i)
# Remove quoted SUP
if re.search("SUP\s'[\w\-]+'", newline):
# Found a quoted sup in an already updated line
newline = re.sub("SUP '([\w\-]+)'", "SUP \g<1>", newline)
else:
if re.search("SUP\s'[\w\-]+'", i):
# Found a quoted sup in the original line
newline = re.sub("SUP '([\w\-]+)'", "SUP \g<1>", i)
# transform continuation lines with only 2 spaces
if re.match("  +|\t", i):
if newline != "":
newline = "  " + newline.strip() + "\n"
else:
newline = "  " + i.strip() + "\n"
if newline != "":
outfile.write(newline)
else:
outfile.write(i)
outfile.close()
if __name__ == "__main__":
sys.exit(main())

Technorati Tags: , , , , , , ,

What’s your OpenDS story ?

Opends Logo Tag TransHave you deployed OpenDS, for proof of concept, pilot or production use ? If so, read on !

We’re adding OpenDS to the “Stories” blog, highlighting real-world use of OpenDS. If you have deployed OpenDS and are using it, please take a look at our standard questionnaire (we now have a standard form to gather data) , and if possible fill it out and mail it to the following email address: stories@sun.com

Alternately, write down and publish on a blog, or create a video bout your implementation and send us the link. We want to show our appreciation for sharing, so for the top 30 stories we receive we will send you a free t-shirt (please include an address in your submission).

Thank for your continued participation in OpenDS !

OpenDS 1.2 is now available to OpenSolaris users

OpenDS 1.2 was released last month and the goal of this release was to make it available as part of OpenSolaris.

And I’m happy to announce that starting with OpenSolaris build 107, you can now get and install OpenDS with the pkg(5) command from http://pkg.opensolaris.org/dev/

OpenDS in OpenSolaris Pkg Repository

Technorati Tags: , , , , ,

OpenDS Tips: Copying instances of OpenDS.

Opends Logo TagOne of the things we are very proud of with the OpenDS project, is its ease of use, and this is very well illustrated with the QuickSetup installer.

Based on our past experience, we’ve made sure that OpenDS server has no use of absolute paths.

For the developer, this is really handy. It allows you to move an installed OpenDS instance from one directory to another very easily: you just stop the server, move the instance to a larger or faster disk, and restart it.

Similarly, you can also create a new instance of the server by copying the installed server to a new location (instead of moving it). If you do this to run both instance, don’t forget to edit the dse.ldif file to change the port numbers (LDAP, LDAPS and Admin), and possibly the replication configuration if replication was enabled on the initial server.

In our daily tests with OpenDS, we use this capability a lot, especially when we run benchmarks. After having installed, configured and tuned the OpenDS instance, we make a copy that we start and run the tests against. When finished, we capture the desired results, and delete the instance. And we repeat the steps, making sure we have consistent results.

As all of our tests are done with multi-master replication enabled, we do tests with 2 instances on separated machines. So, we need to restore 2 instances to their initial state to reproduce a test. The ability to do “cp -r RefInstance/ TestInstance/” on both machine, is definitely a key advantage for us.

Note that if you install OpenDS 1.2 on OpenSolaris from the IPS package repository, there is a separation between the installation path (where the binaries and default configuration is stored) and the instance path (where the data and live configuration is stored). The instance path is stored in a file named instance.loc which is under /etc/opends/. Moving instances can be done, as long as the instance.loc file gets updated (manually).

Technorati Tags: , , , , ,