London Identity Summit 2016

Yesterday, ForgeRock hosted the London Identity Summit, 2016 Series.

Mike Ellis, ForgeRock CEO, launching the London Identity Summit.

Attended by more than 300 customers, prospects, partners, the event was a great success. For a short summary of the event, check ForgeRock blog. Highlights, presentations, etc will all be available shortly at Meanwhile all my photos of the event are available here, and you can get a feel of the pulse of the event through the twitter stream (hashtag #IdentitySummit)

The next Identity Summit will be held in Paris on November 15th. I hope to see you there.


Storing JSON objects in LDAP attributes…

jsonUntil recently, the only way to store a JSON object to an LDAP directory server, was to store it as string (either a Directory String i.e a sequence of UTF-8 characters, or an Octet String i.e. a blob of octets).

But now, in OpenDJ, the Open source LDAP Directory services in Java, there is now support for new syntaxes : one for JSON objects and one for JSON Query. Associated with the JSON query, a couple of matching rules, that can be easily customised and extended, have been defined.

To use the syntax and matching rules, you should first extend the LDAP schema with one or more new attributes, and use these attributes in object classes. For example :

dn: cn=schema
objectClass: top
objectClass: ldapSubentry
objectClass: subschema
attributeTypes: ( NAME 'json'
objectClasses: ( NAME 'jsonObject'
SUP top MUST (cn $ json ) )

Just copy the LDIF above into config/schema/95-json.ldif, and restart the OpenDJ server. Make sure you use your own OIDs when defining schema elements. The ones above are samples and should not be used in production.

Then, you can add entries in the OpenDJ directory server like this:

$ ldapmodify -a -D cn=directory\ manager -w secret12 -h localhost -p 1389

dn: cn=bjensen,ou=people,dc=example,dc=com
objectClass: top
objectClass: jsonObject
cn: bjensen
json: { "_id":"bjensen", "_rev":"123", "name": { "first": "Babs", "surname": "Jensen" }, "age": 25, "roles": [ "sales", "admin" ] }

dn: cn=scarter,ou=people,dc=example,dc=com
objectClass: top
objectClass: jsonObject
cn: scarter
json: { "_id":"scarter", "_rev":"456", "name": { "first": "Sam", "surname": "Carter" }, "age": 48, "roles": [ "manager", "eng" ] }

The very nice thing about the JSON syntax and matching rules, is that OpenDJ understands how the values of the json attribute are structured, and it becomes possible to make specific queries, using the JSON Query syntax.

Let’s search for all jsonObjects that have a json value with a specific _id :

$ ldapsearch -D cn=directory\ manager -w secret12 -h localhost -p 1389 -b "dc=example,dc=com" -s sub "(json=_id eq 'scarter')"

dn: cn=scarter,ou=people,dc=example,dc=com
objectClass: top
objectClass: jsonObject
json: { "_id":"scarter", "_rev":"456", "name": { "first": "Sam", "surname": "Carter" }, "age": 48, "roles": [ "manager", "eng" ] }
cn: scarter

We can run more complex queries, still using the JSON Query Syntax:

$ ldapsearch -D cn=directory\ manager -w secret12 -h localhost -p 1389 -b "dc=example,dc=com" -s sub "(json=name/first sw 'b' and age lt 30)"

dn: cn=bjensen,ou=people,dc=example,dc=com
objectClass: top
objectClass: jsonObject
json: { "_id":"bjensen", "_rev":"123", "name": { "first": "Babs", "surname": "Jensen" }, "age": 25, "roles": [ "sales", "admin" ] }
cn: bjensen

For a complete description of the query  filter expressions, please refer to ForgeRock Common  REST (CREST) Query Filter documentation.

The JSON matching rule supports indexing which can be enabled using dsconfig against the appropriate attribute index. By default all JSON fields of the attribute are indexed.

In a followup post, I will give more advanced configuration of the JSON Syntax, detail how to customise the matching rule to index only specific JSON fields, and will outline some best practices with the JSON syntax and attributes.

OpenDJ: Monitoring Unindexed Searches…

FR_plogo_org_FC_openDJ-300x86OpenDJ, the open source LDAP directory services, makes use of indexes to optimise search queries. When a search query doesn’t match any index, the server will cursor through the whole database to return the entries, if any, that match the search filter. These unindexed queries can require a lot of resources : I/Os, CPU… In order to reduce the resource consumption, OpenDJ rejects unindexed queries by default, except for the Root DNs (i.e. for cn=Directory Manager).

In previous articles, I’ve talked about privileges for administratives accounts, and also about Analyzing Search Filters and Indexes.

Today, I’m going to show you how to monitor for unindexed searches by keeping a dedicated log file, using the traditional access logger and filtering criteria.

First, we’re going to create a new access logger, named “Searches” that will write its messages under “logs/search”.

dsconfig -D cn=directory\ manager -w secret12 -h localhost -p 4444 -n -X \
    create-log-publisher \
    --set enabled:true \
    --set log-file:logs/search \
    --set filtering-policy:inclusive \
    --set log-format:combined \
    --type file-based-access \
    --publisher-name Searches

Then we’re defining a Filtering Criteria, that will restrict what is being logged in that file: Let’s log only “search” operations, that are marked as “unindexed” and take more than “5000” milliseconds.

dsconfig -D cn=directory\ manager -w secret12 -h localhost -p 4444 -n -X \
    create-access-log-filtering-criteria \
    --publisher-name Searches \
    --set log-record-type:search \
    --set search-response-is-indexed:false \
    --set response-etime-greater-than:5000 \
    --type generic \
    --criteria-name Expensive\ Searches

Voila! Now, whenever a search request is unindexed and take more than 5 seconds, the server will log the request to logs/search (in a single line) as below :

$ tail logs/search
[12/Sep/2016:14:25:31 +0200] SEARCH conn=10 op=1 msgID=2 base="dc=example,
dc=com" scope=sub filter="(objectclass=*)" attrs="+,*" result=0 nentries=
10003 unindexed etime=6542

This file can be monitored and used to trigger alerts to administrators, or simply used to collect and analyse the filters that result into unindexed requests, in order to better tune the OpenDJ indexes.

Note that sometimes, it is a good option to leave some requests unindexed (the cost of indexing them outweighs the benefits of the index). If these requests are unfrequent, run by specific administrators for reporting reasons, and if the results are expecting to contain a lot of entries. If so, a best practice is to have a dedicated replica for administration and run these expensive requests. Also, it is better if the client applications are tuned to expect these requests to take a long time.

Data Confidentiality with OpenDJ LDAP Directory Services

FR_plogo_org_FC_openDJ-300x86Directory Servers have been used and continue to be used to store and retrieve identity information, including some data that is sensitive and should be protected. OpenDJ LDAP Directory Services, like many directory servers, has an extensive set of features to protect the data, from securing network connections and communications, authenticating users, to access controls and privileges… However, in the last few years, the way LDAP directory services have been deployed and managed has changed significantly, as they are moving to the “Cloud”. Already many of ForgeRock customers are deploying OpenDJ servers on Amazon or MS Azure, and the requirements for data confidentiality are increasing, especially as the file system and disk management are no longer under their control. For that reason, we’ve recently introduced a new feature in OpenDJ, giving the ability to administrators to encrypt all or part of the directory data before writing to disk.clouddataprotection

The OpenDJ Data Confidentiality feature can be enabled on a per database backend basis to encrypt LDAP entries before being stored to disk. Optionally, indexes can also be protected, individually. An administrator may chose to protect all indexes, or only a few of them, those that contain data that should remain confidential, like cn (common name), sn (surname)… Additionally, the confidentiality of the replication logs can be enabled, and then it’s enabled for all changes of all database backends. Note that if data confidentiality is enabled on an equality index, this index can no longer be used for ordering, and thus for initial substring nor sorted requests.

Example of command to enable data confidentiality for the userRoot backend:

dsconfig set-backend-prop \
 -h -p 4444 \
 -D "cn=Directory Manager" -w secret12 -n -X \
 --backend-name userRoot --set confidentiality-enabled:true

Data confidentiality is a dynamic feature, and can be enabled, disabled without stopping the server. When enabling on a backend, only the updated or created entries will be encrypted. If there is existing data that need confidentiality, it is better to export and reimport the data. With indexes data confidentiality, the behaviour is different. When changing the data confidentiality on an index, you must rebuild the index before it can be used with search requests.

Key Management - Photo adapted from

When enabling data confidentiality, you can select the cipher algorithm and the key length, and again this can be per database backend. The encryption key itself is generated on the server itself and securely distributed to all replicated servers through the replication of the Admin Backend (“cn=admin data”), and thus it’s never exposed to any administrator. Should a key get compromised, we provide a way to mark it so and generate a new key. Also, a backup of an encrypted database backend can be restored on any server with the same configuration, as long as the server still has its configuration and its Admin backend intact. Restoring such backend backup to fresh new server requires that it’s configured for replication first.

The Data Confidentiality feature can be tested with the OpenDJ nightly builds. It is also available to ForgeRock customers as part of our latest update of the ForgeRock Identity Platform.

Migrating from SunDSEE to OpenDJ

Sun DSEE 7.0 DVDAs the legacy Sun product has reached its end of life, many companies are looking at migrating from Sun Directory Server Enterprise Edition [SunDSEE] to ForgeRock Directory Services, built on the OpenDJ project.

Several of our existing customers have already done this migration, whether in house or with the help of partners. Some even did the migration in 2 weeks. In every case, the migration was smooth and easy. Regularly, I’m asked if we have a detailed migration guide and if we can recommend tools to keep the 2 services running side by side, synchronized, until all apps are moved to the ForgeRock Directory Services deployment.

My colleague Wajih, long time directory expert, has just published an article on wikis that described in details how to do DSEE to OpenDJ system to system synchronization using ForgeRock Identity Management product.

If you are planning a migration, check it out. It is that simple !



  • Update on June 8th to add link to A Global Bank case study.