ForgeRock Directory Services 6.5 is Available

The ForgeRock Identity Platform was released and publicly announced early December this year (also here).

As you may guess from the announcement, an important part of the new features has to do with DevOps, running in Docker, automated with Kubernetes.

The underlying datastore for the ForgeRock Identity Platform is ForgeRock Directory Services, and the new 6.5 release comes with a set of new features and improvements, that are detailed in the Release Notes, but here’s some highlights:

Ease of use has always been important for us, and DS 6.5 brings it to a new level for the customers that are deploying other ForgeRock products. Starting with this version, you can now select, at the time of installation, one or more profiles. A profile contains the complete configuration for a specific use, from base DN, backend, indexes, schema, specific configuration parameters, administrative users, ACI and privileges.. Out of the box, we are delivering 3 profiles for ForgeRock Access Management: Identity Store, Configuration Store and the Core Token Service Store; 1 profile for ForgeRock Identity Management: Managed Object Store; and 1 profile for Directory Services evaluation, that contains the data and configuration that is used through our documentation, and allows you to copy and paste the command examples of the guides and replay them against a running server.

To learn more about profiles, get DS 6.5, and run

setup –help-profiles

. To learn about a specific profile, you can run

setup –help-profile am-cts:6.5.0

With regards to DevOps, containers and automation in the cloud, we’ve continued the efforts that we had started with previous releases.

  • DS 6.5 now supports a method to run post upgrade tasks to the data, such as rebuilding indexes.
  • The server has 2 new HTTP endpoints to poke about its status. /isReady indicates that the server is up and running. /isHealty indicates if its current state is optimal, or if there are some temporary limitations, such as a database backend is offline for maintenance, or the replication is lagging too much (with too much being fully configurable).
  • The Grafana sample dashboard has been updated
  • Like all ForgeRock Identity Platform’s products, DS comes with a Common Audit handler that published log messages to stdout, a common practice when working with Docker containers.

Directory Proxy Server 6.5 now supports “sharding”, i.e. distributing data into multiple discrete replicated directory services. Such deployments make very large amount of data easier to manage and give better write scalability. In this version, the number of “shards” is fixed, but we are working on making the service dynamically scaling as the data grows, in future versions.

Directory Services 6.5 now supports limiting the number of connections that can be opened from a single client application. By IP address, a client may be denied, fully allowed or restricted in its number of opened connections, offering a greater protection against misbehaving applications.

The product also now supports the LDAP Relax Rules Control, that allow an administrator to add or modify attributes that are normally read-only. This feature can be used when having to synchronise data between different LDAP products, so they have the same timestamps for their creation or modification dates.

We’ve made the “cn=Changelog” suffix and data available on servers that are only acting as Replication hubs (RS), since they are persisting all the changes to replicate them.

We’ve added a couple of troubleshooting tools with the release. One tool, changelogstat) allows to list and dump the content of the replication changelog databases. The supportextract tool allows an administrator to capture the state and logs of a Directory Services instance and make the file available to ForgeRock support quickly.

Java 11 is now fully supported, both Oracle JVM and OpenJDK builds (from Oracle, Red-Hat or Azul Systems).

Finally, like with all releases of Directory Services, we have enhanced the performance and the reliability of the server in many areas. But most importantly, we have fully tested that you can upgrade to 6.5 without any service interruption: from 2.6 to 6.0, you can upgrade an instance and let it replicate with the other instances, then start upgrading the next one, until all instances are on the latest and greatest version. If you use VMs or containers, you can stop an existing instance and replace it with a new one. Or add a new one and then stop an old one… Your choice, but both scenarios are supported.

For further details, read the complete Release Notes. I’m looking forward to your feedback on the features and improvements of the Directory Services 6.5 release!

DDOS Attacks leveraging LDAP !

21382575392_223304551e_z
photo by Christiaan Colen

Yesterday, DDoS mitigation provider Corero Network Security disclosed a zero-day distributed denial of service attack (DDoS) technique, observed in the wild, that is capable of amplifying malicious traffic by a factor of as much as 55x. Several sites published the story as “Attackers are now abusing exposed LDAP servers to amplify DDoS attacks”.

 

According to Corero, the attacks exploited the Lightweight Directory Access Protocol (LDAP), but reading the details of the press release, it appears that the attackers were using Connectionless LDAP services (CLDAP) .

In this case, the attacker sends a simple query to a vulnerable reflector supporting the Connectionless LDAP service (CLDAP) and using address spoofing makes it appear to originate from the intended victim. The CLDAP service responds to the spoofed address, sending unwanted network traffic to the attacker’s intended target.

Connectionless LDAP  is a very old technical specification, published in 1995 as RFC 1798.  In 2003, this specification was obsoleted by RFC 3352 and moved to historical status. One of the main reason for obsoleting the proposed standard was its insufficient security capabilities.

OpenDJ, the open source LDAP Directory Services in Java, has never supported CLDAP and thus cannot be used in such attack. So, if you are a  ForgeRock customer, you should not worry about this kind of attack. But if you’re running a legacy product, that has CLDAP enabled by default, it is probably time to think about moving to a more recent and up to date directory service, such as OpenDJ.

 

Managing OpenDJ with REST

OpenDJ, the open source LDAP Directory Server, was the first to propose a native HTTP REST / JSON access to the data.

In the next major release, OpenDJ will be providing many enhancements to the REST interface, that I will describe in a series of posts. To start with, let’s talk about the new administrative interfaces added to manage the OpenDJ server.

When the HTTP access is enabled, OpenDJ creates by default 2 administrative endpoints: /admin/config and /admin/monitor.

/admin/config provides a read-write access to the configuration, with the same view and hierarchy of objects as the LDAP access. All of the operations that are possible with the dsconfig command, can be done over LDAP, and now REST.  As a matter of fact, the /admin/config API is automatically generated from the same XML description files that are used to generate the LDAP view and the dsconfig command line utilities. This means that any extension, plugin added to the server will also be exposed via REST without additional code.

screen-shot-2016-10-25-at-15-03-54

Above is an example of query of the /admin/config endpoint, querying for all  backends , done as a user who has the privilege to read the configuration. A similar query done with a user that doesn’t have the config-read privilege does fail as below:

$ curl -s -u user.2 http://localhost:8080/admin/config/backends/userRoot
Enter host password for user 'user.2': 
{
 "message" : "Insufficient Access Rights: You do not
have sufficient privileges to perform search operations
in the Directory Server configuration",
 "code" : 403,
 "reason" : "Forbidden"
}

/admin/monitor provides a read-only view on all of the OpenDJ monitoring information that was already accessible via LDAP under the "cn=Monitor" naming context, and JMX.

$ curl -s -u user.0 http://localhost:8080/admin/monitor/
Enter host password for user 'user.0':
{
 "_id" : "monitor",
 "upTime" : "0 days 2 hours 49 minutes 54 seconds",
 "currentConnections" : "1",
 "totalConnections" : "32",
 "currentTime" : "20161024103215Z",
 "startTime" : "20161024074220Z",
 "productName" : "OpenDJ Server",
 "_rev" : "00000000644a67b2",
 "maxConnections" : "3"
}

The /admin REST endpoints can be protected with different authorization mechanisms, from HTTP basic to OAuth2. And the whole endpoint can be disabled as well if needed using dsconfig.

These administrative REST endpoints can be tested with the OpenDJ nightly builds. They are also available to ForgeRock customers as part of our latest update of the ForgeRock Identity Platform.

Migrating from SunDSEE to OpenDJ

Sun DSEE 7.0 DVDAs the legacy Sun product has reached its end of life, many companies are looking at migrating from Sun Directory Server Enterprise Edition [SunDSEE] to ForgeRock Directory Services, built on the OpenDJ project.

Several of our existing customers have already done this migration, whether in house or with the help of partners. Some even did the migration in 2 weeks. In every case, the migration was smooth and easy. Regularly, I’m asked if we have a detailed migration guide and if we can recommend tools to keep the 2 services running side by side, synchronized, until all apps are moved to the ForgeRock Directory Services deployment.

My colleague Wajih, long time directory expert, has just published an article on ForgeRock.org wikis that described in details how to do DSEE to OpenDJ system to system synchronization using ForgeRock Identity Management product.

If you are planning a migration, check it out. It is that simple !

 

Updates:

  • Update on June 8th to add link to A Global Bank case study.

What’s new in OpenDJ 3.0, Part III

FR_plogo_org_FC_openDJ-300x86In the previous posts, I talked about the new PDB Backend in OpenDJ 3.0, and the other changes with backends, replication and the changelog.

In this last article about OpenDJ 3.0, I’m presenting the most important new features and enhancements in this major release:

Certificate Matching Rules.

OpenDJ now implements the CertificateExactMatch matching rule in compliance with “Lightweight Directory Access Protocol (LDAP) Schema Definitions for X.509 Certificates” (RFC 4523) and implements the schema and the syntax for certificates, certificate lists  and certificate pairs.

It’s now possible to search a directory to find an entry with a specific certificate, using a filter such as below:

(userCertificate={ serialNumber 13233831500277100508, issuer rdnSequence:"CN=Babs Jensen,OU=Product Development,L=Cupertino,C=US" })

Password Storage Schemes

The PKCS5S2 Password Storage Scheme has been added to the list of supported storage schemes. While this one is less secure and flexible than PBKDF2, it allows some of our customers to migrate from systems that use the PKCS5S2 algorithm. Other password storage schemes have been enhanced to support arbitrary salt length and thus helping with other migrations (without requiring all users to have a new password).

Disk Space Monitoring.

In previous releases, each backend had a disk space monitoring function, regardless of the filesystems or disks used. In OpenDJ 3.0, we’ve created a disk space monitoring service, and backends, replication, log services register to it. This allows the server to optimise its resource consumption to monitor, as well as ensuring that all disks that contain writable data are monitored, and alerts raised when reaching some low threshold.

Improvements

There are many improvements in many areas of the server: in the REST to LDAP services and gateway, optimisations on indexes, dsconfig batch mode, DSML Gateway supporting SOAP 1.2, native packages… For the complete details, please read the Release Notes.

As always, the best way to really see and feel the difference is by downloading and installing the OpenDJ server, and playing with it. We’re providing a Zip installation, an RPM and a Debian Package, the DSML Gateway and the REST to LDAP Gateway as war files.

Over the course of the development of OpenDJ 3.0, we’ve received many contributions, in form of code, issues raised in our JIRA, documentation… We address our deepest thanks to all the contributors and developers :

Andrea Stani, Auke Schrijnen, Ayami Tyndal, Brad Tumy, Bruno Lavit, Bernhard Thalmayr, Carole Forel, Chris Clifton, Chris Drake, Chris Ridd, Christian Ohr, Christophe Sovant, Cyril Grosjean, Darin Perusich, David Goldsmith, Dennis Demarco, Edan Idzerda, Emidio Stani, Fabio Pistolesi, Gaétan Boismal, Gary Williams, Gene Hirayama, Hakon Steinø, Ian Packer, Jaak Pruulmann-Vengerfeldt, James Phillpotts, Jeff Blaine, Jean-Noël Rouvignac, Jens Elkner, Jonathan Thomas, Kevin Fahy, Lana Frost, Lee Trujillo, Li Run, Ludovic Poitou, Manuel Gaupp, Mark Craig, Mark De Reeper, Markus Schulz, Matthew Swift, Matt Miller, Muzzol Oliba, Nicolas Capponi, Nicolas Labrot, Ondrej Fuchsik, Patrick Diligent, Peter Major, Quentin Cassel, Richard Kolb, Robert Wapshott, Sébastien Bertholet, Shariq Faruqi, Stein Myrseth, Sunil Raju, Tomasz Jędrzejewski, Travis Papp, Tsoi Hong, Violette Roche-Montané, Wajih Ahmed, Warren Strange, Yannick Lecaillez. (I’m sorry if I missed anyone…)

What’s new in OpenDJ 3.0 – Part II

FR_plogo_org_FC_openDJ-300x86Yesterday, I’ve talked about the most important change in OpenDJ 3.0, that is the new PDB Backend. Let me detail other new and improved features of OpenDJ 3.0, still related to backends and replication.

As part of the work for the new backend, we’ve worked on the import process, in order to make it more I/O efficient and thus faster.

Here’s some numbers, importing 1 000 000 users in OpenDJ.

In OpenDJ 2.6.3:

$ import-ldif -l ../1M.ldif -n userRoot
[03/Feb/2016:15:41:42 +0100] category=RUNTIME_INFORMATION severity=NOTICE msgID=20381717 msg=Installation Directory: /Space/Tests/Blog/2.6/opendj
...
[03/Feb/2016:15:42:54 +0100] category=JEB severity=NOTICE msgID=8847454 msg=Processed 1000002 entries, imported 1000002, skipped 0, rejected 0 and migrated 0 in 71 seconds (average rate 13952.5/sec)

In OpenDJ 3.0, with the JE Backend:

$ import-ldif -l ../../1M.ldif -n userRoot
[03/02/2016:15:45:19 +0100] category=UTIL seq=0 severity=INFO msg=Installation Directory: /Space/Tests/Blog/3.0/opendj
...
[03/02/2016:15:46:22 +0100] category=PLUGGABLE seq=74 severity=INFO msg=Processed 1000002 entries, imported 1000002, skipped 0, rejected 0 and migrated 0 in 62 seconds (average rate 15961.2/sec)

In OpenDJ 3.0, with the PDB Backend

$ import-ldif -l ../../1M.ldif -n userRoot
[03/02/2016:15:59:38 +0100] category=UTIL seq=0 severity=INFO msg=Installation Directory: /Space/Tests/Blog/3.0/opendj
...
[03/02/2016:16:00:38 +0100] category=PLUGGABLE seq=48 severity=INFO msg=Processed 1000002 entries, imported 1000002, skipped 0, rejected 0 and migrated 0 in 58 seconds (average rate 17038.7/sec)

We’ve also completely reworked the storage layer for the replication changes, moving away from the BDB JE database. Instead, we’re using direct files, again providing much smaller disk occupancy (and thus optimising I/Os) but also allowing much more efficient purging of old data.

As part of these changes, we’ve made serious improvements to the way the replication changes can be read and searched using LDAP under the “cn=Changelog” suffix. More importantly, we’ve now have a way to ensure a complete ordering of the changes published, and thus consistency of their “changeNumbers”. That is to say that now, when reading “cn=Changelog” on different replicated servers, the change with “ChangeNumber=N” will be the same on all servers, allowing applications that read these changes to failover from one server to another. We’ve added a way to resynchronise these ChangeNumbers when adding a new replica to an existing topology, or when restoring one after a maintenance period.

Still on the subject of the ChangeLog, we’ve added another level of security to it, by introducing a “changelog-read” privilege that provides a better control on which applications and users are allowed to read the data from the “cn=Changelog” suffix.

That’s all for today. Tomorrow, I will continue with all the other new features and enhancements in OpenDJ 3.0.

If you have done it yet, you can download OpenDJ 3.0 from ForgeRock’s BackStage and start playing with it. And check the Release Notes for more information.

OpenDJ 3.0.0 has been released…

FR_plogo_org_FC_openDJ-300x86As part of the release of the ForgeRock Identity Platform that we did last week, we’ve released a major version of our Directory Services product : OpenDJ 3.0.0.

The main and most important change in OpenDJ 3.0 is the work on the backend layer, with the introduction of a new backend database, supported by a new low level key-value store. When installing a new instance of OpenDJ, administrators now have the choice of creating a JE Backend (which is based on Berkeley DB Java Edition, as with previous releases of OpenDJ), or a PDB Backend (which is based on the new PersistIt library). When upgrading, the existing local backends will be transparently upgraded in JE Backends, but indexes will need to be rebuilt (and can be rebuilt automatically during the upgrade process).

Both backends have the same capabilities, and very similar performances. Most importantly, both backends benefit from a number of improvements compared with previous releases : the size of databases and index records are smaller, some indexes have been reworked to deliver better performances both for updates and reads. Overall, we’ve been increasing the throughput of Adding/Deleting entries in OpenDJ by more than 15 %.

But the 2 backends are different, especially in the way they deal with database compression. Because of the way it’s dealing with journals and compression, the new PDB backend may deliver better overall throughput, but may increase its disk occupancy significantly under heavy load (it favours updates over compression). Once the throughput is reduced under a certain threshold, compression will be highly effective and the overall disk occupancy will be optimised.

A question I often get is “Which backend should I use? “. And I don’t have a definitive answer. If you have an OpenDJ instance and you’re upgrading to 3.0, keep the JE Backend. This is a simple and automated upgrade. If you’re installing a new instance of OpenDJ, then I would say it’s a matter of risks. We don’t have the same wide experience with the PDB backend than we have had with the JE backend over the last 10 years. So, if you want to be really safe, chose the JE Backend. If you have time to test, stage your directory service before putting it in production, you might want to go with the PDB Backend. As, moving forward, we will focus our performance testing and improvements on the PDB backend essentially.

That’s all for now. In a followup post, I will continue to review the changes in OpenDJ 3.0…

Meanwhile, you can download OpenDJ 3.0 from ForgeRock’s BackStage and start playing with it. And check the Release Notes for more information.

PS: The followup posts have been published: