Better index troubleshooting with ForgeRock DS / OpenDJ

Many years ago, I wrote about troubleshooting indexes and search performances, explaining the magicdebugSearchIndex” operational attribute, that allows an administrator to get from the server information about the processing of indexes for a specific search query.

The returned value provides insights on the indexes that were used for a particular search, how they were used and how the resulting set of candidates was built, allowing an administrator to understand whether indexes are used optimally or need to be tailored better for specific search queries and filters, in combination with access logs and other tools such as backendstat.

In DS 6.5, we’ve made some improvements in the search filter processing and we’ve changed the format of the debugSearchIndex value to provide a better reporting of how indexes are used.

The new format is now JSON based, which allow to give it more structure and all could be processed programatically. Here are a few examples of output of the new debugSearchIndex attribute values.

$ bin/ldapsearch -h localhost -p 1389 -D "cn=directory manager" -b "dc=example,dc=com" "(&(cn=*Den*)(mail=user.19*))" debugsearchindex
Password for user 'cn=directory manager': *********

dn: cn=debugsearch
debugsearchindex: {"filter":{"intersection":[{"index":"mail.caseIgnoreIA5SubstringsMatch:6", "exact":"ser.19","candidates":111,"retained":111},{"index":"mail.caseIgnoreIA5SubstringsMatch:6", "exact":"user.1","candidates":1111,"retained":111},
{"filter":"(cn=*Den*)", "index":"cn.caseIgnoreSubstringsMatch:6",
"range":"[den,deo[","candidates":103,"retained":5}], "candidates":5},"final":5}

Let’s look at the debugSearchIndex value and interpret it:

{
"filter": {
"intersection": [
{
"index": "mail.caseIgnoreIA5SubstringsMatch:6",
"exact": "ser.19",
"candidates": 111,
"retained": 111
},
{
"index": "mail.caseIgnoreIA5SubstringsMatch:6",
"exact": "user.1",
"candidates": 1111,
"retained": 111
},
{
"filter": "(cn=*Den*)",
"index": "cn.caseIgnoreSubstringsMatch:6",
"range": "[den,deo[",
"candidates": 103,
"retained": 5
}
],
"candidates": 5
},
"final": 5
}

The filter had 2 components: (cn=*Den*) and (mail=user.19*). Because the whole filter is an AND, the result set is an intersection of several index lookups. Also, both substring filters, but one is a substring of 3 characters and the second one a substring of 7 characters. By default, substring indexes are built with substrings of 6 characters. So the filters are treated differently. The server optimises the processing of indexes so that it will try to first to use the queries that are the most effective. In the case above, the filter (mail=user.19*) is preferred. 2 records are read from the index, and that results in a list of 111 candidates. Then, the server use the remaining filter to narrow the result list. Because the string Den is shorter than the indexed substrings, the server scans a range of keys in the index, starting from the first key match “den” and stopping before the key that matches “deo”. This results in 103 candidates, but only 5 are retained because they were parts of the previous result set. So the result is 5 entries that are matching these filters.

Note the [den,deo[ notation is similar to mathematical Set representation where [ and ] indicate whether a set includes or excludes the boundaries.

Let’s take an example with an OR filter:

$ bin/ldapsearch -h localhost -p 1389 -D "cn=directory manager" -b "dc=example,dc=com" "(|(cn=*Denice*)(uid=user.19))" debugsearchindex
Password for user 'cn=directory manager': *********

dn: cn=debugsearch
debugsearchindex: {"filter":{"union":[{"filter":"(cn=*Denice*)", "index":"cn.caseIgnoreSubstringsMatch:6","exact":"denice","candidates":1}, {"filter":"(uid=user.19)", "index":"uid.caseIgnoreMatch","exact":"user.19","candidates":1}],"candidates":2},"final":2}

As you can see, the result is now a union of 2 exact match (i.e. reads of index keys), each resulting a 1 candidate.

Finally here’s another example, where the scope is used to attempt to reduce the candidate list:

$ bin/ldapsearch -h localhost -p 1389 -D "cn=directory manager" -b "ou=people,dc=example,dc=com" -s one "(mail=user.1)" debugsearchindex
Password for user 'cn=directory manager': *********

dn: cn=debugsearch
debugsearchindex: {"filter":{"filter":"(mail=user.1)","index":"mail.caseIgnoreIA5SubstringsMatch:6", "exact":"user.1","candidates":1111},"scope":{"type":"one","candidates":"[NOT-INDEXED]","retained":1111},"final":1111}

You can find more information and details about the debugsearchindex attribute in the ForgeRock Directory Services 6.5 Administration Guide.

Happy New Year 2019!

Wow, 2018 is already over, and a new year has started. It’s a usual time to look back at what we’ve achieved, make a clean slate (zero inbox), take a deep breath and jump ahead to the new projects, goals or adventures, both personal and professional. Including posting more on this blog ! 🙂

I’m wishing every day of the new year to be filled with love, success, happiness and prosperity for you.
Happy New Year!
Bonne Année !
Godt Nytt í…r !
¡ Feliz año nuevo !
あけましておめでとうございます。

ForgeRock Directory Services 6.5 is Available

The ForgeRock Identity Platform was released and publicly announced early December this year (also here).

As you may guess from the announcement, an important part of the new features has to do with DevOps, running in Docker, automated with Kubernetes.

The underlying datastore for the ForgeRock Identity Platform is ForgeRock Directory Services, and the new 6.5 release comes with a set of new features and improvements, that are detailed in the Release Notes, but here’s some highlights:

Ease of use has always been important for us, and DS 6.5 brings it to a new level for the customers that are deploying other ForgeRock products. Starting with this version, you can now select, at the time of installation, one or more profiles. A profile contains the complete configuration for a specific use, from base DN, backend, indexes, schema, specific configuration parameters, administrative users, ACI and privileges.. Out of the box, we are delivering 3 profiles for ForgeRock Access Management: Identity Store, Configuration Store and the Core Token Service Store; 1 profile for ForgeRock Identity Management: Managed Object Store; and 1 profile for Directory Services evaluation, that contains the data and configuration that is used through our documentation, and allows you to copy and paste the command examples of the guides and replay them against a running server.

To learn more about profiles, get DS 6.5, and run

setup –help-profiles

. To learn about a specific profile, you can run

setup –help-profile am-cts:6.5.0

With regards to DevOps, containers and automation in the cloud, we’ve continued the efforts that we had started with previous releases.

  • DS 6.5 now supports a method to run post upgrade tasks to the data, such as rebuilding indexes.
  • The server has 2 new HTTP endpoints to poke about its status. /isReady indicates that the server is up and running. /isHealty indicates if its current state is optimal, or if there are some temporary limitations, such as a database backend is offline for maintenance, or the replication is lagging too much (with too much being fully configurable).
  • The Grafana sample dashboard has been updated
  • Like all ForgeRock Identity Platform’s products, DS comes with a Common Audit handler that published log messages to stdout, a common practice when working with Docker containers.

Directory Proxy Server 6.5 now supports “sharding”, i.e. distributing data into multiple discrete replicated directory services. Such deployments make very large amount of data easier to manage and give better write scalability. In this version, the number of “shards” is fixed, but we are working on making the service dynamically scaling as the data grows, in future versions.

Directory Services 6.5 now supports limiting the number of connections that can be opened from a single client application. By IP address, a client may be denied, fully allowed or restricted in its number of opened connections, offering a greater protection against misbehaving applications.

The product also now supports the LDAP Relax Rules Control, that allow an administrator to add or modify attributes that are normally read-only. This feature can be used when having to synchronise data between different LDAP products, so they have the same timestamps for their creation or modification dates.

We’ve made the “cn=Changelog” suffix and data available on servers that are only acting as Replication hubs (RS), since they are persisting all the changes to replicate them.

We’ve added a couple of troubleshooting tools with the release. One tool, changelogstat) allows to list and dump the content of the replication changelog databases. The supportextract tool allows an administrator to capture the state and logs of a Directory Services instance and make the file available to ForgeRock support quickly.

Java 11 is now fully supported, both Oracle JVM and OpenJDK builds (from Oracle, Red-Hat or Azul Systems).

Finally, like with all releases of Directory Services, we have enhanced the performance and the reliability of the server in many areas. But most importantly, we have fully tested that you can upgrade to 6.5 without any service interruption: from 2.6 to 6.0, you can upgrade an instance and let it replicate with the other instances, then start upgrading the next one, until all instances are on the latest and greatest version. If you use VMs or containers, you can stop an existing instance and replace it with a new one. Or add a new one and then stop an old one… Your choice, but both scenarios are supported.

For further details, read the complete Release Notes. I’m looking forward to your feedback on the features and improvements of the Directory Services 6.5 release!

Après Londres, Identity Live arrive à Paris

Le ForgeRock Identity Live de Londres vient tout juste de se terminer, et déjà je suis impatient du prochain, le dernier pour l’année 2018: Identity Live Paris.

parissummitsocial_01

Venez nous retrouver, rencontrer des clients, des leaders d’opinions, des experts technique et autres professionnels de l’identité numérique. Pour la première fois, cette année, vous aurez aussi la possibilité, le 14 Novembre, de rencontrer et de discuter avec les experts techniques des produits, les développeurs, sous un format “UnConference” : agenda mouvant, discussions interactives sur les nouvelles fonctionnalités, sur les bonnes pratiques avec les containeurs Docker et Kubernetes…

Il est encore temps de s’inscrire !

En espérant vous retrouver à Paris les 13 et 14 Novembre…

[Mise à jour post-évenement]:
Vous pouvez trouver les quelques photos que j’ai faites ici.

Untitled

Identity Live London is over, Paris is next…

It’s been a couple of intense days in London with over 200 attendees at the London stop of the ForgeRock Identity Live world tour.

Untitled

In London, we’ve had 3 important customers that explained how they are innovating with the help of digital identities, each of them providing online services to over 30 millions users: The BBC, Maerks and Pearson. And we’ve had 3 major UK banks that joined a panel to discuss OpenBanking and APIs in the banking industry. I have particularly enjoyed the well mastered presentations by Bianca Lopes about the data that we leave online and that ties back to our identity, and by Spencer Kelly, technology presenter of the BBC show “Click”.

UntitledToday, we had our “unConference” day, where the engineering team is joining the product management one and discuss with our customers and partners on how to leverage the newest features of the ForgeRock Identity Platform, whether already released or soon to be.

My photos of the Identity Live London are now publicly visible here: https://www.flickr.com/photos/ludovicpoitou/albums/72157701508676261

And now, on to the next and last stop for 2018: Paris, November 13 and 14. Register and join us!

parissummitsocial_01

[Post Event Update]

You can find the few photos that I’ve taken on the Flickr album.

Untitled

ForgeRock IdentityLive APAC

Last month, ForgeRock hosted two IdentityLive events in the Asia-Pacific region.

One in Sydney on August 7 and 8

Sydney Australia

And the second one in Singapore the week after.

Singapore

This was my first participation to the events in this region (somehow I managed to convince my family to move our vacation earlier so I could attend), and it was great meeting in person with many customers and prospects I’ve interacted with over the phone, as well as meeting the ForgeRock colleagues I hadn’t seen for a while. As usual, the conversations around our products and the customers solutions were very rich and open, and I came back with great inputs and confirmations for our roadmaps.

You can find my photos of the events in the following Flickr albums:

The next IdentityLive events will take place in London on October 30-31, and Paris on November 13-14. I hope to see you there!

ForgeRock Identity Live Berlin

The second show of the ForgeRock worldwide tour of Identity Live events took place last week in the beautiful city of Berlin.LP0_4079

My colleagues from the Marketing team have already put a summary of the event with an highlight video and links to slides, videos of the sessions.

And my photo album of the event is also visible online here:

ForgeRock Identity Live Berlin 2018

See you at the next Identity Live in Sydney or in Singapore in August.