OpenDJ Tips: More on troubleshooting indexes and search performances

In a previous post I talked about analyzing search filters and indexes. Matt added in a comment that OpenDJ has another mean of understanding how indexes are used in a search. Here’s a detailed post.

The OpenDJ LDAP directory server supports a “magic” operational attribute that allows an administrator to get from the server information about the processing of indexes for a specific search query: debugsearchindex.

If the attribute is set in the requested attributes in a search operation, the server will not return all entries as expected, but a single result entry with a fixed distinguished name and a single valued attribute debugsearchindex that contains the information related to the index processing, including the number of candidate entries per filter component, the overall number of candidate, and whether any or all of the search is indexed.

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(mail=user.*)(cn=*Denice*))" debugsearchindex
Password for user 'cn=Directory Manager': *******
dn: cn=debugsearch
debugsearchindex: filter=(&(mail=user.*)[INDEX:mail.substring][COUNT:2000](cn=*Denice*)[INDEX:cn.substring][COUNT:1])[COUNT:1] final=[COUNT:1]

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "objectclass=*" debugsearchindex
Password for user 'cn=Directory Manager': *********
dn: cn=debugsearchdebugsearchindex: filter=(objectClass=*)[NOT-INDEXED] scope=wholeSubtree[COUNT:2007] final=[COUNT:2007]

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "mail=user.1*" debugsearchindex
Password for user 'cn=Directory Manager': *********
dn: cn=debugsearch
debugsearchindex: filter=(mail=user.1*)[INDEX:mail.substring][COUNT:1111] scope=wholeSubtree[COUNT:2007] final=[COUNT:1111]

Note that sometimes, OpenDJ tries to optimize the query and use some other index than the regular one for the query. For example, it might use the equality index for an initial substring filter. The index used during the search does appear in the debugsearchindex attribute. Also, once the result set has been narrowed down to very few entries, it will stop using index and evaluate directly the entry set, as for the example below:

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(cn=Denice*)(mail=user.9*))" debugsearchindex
Password for user 'cn=Directory Manager':
dn: cn=debugsearch
debugsearchindex: filter=(&(cn=Denice*)[INDEX:cn.equality][COUNT:1])[COUNT:1] final=[COUNT:1]

9 thoughts on “OpenDJ Tips: More on troubleshooting indexes and search performances

  1. Manasa 11 April 2012 / 22:05

    hi,

    I am trying to rebuild indexes on one of the servers, and the data got deleted. Is there a forum where we can post questions related to openDJ. below is the command that I used. Has anyone experiened a similar issue. Could you send us the correct way to build indexes in opendj.

    ./dsconfig create-local-db-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –backend-name userRoot –type generic –index-name roleId –set index-type:equality –trustAll –no-prompt

    ./dsconfig create-local-db-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –backend-name userRoot –type generic –index-name resourceId –set index-type:equality –trustAll –no-prompt

    ./rebuild-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –baseDN dc=ACS –index roleId –start 0 –trustAll

    ./rebuild-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –baseDN dc=ACS –index resourceId –start 0 –trustAll

    • Ludo 12 April 2012 / 07:52

      Hi,

      I do not think the data was deleted. There is a known issue with OpenDJ rebuild-index, which is that if the command contains erroneous parameter, the backend is left disabled. You just need to re-enable it to see the data again, and possibly re-run rebuild-index with the proper parameters to have the index working.
      Enabling/disabling a backend is done with dsconfig set-backend-prop …

      The best place to post question related to OpenDJ is the OpenDJ mailing list : https://lists.forgerock.org/mailman/listinfo/opendj.
      Otherwise, there is also a #opendj channel on irc.freenode.net.
      Kind regards,

      Ludo

  2. Ritesh 21 May 2016 / 20:25

    Hi Ludo,

    I am designing a ldap structure, where i have to store 800+ million records.

    my questions is related to index.

    I have a 4 types of records. Each has a common unique id associated with it and i have to use that to fetch all entries associated with that unique id.

    for example

    ou=root,ou=A : contains 50M records with one attribute uid=123
    ou=root,ou=B : contains 200M records with one attribute uid=123
    ou=root,ou=C : contains 200M records with one attribute uid=123
    ou=root,ou=D : contains 400M records with one attribute uid=123

    now i am creating a index on uid, that index would be for 850+ M records.

    Would it be good if I store same uid with different attribute name and create 4 separate index for it like below ?

    ou=root,ou=A : contains 50M records with one attribute uidA=123
    ou=root,ou=B : contains 200M records with one attribute uidB=123
    ou=root,ou=C : contains 200M records with one attribute uidC=123
    ou=root,ou=D : contains 400M records with one attribute uidD=123

    Does 4 separate index would give better search/add/mod/delete performance or one single index would be good ?
    does creating 4 separate index would be extra overhead ?

    which back end would you suggest to go for JE or PDB ?

    • Ludo 22 May 2016 / 20:46

      I would not recommend to multiple the attribute names just to be able to index them.
      But you may want to consider using separate backend-databases for each suffix, which will allow to index the uid attribute for each suffix.
      There are many other parameters to take into consideration: such as the size of the record, the other possible indexes, the future growth, the expected response time and throughput, the availability requirements, backups…
      As for which OpenDJ database backend to use with OpenDJ 3, I would say that JE is a well known backend and it’s been benchmarked with 500 millions records, tested with up to 3B. PDB is a newer backend and we’ve run test and stress with 100M records, but we don’t have yet the experience accumulated with JE in the last 10 years. This said, you should run your own comparative tests, because while both backends have overall similar performances, there can be some differences with specific operations and access.

  3. Hamlet 13 June 2016 / 22:37

    Hi Ludo,

    I have a question about the index on the attribute o.

    From the manage Indexes intereface, i select new index.

    What is the search scope of this index o?

    base object, single Level, subordinate subtree whole?

    Thanks you!

  4. Ludo 13 June 2016 / 22:54

    Indexes are for the whole database backend. They don’t have “scope”. Search requests have a scope.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s