In a previous post I talked about analyzing search filters and indexes. Matt added in a comment that OpenDJ has another mean of understanding how indexes are used in a search. Here’s a detailed post.
The OpenDJ LDAP directory server supports a “magic” operational attribute that allows an administrator to get from the server information about the processing of indexes for a specific search query: debugsearchindex.
If the attribute is set in the requested attributes in a search operation, the server will not return all entries as expected, but a single result entry with a fixed distinguished name and a single valued attribute debugsearchindex that contains the information related to the index processing, including the number of candidate entries per filter component, the overall number of candidate, and whether any or all of the search is indexed.
$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(mail=user.*)(cn=*Denice*))" debugsearchindex Password for user 'cn=Directory Manager': ******* dn: cn=debugsearch debugsearchindex: filter=(&(mail=user.*)[INDEX:mail.substring][COUNT:2000](cn=*Denice*)[INDEX:cn.substring][COUNT:1])[COUNT:1] final=[COUNT:1] $ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "objectclass=*" debugsearchindex Password for user 'cn=Directory Manager': ********* dn: cn=debugsearchdebugsearchindex: filter=(objectClass=*)[NOT-INDEXED] scope=wholeSubtree[COUNT:2007] final=[COUNT:2007] $ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "mail=user.1*" debugsearchindex Password for user 'cn=Directory Manager': ********* dn: cn=debugsearch debugsearchindex: filter=(mail=user.1*)[INDEX:mail.substring][COUNT:1111] scope=wholeSubtree[COUNT:2007] final=[COUNT:1111]
Note that sometimes, OpenDJ tries to optimize the query and use some other index than the regular one for the query. For example, it might use the equality index for an initial substring filter. The index used during the search does appear in the debugsearchindex attribute. Also, once the result set has been narrowed down to very few entries, it will stop using index and evaluate directly the entry set, as for the example below:
$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(cn=Denice*)(mail=user.9*))" debugsearchindex Password for user 'cn=Directory Manager': dn: cn=debugsearch debugsearchindex: filter=(&(cn=Denice*)[INDEX:cn.equality][COUNT:1])[COUNT:1] final=[COUNT:1]
Ludo Sketches 
hi,
I am trying to rebuild indexes on one of the servers, and the data got deleted. Is there a forum where we can post questions related to openDJ. below is the command that I used. Has anyone experiened a similar issue. Could you send us the correct way to build indexes in opendj.
./dsconfig create-local-db-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –backend-name userRoot –type generic –index-name roleId –set index-type:equality –trustAll –no-prompt
./dsconfig create-local-db-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –backend-name userRoot –type generic –index-name resourceId –set index-type:equality –trustAll –no-prompt
./rebuild-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –baseDN dc=ACS –index roleId –start 0 –trustAll
./rebuild-index –port 4444 –hostname –bindDN “cn=Directory Manager” –bindPassword secret12 –baseDN dc=ACS –index resourceId –start 0 –trustAll
Hi,
I do not think the data was deleted. There is a known issue with OpenDJ rebuild-index, which is that if the command contains erroneous parameter, the backend is left disabled. You just need to re-enable it to see the data again, and possibly re-run rebuild-index with the proper parameters to have the index working.
Enabling/disabling a backend is done with dsconfig set-backend-prop …
The best place to post question related to OpenDJ is the OpenDJ mailing list : https://lists.forgerock.org/mailman/listinfo/opendj.
Otherwise, there is also a #opendj channel on irc.freenode.net.
Kind regards,
Ludo
Hi Ludo,
I am designing a ldap structure, where i have to store 800+ million records.
my questions is related to index.
I have a 4 types of records. Each has a common unique id associated with it and i have to use that to fetch all entries associated with that unique id.
for example
ou=root,ou=A : contains 50M records with one attribute uid=123
ou=root,ou=B : contains 200M records with one attribute uid=123
ou=root,ou=C : contains 200M records with one attribute uid=123
ou=root,ou=D : contains 400M records with one attribute uid=123
now i am creating a index on uid, that index would be for 850+ M records.
Would it be good if I store same uid with different attribute name and create 4 separate index for it like below ?
ou=root,ou=A : contains 50M records with one attribute uidA=123
ou=root,ou=B : contains 200M records with one attribute uidB=123
ou=root,ou=C : contains 200M records with one attribute uidC=123
ou=root,ou=D : contains 400M records with one attribute uidD=123
Does 4 separate index would give better search/add/mod/delete performance or one single index would be good ?
does creating 4 separate index would be extra overhead ?
which back end would you suggest to go for JE or PDB ?
I would not recommend to multiple the attribute names just to be able to index them.
But you may want to consider using separate backend-databases for each suffix, which will allow to index the uid attribute for each suffix.
There are many other parameters to take into consideration: such as the size of the record, the other possible indexes, the future growth, the expected response time and throughput, the availability requirements, backups…
As for which OpenDJ database backend to use with OpenDJ 3, I would say that JE is a well known backend and it’s been benchmarked with 500 millions records, tested with up to 3B. PDB is a newer backend and we’ve run test and stress with 100M records, but we don’t have yet the experience accumulated with JE in the last 10 years. This said, you should run your own comparative tests, because while both backends have overall similar performances, there can be some differences with specific operations and access.
Hi Ludo,
I have a question about the index on the attribute o.
From the manage Indexes intereface, i select new index.
What is the search scope of this index o?
base object, single Level, subordinate subtree whole?
Thanks you!
Indexes are for the whole database backend. They don’t have “scope”. Search requests have a scope.