Tips: resource limits in OpenDJ

Photo by Scallop Holden http://www.flickr.com/photos/scallop_holden/
Photo by Scallop Holden http://www.flickr.com/photos/scallop_holden/

OpenDJ, the open source LDAP directory services in Java, defines a few global resource limits to prevent client connections or operations from abusing the server’s resources. These limits are

  • the maximum number of entries returned to a search request (size-limit, default is 1000),
  • the maximum amount of time to spend returning results to a client (time-limit, default is 60 seconds),
  • the maximum number of entries to look through while processing a search request (lookthrough-limit, default is 5000),
  • the maximum amount of time a connection can sit idle before the server disconnect it (idle-time-limit, default is unlimited).

There are default values for all of these limits in the Global configuration, but they can also be set on a per user basis. The global limits are read or set using dsconfig :

$ bin/dsconfig get-global-configuration-prop -p 4444 -X -n -h localhost \
 -D cn=directory\ manager -w secret12
Property : Value(s)
--------------------------------------:------------------------
bind-with-dn-requires-password : true
default-password-policy : Default Password Policy
disabled-privilege : -
entry-cache-preload : false
etime-resolution : milliseconds
idle-time-limit : 0
lookthrough-limit : 5000
max-allowed-client-connections : 0
max-psearches : unlimited
proxied-authorization-identity-mapper : Exact Match
reject-unauthenticated-requests : false
return-bind-error-messages : false
save-config-on-successful-startup : true
size-limit : 1000
smtp-server : -
time-limit : 60 s
writability-mode : enabled

The per user limits have a different LDAP attribute name and can be found or set directly in users’ entry, or through Collective Attributes. The Directory Manager entry has such specific limits set, so that everything is unlimited.

$ bin/ldapsearch -D "cn=directory manager" -w secret12 -p 1389 -X -b "cn=config" \
  '(objectClass=inetOrgPerson)' ds-rlim-time-limit ds-rlim-size-limit \
  ds-rlim-lookthrough-limit ds-rlim-idle-time-limit
dn: cn=Directory Manager,cn=Root DNs,cn=config
ds-rlim-lookthrough-limit: 0
ds-rlim-time-limit: 0
ds-rlim-idle-time-limit: 0
ds-rlim-size-limit: 0

If you decide to change the default global settings, for example the idle-time-limit, to force idle connections to be closed by the server after some time (often a smaller time than the settings of the load-balancer in between your applications and the OpenDJ servers), please remember that you might also want to change the limit for “cn=Directory Manager”, especially if your client applications are connecting with Directory Manager credentials.

Cache strategy for OpenDJ LDAP directory server

System administrators that are familiar with legacy LDAP directory servers know that one of the key for the best performance is caching the data. With Sun Directory Server or OpenLDAP, there are 3 levels of caching that could be done : the filesystem level, the database level and the entries level. The filesystem level cache is managed by the OS and cannot be controlled by the application. Using the filesystem cache is good when the directory server is the only process on the machine, and/or for initial performance. The database level cache allows faster read or write operations, and also includes the indexes. The later cache is the higher level cache, and usually the one that provides the best performances as it requires the least processing from the server to return entries to the applications, and it has the least contention.

OpenDJ has a different design for its database and core server, and thus the caching strategy needs to be different.

By default, OpenDJ does have a database cache enabled, and 3 different kind of entry caches, all disabled. The reason for the 3 entry caches is that they are implementing for different needs and access patterns. But all have in common a specific filter to select which entries to cache, and some settings as to how much memory to use. During our stress and performance tests, we noticed that using an entry cache for all accessed entries added a lot of pressure on the garbage collector, and also caused more garbage collection from the old generation, often leading to either fragmentation of the memory, or more frequent full GC (also known as “Stop the world GC”). This resulted in an overall lower consistent average response time and throughput.

So, we recommend that you favor the database cache, and do not setup an entry cache, except for specific needs (and do not try to activate all 3 entry caches, this may lead to some really strange behavior).

The default settings with OpenDJ 2.4 is that 10 % of the JVM heap space will be used for the database cache. With OpenDJ 2.5 (soon to be released), we have bumped the default to 50% of the heap space. If you’re tuning the heap size and make it larger than 2GB, we recommend that you keep that 50% ratio or even increase it if the heap size exceeds the 3GB.

If you do have a few very specific entries that are very often accessed, like large static groups that are constantly used for ACI or group membership by application, then the entry cache becomes handy, and then you want to set a filter so only these specific entries are cached.

For example, if you want to cache at most 5 entries, that are groupOfNames, you can use the following dsconfig command:

bin/dsconfig set-entry-cache-prop --cache-name FIFO
 --set include-filter:\(objectclass=GroupOfNames\)
 --set max-entries:5 --set max-memory-percent:90 --set enabled:true
 -h localhost -p 4444 -D "cn=Directory Manager" -w secret12 -X -n

Otherwise, you’d better of running with no entry cache. OpenDJ read performance are such that the directory server can respond to tens of thousands if not hundred of thousands searches per second with average response time in the order of a milli-second. This should be good enough for most applications !

OpenDJ: Extensible indexes for Internationalization.

While taming the subject of indexes, we recently had some discussion with one of our users who complained about long response times with some language specific search filter such as (cn:fr.6:=*John*).

These extended filters rely on I18N Collation matching rules and indexes that I’ve described in an old post for OpenDS.

It turned out that he had defined the matching rule for the index, and rebuilt it, but had missed an important part: the index-type did not include “extensible”.

The proper command to create an extensible index is the following :

dsconfig set-local-db-index-prop --backend-name userRoot --index-name cn \
 --set index-extensible-matching-rule:fr.6 \
 --add index-type:extensible \
 --hostname localhost --port 4444 \
 --bindDN cn=Directory\ Manager --bindPassword ****** \
 -X -n

fr.6 is the shortcut for the French substring collation matching rule which full OID is 1.3.6.1.4.1.42.2.27.9.4.76.1

Note that if you don’t specify the extensible index-type, the server will not build the index for the extensible matching rule. The use of the index-type is consistent with the other types of index, equality or else, and allows you to disable and re-enable extensible indexes without having to re-enter all  OIDs.

OpenDJ Tips: More on troubleshooting indexes and search performances

In a previous post I talked about analyzing search filters and indexes. Matt added in a comment that OpenDJ has another mean of understanding how indexes are used in a search. Here’s a detailed post.

The OpenDJ LDAP directory server supports a “magic” operational attribute that allows an administrator to get from the server information about the processing of indexes for a specific search query: debugsearchindex.

If the attribute is set in the requested attributes in a search operation, the server will not return all entries as expected, but a single result entry with a fixed distinguished name and a single valued attribute debugsearchindex that contains the information related to the index processing, including the number of candidate entries per filter component, the overall number of candidate, and whether any or all of the search is indexed.

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(mail=user.*)(cn=*Denice*))" debugsearchindex
Password for user 'cn=Directory Manager': *******
dn: cn=debugsearch
debugsearchindex: filter=(&(mail=user.*)[INDEX:mail.substring][COUNT:2000](cn=*Denice*)[INDEX:cn.substring][COUNT:1])[COUNT:1] final=[COUNT:1]

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "objectclass=*" debugsearchindex
Password for user 'cn=Directory Manager': *********
dn: cn=debugsearchdebugsearchindex: filter=(objectClass=*)[NOT-INDEXED] scope=wholeSubtree[COUNT:2007] final=[COUNT:2007]

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "mail=user.1*" debugsearchindex
Password for user 'cn=Directory Manager': *********
dn: cn=debugsearch
debugsearchindex: filter=(mail=user.1*)[INDEX:mail.substring][COUNT:1111] scope=wholeSubtree[COUNT:2007] final=[COUNT:1111]

Note that sometimes, OpenDJ tries to optimize the query and use some other index than the regular one for the query. For example, it might use the equality index for an initial substring filter. The index used during the search does appear in the debugsearchindex attribute. Also, once the result set has been narrowed down to very few entries, it will stop using index and evaluate directly the entry set, as for the example below:

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(cn=Denice*)(mail=user.9*))" debugsearchindex
Password for user 'cn=Directory Manager':
dn: cn=debugsearch
debugsearchindex: filter=(&(cn=Denice*)[INDEX:cn.equality][COUNT:1])[COUNT:1] final=[COUNT:1]

Disabling Replication in OpenDJ 2.4.

Enabling replication between multiple instances of the OpenDJ LDAP directory server is pretty simple and straightforward. You can check for yourself in the Replication chapter of the Administration Guide.

But fully disabling replication can be tricky with OpenDJ 2.4, mostly because of a known issue with the dsreplication disable –disableAll command : OPENDJ-249 : Doing dsreplication disable –disableAll is throwing a javax.naming.CommunicationException when removing contents of “cn=admin data”.

We are fixing this issue in OpenDJ 2.5, but for those who have deployed OpenDJ 2.4 and want to know how to fully remove all references to a replica in the topology, here are the steps to manually disable replication :

Note, all these steps should be done using ldapmodify, or an LDAP browser such as OpenDJ Control-Panel’s Manage Entry or Apache Directory Studio.

  1. For each replica to be disabled connect to it on the admin port (4444) and:
    1. MANDATORY: set the “ds-cfg-enabled” property to “false” in “cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config”
    2. OPTIONAL: recursively remove the entries beneath “cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config” using individual delete operations. Note that the configuration backend does not support the sub-tree delete control, so this has to be done iteratively. This step is also not mandatory, since replication was fully disabled in the previous step
    3. MANDATORY: remove each entry beneath “cn=Servers,cn=admin data” except the entry itself. I find the easiest way to do this is to perform a sub-tree delete and then add back the base entry
    4. OPTIONAL: remove (purge) unused instance keys from beneath “cn=instance keys,cn=admin data” *except* own key. This step is really independent of replication: administrators should periodically purge unused instance keys anyway when they are sure that they are no longer needed (e.g. used for signing backups, etc)
    5. MANDATORY: delete “uniqueMember” in “cn=all-servers,cn=Server Groups,cn=admin data”
  2. On one of the remaining enabled replicas, connect to it via the admin port and:
    1. MANDATORY: remove each disabled server beneath “cn=Servers,cn=admin data”
    2. OPTIONAL: remove (purge) each disabled instance key beneath “cn=Servers,cn=admin data” (see 1.4)
    3. MANDATORY: remove each disabled server from uniqueMember in “cn=all-servers,cn=Server Groups,cn=admin data”
    4. MANDATORY: get list of all remaining servers from “cn=all-servers,cn=Server Groups,cn=admin data”
  3. For each of the remaining enabled replicas obtained in step 2.4 connect to it via the admin port and:
    1. MANDATORY: remove each disabled server(rsPort) from ds-cfg-replication-server in “cn=replication server,cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config”
    2. MANDATORY: remove each disabled server(rsPort) from ds-cfg-replication-server in “cn=*,cn=domains,cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config”

An important tuning flag for OpenDJ with 64bit JVM…

If you’re running OpenDJ with a 64bit JVM with less than 32GB of heap size, be aware of the need to explicitly set the -XX:+UseCompressedOops option (unless you want to disable it).

Compressed oops is supported and enabled by default in Java SE 6u23 and later, when running a 64bit JBM with a value of -Xmx lower than 32GB. You can find more information about Compressed Oops in Java technical notes here: http://download.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html

However, OpenDJ internal database, in order to estimate properly the occupation of the DB cache and tune the cache eviction threads, needs to take into account the compressed oops option. For this is relies on the JVM option to be set explicitly. If the option is not explicitly set, the database may consider the cache full when it’s not, and run cache eviction too early, resulting in less optimized performances.

So, with 64bit JVM, make sure you add the -XX:+UseCompressedOops option to the start-ds line in the config/java.properties file. Then run bin/dsjavaproperties and restart OpenDJ to benefit from the new settings.

LDAP: Matching against the current time in OpenDJ

In LDAP, attributes have different syntaxes. The one used to indicate date and time is the GeneralizedTime, a string representation of the date and time, typically expressed in GMT time. For example, when an entry is modified, the server maintains the modifytimestamp attribute and sets a value like 20110825120001Z (for 2011, Aug 25, 12:00:01 GMT).

LDAP client applications often have to search for entries based on these date and time attributes, whether it is to find the entries that have been modified , or had the password changed recently… The way it is typically done, is the following: get from the system the current date, add or substract some fixed time (for example if you want to know the entries modified in the last 10 minutes), transform  to a GeneralizedTime, use that string in a search filter: (modifyTimestamp >= 20110825130000Z). If the application repeats that search a minute later, it has to recompute the value again, and again…

Ideally what application writers would like is to express the filter as an expression like (modifyTimestamp>=${CurrentTime} – 10 mn). However this is not compliant with LDAP. The proper way to solve this is to use extensible matching rules, and for that purpose, we’ve added 2 “relative time” matching rules in OpenDJ, the Open source LDAP Directory services for Java: one for “lower than” and one for “greater than”.

matchingrules: ( 1.3.6.1.4.1.26027.1.4.6 NAME ( 'relativeTimeLTOrderingMatch' 'relativeTimeOrderingMatch.lt' )
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 )
matchingrules: ( 1.3.6.1.4.1.26027.1.4.5 NAME ( 'relativeTimeGTOrderingMatch' 'relativeTimeOrderingMatch.gt' )
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 )

The way the matching rules work is pretty simple : (attribute:MatchingRule:=Offset), where the offset is a signed integer follow by its unit, either s for seconds, m for minutes, h for hours, d for days or w for weeks.

You can translate a statement to “is Attribute greater than (or lower than) CurrentTime +/- Offset”

(lastLoginTime:1.3.6.1.4.1.26027.1.4.6:=-4w) will match all entries who have a lastLoginTime value smaller than the Current Time minus 4 weeks, i.e. all entries who have a lastLoginTime older than 4 weeks.

(pwdExpirationTime:1.3.6.1.4.1.26027.1.4.5:=5d) will match all entries that have pwdExpirationTime greater than the Current Time plus 5 days, i.e. all entries that will expire in more than 5 days.

The true benefit of those matching rules, is actually when expressing policies in the OpenDJ server, for example for granting or denying access based on some attribute with a generalizedTime syntax, such as last login time, pwdChangedTime, modifyTimeStamp …

For example, imagine an auxiliary objectClass representing a service, with some specific attributes including an expiration date : validUntil. Now, you want to allow these attributes to be read only if the expiration date is not passed.

aci: (targetattr="serviceAttr1 || serverAttr2")(targetfilter="(validUntil:1.3.6.1.4.1.26027.1.4.5:=0s)")
  (version 3.0; acl "Read Valid service attributes"; allow (read, search, compare)
  userdn="ldap:///all";)

As you can see, this is a good way to hide (deny access to) stale data in a directory server, and to simplify client applications that need to search for entries based on some generalizedTime attributes. For example, consider using these “relative time” matching rules for all your audit queries for expired or unused accounts.

Finally, remember that the OpenDJ directory server doesn’t allow unindexed searches by default. So you might also want to create an index for the “relative time” matching rules. That’s a 2 steps process :

Define the index

$ bin/dsconfig create-local-db-index --backend-name userRoot --set index-type:extensible \
 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.5 \
 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.6 \
 --index-name createTimestamp -h localhost -p 4444 \
 -D cn=Directory\ Manager -w secret12 -n -X

Rebuild the index

$ bin/rebuild-index -b dc=example,dc=com -i createTimestamp \
 -h localhost -p 4444 -D cn=directory\ manager -w secret12 -X