Glassfish Application Server Performance tunings

If you want to improve performance of your application running on Glassfish application server, you can try out following JVM options. We got significant performance improvement by using them.

<jvm-options>-server</jvm-options>

<jvm-options>-Xms3000m</jvm-options>
<jvm-options>-Xmx3000m</jvm-options>
<jvm-options>-XX:MaxPermSize=192m</jvm-options>
<jvm-options>-XX:NewRation=2</jvm-options>
<jvm-options>-XX:+AggressiveHeap</jvm-options>
<jvm-options>-XX:+AggressiveOpts</jvm-options>
<jvm-options>-XX:+UseParallelGC</jvm-options>
<jvm-options>-XX:+UseParallelOldGC</jvm-options>
<jvm-options>-XX:ParallelGCThreads=5</jvm-options>

By using these performance tunings, we are able to get almost 30% more TPS (Transactions per Seconds).If I can be helpful, please do let me know. I have spent enough time on Glassfish and OpenESB performance tunings.

Absolute Singleton. Not Really...

Singleton are not true singleton and there are many reasons: One of them is multiple class loaders can have their own copy of singleton. See How single can your singleton instance be?. I started googling for true singleton and came across Absolute Singleton.

When I use this approach in my product, it didn't work instead it was throwing exception java.lang.ClassNotFoundException on the Singleton class.

java.lang.ClassNotFoundException: com.example.common.utils.JCSCache at com.sun.enterprise.util.ConnectorClassLoader.loadClass(ConnectorClassLoader.java:222)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at com.example.common.utils.JCSCache.getInstance(JCSCache.java:131)


After looking at the exception and class initialization, I found that what happens if the first instance is initiated by custom class loader and not the default one?

It will not find an existing instance because while creating the instance, due to custom class loader, it will go into "if (! myClassLoader.toString().startsWith("sun.")) {" (see the Absolute Singleton) condition and try to find existing instance instead of creating new which doesn't exits yet so it throws the java.lang.ClassNotFoundException.

The work around I found is to catch the exception and create the instance of AbsoluteSingleton class.

But next time, custom class loader try to get the instance, again it's not going to find it and create it's own instance.
See the below log.

First instance of JCSCache ( com.example.common.utils.JCSCache@124adaf) created by class loader: org.glassfish.openesb.pojose.jbi.su.SUClassLoader@1b78d60
and

2nd instance (com.example.common.utils.JCSCache@1a859dc) is created by EJBClassLoader.

So Absolute singleton is not really absolute singleton.

[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | ENTRY] -->getInstance()
[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | DEBUG] Class Loader:org.glassfish.openesb.pojose.jbi.su.SUClassLoader@1b78d60
[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | ERROR] Exception:com.example.common.utils.JCSCache
[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | ENTRY] -->JCSCache()
[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | DEBUG] JCSCache constructor called.
[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | EXIT] <--JCSCache() [2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | ENTRY] -->init()
[2009-09-27 21:56:04,713 | Thread-79 | common.utils.JCSCache | DEBUG] Setting the cache configuratio name:/cache.ccf
[2009-09-27 21:56:04,721 | Thread-79 | common.utils.JCSCache | DEBUG] Initializing region base cache objects.
[2009-09-27 21:56:04,786 | Thread-79 | common.utils.JCSCache | EXIT] <--init() true [2009-09-27 21:56:04,786 | Thread-79 | common.utils.JCSCache | EXIT] <--getInstance() com.example.common.utils.JCSCache@124adaf

2nd instance:

2009-09-27 21:56:06,812 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | ENTRY] -->getInstance()
[2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | DEBUG] Class Loader:EJBClassLoader :
[2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | ERROR] Exception:com.example.common.utils.JCSCache
[2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | ENTRY] -->JCSCache()
[2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | DEBUG] JCSCache constructor called.
[2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | EXIT] <--JCSCache() [2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | ENTRY] -->init()
[2009-09-27 21:56:06,813 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | DEBUG] Setting the cache configuratio name:/cache.ccf
[2009-09-27 21:56:06,818 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | DEBUG] Initializing region base cache objects.
[2009-09-27 21:56:06,926 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | EXIT] <--init() true [2009-09-27 21:56:06,926 | httpSSLWorkerThread-8080-0 | common.utils.JCSCache | EXIT] <--getInstance() com.example.common.utils.JCSCache@1a859dc

How to Enabled logging in OpenLdap

I am working on performance improvement task for our new product where I wanted avoid duplicate Ldap seaches. To view the ldap searches in sldap log file, we have to enable the logging. This is how you do it.

To enabled logging:

ldapmodify -x -H ldap://ldaphost -D cn=directory\ manager,cn=config -w password
dn: cn=Config
replace: olcLogLevel
olcLogLevel: Stats Stats2 None
Ctrl+d

Now you can view the log file at /var/log/slapd.log. You will need a root/sudo access to view this log file.

To disabled logging:
ldapmodify -x -H ldap://ldaphost -D cn=directory\ manager,cn=config -w password
dn: cn=Config
replace: olcLogLevel
olcLogLevel: None
Ctrl+d

NOTE: Here is None is used for reporting errors so don't forget to specify None when you enable or disable logging.

If you want to modify existing subscriber data in ldap, you can use this command
ldapmodify -x -H ldap://host -D cn=directory\ manager,o=example.com -w password123 -f modifieddata.ldif

where modifieddata.ldif would contain
dn: uniqueIdentifier=0209091514-1015999999,ou=subscribers,ou=OU,o=example.com
changetype: modify
replace:
:

dn: uniqueIdentifier=0209091514-1016000000,ou=subscribers,ou=OU,o=example.com
changetype: modify
replace:
:

Berkeley DBXML and thread deadlock

For my XCAPServer project, I use Berkeley DB XML for XML document repository. I am kind of stuck due to all threads goes into deadlock while adding and removing documents.

See more information on dbxml forum at: DBXML 2.4.16: Accessing containers from multiple threads and deadlock

Solution: To avoid the deadlock, call set_lk_detect(DB_LOCK_MINWRITE) on DbEnv variable. Also make sure when exceptions are thrown, transactions must be either aborted or committed.

Here is a link to the source code: dbxml_db.cpp

XPATH Library performance

I am working on a project which requires invoking multiple JBI endpoints based on routing requirements and type of input message. It is developed using heavy xpath expressions to choose the right path in the routing logic.

I started with default XPath library which is part of the JDK 1.6. To avoid the performance problem, I pre-compiled all the xpath expressions at startup time.

When I ran performance test on this router application, I was getting around 670 TPS (transactions per second) with 20% CPU idle (it's a 8 core machine) which is much slower than what I expected.

So I started looking for alternate XPATH libraries and I found VDT-XML benchmark where you can see VDT-XML outperforms jaxen and default jkd xpath implementation.

VDT-XML seem to be much faster (based on benchmark results :) )but due to it's license, I decided not to use for my project so I only option left was to try jaxen xpath library.

When I replaced default xpath library with jaxen, performance numbers changes drastically. Now I get 800 TPS with 70% CPU idle. In-fact I couldn't drive more CPU usage with my jmeter running on one machine.

So I would recommend jaxen library to anyone who is looking high performance, open source/free xpath library.

Boost Logging Library

I was evaluating boost log library for my upcoming project XCAPServer. When I googled for boost and logging, I found two boost based logging library.

1. http://torjo.com/log2/doc/html/index.html
2. https://sourceforge.net/projects/boost-log/.

I started with torjo's log library documentation but it looked more complex so I gave up on that.

boost-log library documentation is really good and getting started was much easier so I downloaded the boost-log source and compiled along with my other boost libraries. You can find how to compile boost libraries here.

After spending some time on this library what I realized is that it doesn't support basic logging requirements for my multi-threading application so I had to use C++ macros to achieve what I wanted.

Here is my logger files:
xcap_logger.hpp

xcap_logger.cpp

I just don't understand why logging libraries have to be so complex and still doesn't provide basic logging features e.g file-name, line-number, thread-id etc..we need to learn from java community.

Sometime I feel that boost community should define the interface and let people implement rather than people contributing the implementations and boost community keep on rejecting :)

Here is what I am looking in boost logging library.

My basic requirements for logging libraries are below.

1. Date and Time
2. File Name. Similar to __FILE__ macro
3. Line Number. Similar to __LINE__ macro
4. Current thread id. (pthread_self)
5. Parent thread id. (getpid())
6. Log level (from trace, debug, info, warn, error, critical)
7. Module name
8. Data
e.g
|2009-Sep-22 22:04:47.340047|file.cpp:212|0xb0103000:0xb01020000|INFO|XCAP|test-data

Compiling Boost libraries on Mac/Linux

1. Download/checkout the boost source code. Let's say we are downloading it at /opt/boost directory
>cd /opt/boost
>svn co http://svn.boost.org/svn/boost/trunk boost-trunk
2. Download following external source libraries at /opt/boost directory and unzip and compile them.
a. bzip
b. expat
c. icu
d. zlib
3. cd to boost-trunk directory and run ./bootstrap.sh (you might have to execute chmod 755 bootstrap.sh)
4. Below is the script which sets the environments and compiles the boost libraries.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/usr/local/lib:/usr/lib:.
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH
export BOOST_BIN_ROOT=/opt/boost/boost-binary
export BOOST_INSTALL_DIR=/opt/boost/boost-trunk
export PATH=$PATH:/opt/boost/bjam
export EXPAT_INCLUDE=/opt/boost/expat/include
export EXPAT_LIBPATH=/opt/boost/expat/lib
export HAVE_ICU=1
export BZIP2_SOURCE=/opt/boost/bzip2-1.0.5
export ZLIB_SOURCE=/opt/boost/zlib-1.2.3
export ICU_PATH=/opt/boost/icu/source
cd $BOOST_INSTALL_DIR
bjam threading=multi variant=release link=shared --build-dir=$BOOST_BIN_ROOT --prefix=$BOOST_INSTALL_DIR --without-mpi toolset=darwin install

5. If you want to compile specific library, replace bjam command from above script
bjam threading=multi variant=release link=shared --with-$1 --build-dir=$BOOST_BIN_ROOT --prefix=$BOOST_INSTALL_DIR toolset=darwin install

Here in --with-$1 where $1 would be your library name.