In this blog post, I am going to explain the problem and the solution related to AJP connector executor leaking threads.
We may notice and detect that AJP thread pool with the thread counts will start increase continuously on application servers after upgrade patch with the latest Jboss EAP version. It looks like that the idle threads never recycle or reclaim even after keepalive-time. This behaviour will happen continuously even if the system will remain idle for a long period of time (without any user request). The application leak connections because it is not closing the AJP connection and returning them to the pool.
In the case of unbounded-queue-thread-pool executor configured and the default maximum-connections value being used, a new thread will create for every new request and will remain idle forever. As the number of AJP requests increase, the response from Jboss EAP may get slow eventually.
If we capture a java thread dump (kill -3 Java_ProcessID in Unix to generate thread dump) and check for the AJP threads, we can analyse that all AJP thread wait for requests and are hanging and holding on to the established keepalive connections.
"ajp-/XX.XX.XX.XX:9412-942" daemon prio=2 tid=xxxxx nid=XXX runnable [xxxxx]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1124)
at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1206)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:438)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
at java.lang.Thread.run(Thread.java:745)
The reason for this behaviour is that the AJP thread-pool connections by default have no timeout (default value of "DEFAULT_CONNECTION_TIMEOUT" is -1 for AJP, which means it never recycle) and therefore, the thread connection will persist permanently once established. The same behaviour detectable if we use httpd/mod_cluster in front of the JBoss which will invoke periodic pings to JBoss and fill its connection pool.
The solution is to add the following configuration in standalone.xml or domain.xml to limit the keepalive timeout value for AJP:
<system-properties>>
<!-- default connection timeout is "-1" (means never timeout) -->
<property name="org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT"
value="60000"/>
</system-properties>
Posts
