Jboss AJP thread pool not released idle threads

In this blog post, I am going to explain the problem and the solution related to AJP connector executor leaking threads.

We may notice and detect that AJP thread pool with the thread counts will start increase continuously on application servers after upgrade patch with the latest Jboss EAP version. It looks like that the idle threads never recycle or reclaim even after keepalive-time. This behaviour will happen continuously even if the system will remain idle for a long period of time (without any user request). The application leak connections because it is not closing the AJP connection and returning them to the pool.

In the case of unbounded-queue-thread-pool executor configured and the default maximum-connections value being used, a new thread will create for every new request and will remain idle forever. As the number of AJP requests increase, the response from Jboss EAP may get slow eventually.

If we capture a java thread dump (kill -3 Java_ProcessID in Unix to generate thread dump) and check for the AJP threads, we can analyse that all AJP thread wait for requests and are hanging and holding on to the established keepalive connections.

"ajp-/XX.XX.XX.XX:9412-942" daemon prio=2 tid=xxxxx nid=XXX runnable [xxxxx] 
    java.lang.Thread.State: RUNNABLE 
 at java.net.SocketInputStream.socketRead0(Native Method) 
 at java.net.SocketInputStream.read(SocketInputStream.java:152) 
 at java.net.SocketInputStream.read(SocketInputStream.java:122) 
 at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1124) 
 at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1206) 
 at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:438) 
 at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) 
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) 
 at java.lang.Thread.run(Thread.java:745)

The reason for this behaviour is that the AJP thread-pool connections by default have no timeout (default value of "DEFAULT_CONNECTION_TIMEOUT" is -1 for AJP, which means it never recycle) and therefore, the thread connection will persist permanently once established. The same behaviour detectable if we use httpd/mod_cluster in front of the JBoss which will invoke periodic pings to JBoss and fill its connection pool.

The solution is to add the following configuration in standalone.xml or domain.xml to limit the keepalive timeout value for AJP:

<system-properties>>
 <!-- default connection timeout is "-1" (means never timeout) --> 
 <property name="org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT" 
         value="60000"/> 
</system-properties>

Playing with Java Executor - Tasks ExecutorService Parallel Loop

If the iterations of the loop are independent and we don’t need to wait for all tasks to complete before proceeding, we can use an Executor to transform a sequential loop into a parallel tasks execution.

Transform sequential loop into a parallel using ExecutorService and CompletionService

First consider an example which process all task using sequential loop.

Public void SequentialLoop(List elements) {
  for (Element e : elements)
      process(e);
}

Let's modify our code and use an ExecutorService to execute task in parallel.

public void processLoopInParallel(Executor exec, List elements) {
  for (final Element e : elements)
      exec.execute(new Runnable() {
          public void run() { process(e); }
      });
}

The second loop will return more quickly, since all the task are queued to the Executor, rather than waiting for each tasks to complete.

If you want to submit a set of tasks and you want to retrieve the results as they become available, we can use CompletionService

void solve(Executor e, Collection<callable<result>> solvers)
     throws InterruptedException, ExecutionException {
       CompletionService<result> ecs
           = new ExecutorCompletionService<result>(e);
       for (Callable<result> s : solvers)
           ecs.submit(s);
       //retrieve completed task results and use them  
       int n = solvers.size();
       for (int i = 0; i < n; ++i) {
           Result r = ecs.take().get();
           if (r != null)
               use(r);
       }
   }

The advantage of using CompletionService is that it always returns the first completed task result. In this way, we can ensure that we are not waiting for tasks to complete while uncompleted tasks are running in the background.

If you want to submit a set of tasks and wait for them all to complete, you can use ExecutorService.invokeAll