Jboss AJP thread pool not released idle threads

In this blog post, I am going to explain the problem and the solution related to AJP connector executor leaking threads.

We may notice and detect that AJP thread pool with the thread counts will start increase continuously on application servers after upgrade patch to new Jboss EAP version. It looks like that the idle threads never recycle or reclaim even after keepalive-time. This behaviour will happen continuously even if the system will remain idle for a long period of time (without any user request). The application leak connections because it is not closing the AJP connection and returning them to the pool.

In the case of unbounded-queue-thread-pool executor configured and the default maximum-connections value being used, a new thread will create for every new request and will remain idle forever. As the number of AJP requests increase, the response from Jboss EAP may get slow eventually.

If we capture a java thread dump (kill -3 Java_ProcessID in Unix to generate thread dump) and check for the AJP threads, we can analyse that all AJP thread wait for requests and are hanging and holding on to the established keepalive connections.

"ajp-/XX.XX.XX.XX:9412-942" daemon prio=2 tid=xxxxx nid=XXX runnable [xxxxx] 
    java.lang.Thread.State: RUNNABLE 
 at java.net.SocketInputStream.socketRead0(Native Method) 
 at java.net.SocketInputStream.read(SocketInputStream.java:152) 
 at java.net.SocketInputStream.read(SocketInputStream.java:122) 
 at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1124) 
 at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1206) 
 at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:438) 
 at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) 
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) 
 at java.lang.Thread.run(Thread.java:745)

The reason for this behaviour is that the AJP thread-pool connections by default have no timeout (default value of "DEFAULT_CONNECTION_TIMEOUT" is -1 for AJP, which means it never recycle) and therefore, the thread connection will persist permanently once established. The same behaviour detectable if we use httpd/mod_cluster in front of the JBoss which will invoke periodic pings to JBoss and fill its connection pool.

The solution is to add the following configuration in standalone.xml or domain.xml to limit the keepalive timeout value for AJP:

<system-properties>>
 <!-- default connection timeout is "-1" (means never timeout) --> 
 <property name="org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT" 
         value="60000"/> 
</system-properties>

Google Gmail API

Google introduces new Gmail API which enables developers to use Gmail features into applications and make it easier for others to integrate and use as Plug-in.

According to Google,

The Gmail API gives you flexible, RESTful access to the user's inbox, with a natural interface to Threads, Messages, Labels, Drafts, and History.

The new API provides RESTful access to an inbox and supports CRUD operations on email threads, messages, labels etc.

The Gmail API can be used in variety of applications including the cases where full inbox access is not required and the API can be much quicker compared to IMAP to search for a particular email.

The API provides fine-grained control to applications apposite to IMAP, where developers are required to obtain permission to the whole of an email inbox and therefore, will be restricted to:

  • Read messages from Inbox
  • Send messages
  • Amend the labels affect to messages
  • Search for specific messages

More Reading,

Interface Default Methods in Java 8

Java 8 introduces “Default Method” or (Defender methods) new feature, which allows developer to add new methods to the Interfaces without breaking the existing implementation of these Interface. It provides flexibility to allow Interface define implementation which will use as default in the situation where a concrete Class fails to provide an implementation for that method.

Let consider small example to understand how it works:

public interface OldInterface {
    public void existingMethod();

    default public void newDefaultMethod() {
        System.out.println("New default method"
               + " is added in interface");
    }
}

The following Class will compile successfully in Java JDK 8,

public class OldInterfaceImpl implements OldInterface {
    public void existingMethod() {
     // existing implementation is here…
    }
}

If you create an instance of OldInterfaceImpl:

OldInterfaceImpl obj = new OldInterfaceImpl ();
// print “New default method add in interface”
obj.newDefaultMethod();  

Why Defaut Method?

Reengineering an existing JDK framework is always very complex. Modify one Interface in JDK framework breaks all Classes that extends the Interface which means that adding any new method could break millions of lines of code. Therefore, default methods have introduced as a mechanism to extending Interfaces in a backward compatible way.

Default methods can be provided to an Interface without affecting implementing Classes as it includes an implementation. If each added method in an Interface defined with implementation then no implementing Class is affected. An implementing Class can override the default implementation provided by the Interface.

For Java 8, the JDK collections have been extended and forEach method is added to the entire collection (which work in conjunction with lambdas). With conventional way, the code looks like below,

public interface Iterable<T> {
    public void forEach(Consumer<? super T> consumer);
}

Since this result each implementing Class with compile errors therefore, a default method added with a required implementation in order that the existing implementation should not be changed.

The Iterable Interface with the Default method is below,

public interface Iterable<T> {
    public default void forEach(Consumer
                   <? super T> consumer) {
        for (T t : this) {
            consumer.accept(t);
        }
    }
}

The same mechanism has been used to add Stream in JDK Interface without breaking the implementing Classes.

When to use Default Method over Abstract Classes

Abstract Classes versus Interfaces in Java 8

After introducing Default Method, it seems that Interfaces and abstract Classes are same. However, they are still different concept in Java 8.

Abstract Class can define constructor. They are more structured and can have a state associated with them. While in contrast, default method can be implemented only in the terms of invoking other Interface methods, with no reference to a particular implementation's state. Hence, both use for different purposes and choosing between two really depends on the scenario context.

Default Method and Multiple Inheritance Ambiguity Problems

Since java Class can implement multiple Interfaces and each Interface can define default method with same method signature, therefore, the inherited methods can conflict with each other.

Consider below example,

public interface InterfaceA {  
       default void defaultMethod(){  
           System.out.println("Interface A default method");  
    }  
}

public interface InterfaceB {
   default void defaultMethod(){
       System.out.println("Interface B default method");
   }
}

public class Impl implements InterfaceA, InterfaceB  {
}

The above code will fail to compile with the following error,

java: class Impl inherits unrelated defaults for defaultMethod() from types InterfaceA and InterfaceB

In order to fix this class, we need to provide default method implementation:

public class Impl implements InterfaceA, InterfaceB {
    public void defaultMethod(){
    }
}

Further, if we want to invoke default implementation provided by any of super Interface rather than our own implementation, we can do so as follows,

public class Impl implements InterfaceA, InterfaceB {
    public void defaultMethod(){
        // existing code here..
        InterfaceA.super.defaultMethod();
    }
}

We can choose any default implementation or both as part of our new method.

Difference between Default Method and Regular Method

Default Method is different from the regular method in the sense that default method comes with default modifier. Additionally, methods in Classes can use and modify method arguments as well as the fields of their Class but default method on the other hand, can only access its arguments as Interfaces do not have any state.

In summary, Default methods enable to add new functionality to existing Interfaces without breaking older implementation of these Interfaces.

When we extend an interface that contains a default method, we can perform following,

  • Not override the default method and will inherit the default method.
  • Override the default method similar to other methods we override in subclass..
  • Redeclare default method as abstract, which force subclass to override it.

Some References

JBoss Deployment war error on startup - No content hash available

In this post, we are going to discuss about the deployment problem that prevent JBoss Server to start properly.

Error starting Java JBoss server (in domain mode)

The JBoss Server logs in this case looks like below,
10:16:57,337 INFO [org.jboss.modules] JBoss Modules version 1.1.3.GA-redhat-1
10:16:57,498 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA-redhat-2

10:16:57,686 INFO [org.jboss.as] JBAS015899: JBoss EAP 6.0.1.GA
(AS 7.1.3.Final-redhat-4) starting

10:16:58,310 INFO [org.jboss.as.server.deployment.scanner] JBAS015014: Re-attempting failed deployment abc.warploy

10:16:58,340 INFO [org.jboss.as.server.deployment.scanner] JBAS015003: Found abc.war in deployment directory. To trigger deployment create a file called abc.war.dodeploy

10:16:58,348 ERROR [org.jboss.as.controller.management-operation] JBAS014613: Operation ("add") failed - address: ([("deployment" => "abc-ABC.0.0.21.war")]) - failure description: "JBAS018717: No deployment content with hash 28c2ce34057a6bd5ebf2c28f9d114814faa66b8a is available in the deployment content repository for deployment 'abc-ABC.0.0.21.war'.

This is a fatal boot error. To correct the problem, either restart with the --admin-only switch set and use the CLI to install the missing content or remove it from the configuration, or remove the deployment from the xml configuraiton file and restart."

10:16:58,351 ERROR [org.jboss.as.server.deployment.scanner] JBAS014654: Composite operation was rolled back

10:16:58,351 FATAL [org.jboss.as.server] JBAS015957: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.

10:16:58,358 INFO [org.jboss.as] JBAS015950: JBoss EAP 6.0.1.GA (AS 7.1.3.Final-redhat-4) stopped in 3ms

This error also occurs when we delete directory standalone\data.. and then try to redeploy the same application war. The JBoss server unable to delete entries defined in standalone.xml with the same hash for the deployed application.

The standalone.xml looks like below,
<deployments>
    <deployment name="abc-ABC.0.0.21.war" runtime-name="abc-ABC.0.0.21.war">
         <content sha1="28c2ce34057a6bd5ebf2c28f9d114814faa66b8a"/>
    </deployment>
</deployments>

Removing the entries manually for the application having defined hash solved this problem and JBoss server will start successfully without error.

Eclipse releases new Features and Plugins

Eclipse has recently announced the release of Eclipse Juno which brings new features such as Detecting resource leaks for both Java7 and Java6 etc.

Previous Plugins to Install

I firstly install all my plugins from previous eclipse version. The process is quite simple
  • import and install from existing installation
  • refer to old eclipse location
  • Eclipse will show the list of plugins available and then asked for installation
  • After accepting license, installation should complete smoothly.

New Features available

  • Java 7 coding support in an IDE. Since Java 7 contains many new feature including dynamically-typed languages support and other small enhancements from Coin project, eclipse Java development tooling will include support for these features of Java 7.
  • Detecting resource leaks of Closeable/Autocloseable resources. This feature is really cool and it also works with “old code”. The common IO and JDBC resources now implement the relevant interfaces and the warnings are good enough to be shown on these.
    Resource leaks can be occurred in the following situations:
         1. A resource opened but not closed
         2. A resource may not closed on all control flows
         3. A resource may not closed at a method exit point
         4. In a Java 7 program a resource is closed but the code could still be improved by using a try-with-resources statement.

  • The new version include the global search bar that gives user quick access to almost any Eclipse feature.
    Glabal search bar
  • Code Recommenders This is amazing feature that analyzes code of existing applications and extracts common patterns of how other developers have used. It helps deal with the complexity of large APIs using intelligent code completion attribute.

    More examples can be found from below links,

    http://eclipsesource.com/blogs/2012/06/26/code-recommenders-top-eclipse-juno-feature-2/
    http://code-recommenders.blogspot.com/

  • The addition of integrated debugging of JVM-based domain specific languages in the Xtext development framework, The below link provide extensive examples regarding the mentioned feature, http://eclipsesource.com/blogs/2012/06/25/xtext-xtend-top-eclipse-juno-feature-3/

  • Eclipse for Mobile Developers - now developers can use eclipse more easily with a variety of mobile SDKs, including the Google Android SDK

How to Run Elastic MapReduce Hadoop Job Using Custom Jar - Amazon EMR Tutorial

Amazon EMR is a web service using which developers can easily and efficiently process enormous amounts of data. It uses an hosted Hadoop framework running on the web-scale infrastructure of Amazon EC2 and Amazon S3.
Amazon EMR removes most of the cumbersome details of Hadoop, while take care for provisioning of Hadoop, running the job flow, terminating the job flow, moving the data between Amazon EC2 and Amazon S3, and optimizing Hadoop.
In this tutorial, we will first going to developed WordCount java example using MapReduce framework Hadoop and thereafter, we execute our program on Amazon Elastic MapReduce.

Prerequisites

You must have valid AWS account credentials.You should also have a general familiarity with using the Eclipse IDE before you begin. The reader can also use any other IDE of their choice.

Step 1 – Develop Hadoop MapReduce WordCount Java Program

In this section, we will first going to develop WordCount application. A WordCount program will determine how many times different words appear in a set of files.
  • 1. In Eclipse (or whatever the IDE you are using), Create simple Java Project with name "WordCount".
  • 2. Create a java class name Map and override the map method as follow,
    public class Map extends Mapper<longwritable, 
                               intwritable="" text,=""> {
     private final static IntWritable one = 
                              new IntWritable(1);
     private Text word = new Text();
    
     @Override
     public void map(LongWritable key, Text value, 
                         Context context)
         throws IOException, InterruptedException {
       String line = value.toString();
       StringTokenizer tokenizer = new 
                            StringTokenizer(line);
       while (tokenizer.hasMoreTokens()) {
           word.set(tokenizer.nextToken());
           context.write(word, one);
       }
      }
    }
    
  • 3.Create a java class name Reduce and override the reduce method as below,
    public class Reduce extends Reducer<text, 
                  intwritable,="" intwritable="" text,=""> {
     @Override
     protected void reduce(
       Text key,
       java.lang.Iterable<intwritable> values,
       org.apache.hadoop.mapreduce.Reducer<text, 
               intwritable,="" intwritable="" text,="">.Context context)
       throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable value : values) {
       sum += value.get();
      }
      context.write(key, new IntWritable(sum));
     }
    }
    
  • 4. Create a java class name WordCount and defined the main method as below,
    public static void main(String[] args) 
                          throws Exception {
      Configuration conf = new Configuration();
    
      Job job = new Job(conf, "wordcount");
      job.setJarByClass(WordCount.class);
    
      job.setOutputKeyClass(Text.class);
      job.setOutputValueClass(IntWritable.class);
    
      job.setMapperClass(Map.class);
      job.setReducerClass(Reduce.class);
    
      job.setInputFormatClass(TextInputFormat.class);
      job.setOutputFormatClass(TextOutputFormat.class);
    
      FileInputFormat.addInputPath(job, new Path(args[0]));
      FileOutputFormat.setOutputPath(job, new Path(args[1]));
    
      job.waitForCompletion(true);
    }
    
  • 5. Export the WordCount program in a jar using eclipse and save it to some location on disk. Make sure that you have provided the Main Class (WordCount.jar) during extracting the jar file as shown below.
jar
jar ready
                                 ur jar is ready!!

Step 2 – Upload the WordCount JAR and Input Files to Amazon S3

Now we are going to upload the WordCount jar to Amazon S3. First, go to the following URL: https://console.aws.amazon.com/s3/home Next, click “Create Bucket”, give your bucket a name, and click the “Create” button. Select your new S3 bucket in the left-hand pane. Upload the WordCount JAR and sample input file for counting the words.

Step 3 – Running an Amazon Elastic MapReduce Hadoop job

Running Hadoop WordCount example

Now that the JAR is uploaded into S3, all we need to do is to create a new Job flow. let's execute the below steps. (I encourage reader to check out the following link for details regarding each step, How to Create a Job Flow Using a Custom JAR )
  • 1. Sign in to the AWS Management Console and open the Amazon Elastic MapReduce console at https://console.aws.amazon.com/elasticmapreduce/
  • 2. Click Create New Job Flow.
  • 3. In the DEFINE JOB FLOW page, enter the following details,

    a. Job Flow Name = WordCountJob
    b. Select Run your own application
    c. Select Custom JAR in the drop-down list
    d. Click Continue

  • 4. In the SPECIFY PARAMETERS page, enter values in the boxes using the following table as a guide, and then click Continue.
    JAR Location = bucketName/jarFileLocation
    JAR Arguments =
    s3n://bucketName/inputFileLocation
    s3n://bucketName/outputpath

    Please note that the output path must be unique each time we execute the job. The Hadoop always create folder with same name specify here.

After executing job, just wait and monitor your job that runs through the Hadoop flow. You can also look for errors by using the Debug button. The job should be complete within 10 to 15 minutes (can also depend on the size of input). After completing job, You can view results in the S3 Browser panel. You can also download the files from S3 and can analyze the outcome of the job.

Amazon Elastic MapReduce Resources

Free Open Source CRM and Helpdesk Software

Last week, I was surfing on web to search for an free open source CRM and help desk solution. This post will share the experience and recommend some free open source CRM and Help Desk solution.

Help desk and CRM software's are always essential for any company to provide best customer support. This tools not only helps companies in increasing the customer satisfaction but also customer retention as well. In this post, we'll examine and recommend some free and open source CRM and Help desk solution.

SugarCRM and vtiger

In my opinion, vtiger is the best option as free CRM and helpdesk software. It will work both as an Help desk and CRM.

Comparing SugarCRM and vtiger, both are providing the following major functionalities,

SugarCRM Marketing Sales Support Activities Collaboration Reports
Vtiger Marketing Sales Support Quick Create Tools Analytics

There are many different sub actions defined within these categories which include Accounts, Contracts, Leads, Documents etc. Therefore, after getting some picture of CRM, the overall interface of vtiger seems to be good for me now.

There are also many positive feedbacks available for vtiger on the web. You can find the links in 'more information' section.

There is also community version available for SugarCRM [6] however, it does not contain reporting feature.  We might able to add some plug-ins for that [5] but it might need to explore more regarding how much work would need for this.

Kayako and SugarCRM

These are both awesome and renowned tools, however they are commercial and not free.Also for registered charities and open-source projects, Kayako Fusion, Kayako Resolve and Kayako Engage licenses are available free of cost.
Kayako Download licenses for Kayako Fusion, Kayako Resolve and Kayako Engage are available free of charge to registered charities and open-source projects. Free, perpetual licenses with unlimited users and no strings attached. Finally; some viable, business-class, free help desk software!

Other CRM and helpdesk option

Since vtiger will fulfill the requirement for both help desk and CRM feature, therefore you might not require any free separate help desk software. However, if you still would like to explore, I will highly recommend the below two tools.


Both software are available free and source code can also be downloadable. The source code is in PHP.

I find Trellis Desk awesome as its interface is good and it also provide good functionalities for help desk management systems.

Other tools include (not in PHP)

References and More Information