10/15/2012

Java Applicatioin Process Hangs on Windows and Cached Thread Pool

The following java test app runs and terminates normally on Unix but hangs on Windows.

import java.util.concurrent.*;

public class ThreadPoolTest {
    private static ExecutorService es = Executors.newCachedThreadPool();

    public static void main(String args[]) { 
        es.submit(new Task("## running task 1..."));
        es.submit(new Task("## running task 2..."));
        System.out.println("## after tasks, in main");
    }

    private static class Task implements Runnable {
        private String msg;

        private Task(String s) {
            this.msg = s;
        }

        @Override public void run() {
            System.out.println(msg);
        }
    }
}
The direct reason for hanging is the 2 worker threads were returned to the cached thread pool after finishing the printing task, and stayed alive and idle indefinitely. Pooled threads by default are non-daemon threads. A Java program will terminate only after all non-daemon threads have terminated. Although the main thread in the above test app have finished its job, the Java process will not terminate due to the 2 live worker threads.

A couple of solutions and their pros and cons:

1, So shall we make the pooled threads daemon threads to fix it? Be careful here, since the Java process may just terminate prematurely, immediately after the main thread is done, but before daemon workers complete their work. Since they are daemon threads, their task status is totally ignored in exiting JVM.

If you really want to take the daemon approach, the application will need some logic to poll the stask status, and wait for their completion before exiting the main thread.

2, How about explicitly call ExecutorService.shutdown() method? After all, a task implementing app logic is not a daemon thread, and should be marked as such. Shutting down the thread pool (the actual type of ExecutorService in our example) makes more sense. But pay attention to all possible exit paths and make sure shutdown guaranteed in all paths, such as normal completion, throwable.

If the thread pool is used by various parts of a large application, how do you coordinate them such that shutdown is called only after all clients are finished? Compared to approach 1, additional coordiation (e.g., with wait/notify or CountDownLatch) is needed. To fix the hang using shutdown():

import java.util.concurrent.*;

public class ThreadPoolTest {
    private static ExecutorService es = Executors.newScheduledThreadPool(2);

    public static void main(String args[]) throws InterruptedException {
        es.submit(new Task("# running task 1..."));
        es.submit(new Task("# running task 2..."));
        es.shutdown();

        //block here for all tasks to finish before proceeding in the main thead.
        //if you want to enforce the execution order. optional.
        es.awaitTermination(1, TimeUnit.DAYS);
        System.out.println("# after tasks, in main");
    }

    private static class Task implements Runnable {
        private String msg;

        private Task(String s) {
            this.msg = s;
        }

        @Override public void run() {
            try {
                Thread.sleep(1000*30);
            } catch (InterruptedException e) {
            }
            System.out.println(msg);
        }
    }
}

awaitTermination is totally optional. With it, you will see 2 task output before main thread output:
# running task 2...
# running task 1...
# after tasks, in main

Without it, main thread output comes before task output:
# after tasks, in main
# running task 2...
# running task 1...

3, Is there a default idle timeout for worker threads, so applications don't have to manage the shutdown? Yes, there is a default idle timeout 60 seconds for non-core threads. When creating a thread pool, you can specify the number of core threads, along with other parameters. By default core threads do not time out after becoming idle. But you can certainly make core threads subject to idle timeout by calling threadPool.allowCoreThreadsTimeout(true).

How does this apply to our test app? We created the thread pool by calling newCachedThreadPool() without passing any parameters. By default, the pool is created with 0 core threads and 60-second idle timeout. So any worker threads in the test app should be non-core and should automatically time out after 60 seconds. Yes, that is what happened on Mac OS or Linux. But on windows 7, the same program just hangs.

Do you still want to rely on its default idle timeout, which seems sensitive to OS?

4, Use Thread and Runnable class directly for simple use cases.  When you only need threads, but don't need to maintain a pool, just directly use threads.

To rewrite the above program directly using java.lang.Thread:
public class ThreadTest {
    public static void main(String args[]) throws InterruptedException {
        Thread t1 = new Thread(new Task("## running task 1..."));
        Thread t2 = new Thread(new Task("## running task 2..."));
        t1.start();
        t2.start();
        t1.join();
        t2.join();
        System.out.println("## after tasks, in main");
    }

    private static class Task implements Runnable {
        private String msg;

        private Task(String s) {
            this.msg = s;
        }

        @Override public void run() {
            System.out.println(msg);
        }
    }
}
The 2 join calls are added to wait for the 2 child threads to finish processing before proceeding to the main thread execution.

10/11/2012

How to Configure hprof in GlassFish 3.x

These are the steps to configure hprof profiler in GlassFish 3.x:

1, Identify the target JVM to profile.  In most cases, it's the domain administration (DAS) JVM, but it can be other JVM such as another standalone server instance, or a cluster server instance.

2, Edit $GLASSFISH_HOME/config/osgi.properties, locate org.osgi.framework.bootdelegation property, and append hprof classes to it, using , (comma) as the package separator.  Do not forget to add a comma at the end of the existing value.  The resulting property should look like this:

org.osgi.framework.bootdelegation=${eclipselink.bootdelegation}, \
                                  com.sun.btrace, com.sun.btrace.*, \
                                  org.netbeans.lib.profiler, org.netbeans.lib.profiler.*, \
                                  sun.tools.hprof,com.sun.tools.hat.internal.parser,com.sun.demo.jvmti.hprof

3, Start the domain, and go to admin console to add the hprof profiler:

asadmin start-domain
http://localhost:4848

On the left, choose Configurations | server-config | JVM Settings, on the rigth content panel, choose Profiler tab,

Name: hprof
Status: enabled yes
Classpath: not needed
Native library path: no needed
add a JVM Option:

-agentlib:hprof=file=log.txt,thread=y,depth=3

or using the old non-standard option:

-Xrunhprof:file=log.txt,thread=y,depth=3

Create this profiler and restart the domain from the command line.  You will see the following elements are added to $GLASSFISH_HOME/domains/domain1/config/domain.xml,

<profiler name="hprof" native-library-path="" classpath="">
  <jvm-options>-Xrunhprof:file=log.txt,thread=y,depth=3</jvm-options>
</profiler>

So you could also directly edit domain.xml (not recommended) to save you some clicks, if you know where to add this new element.  For server-config, which corresponds to DAS, it is usually after the first group of jvm-options elements.

There is also asadmin CLI commands to create and delete profiler configuration:

asadmin create-profiler hprof

asadmin delete-profiler

But create-profiler command doesn't allow you to specify hprof jvm options, so you would still need to come back to domain.xml to add jvm options.  However, it takes a --target param, where you can specify where to apply the profiler configuration.

4, After starting and stopping the domain, the hprof data file is created in $GLASSFISH_HOME/domains/domain1/config/ directory,  which is the ${user.dir} for GlassFish server process.

Another related post: How to Configure hprof in JBoss AS7

10/08/2012

My Intellij Notes

To search by keywords for a command, setting, shortcut, etc, and execute it:
Command + Shift + A

To maximize or full screen (useful when reading long lines).  You can also use the same shortcut to go full screen in FireFox, Safari, Chrome, Preview, iTunes, etc.
Command + Ctrl + F

To see javadoc for the method in focus:
Ctrl + J

To see which method/class the current line is in (with super long method body):
Ctrl + Shift + Q

To paste from clipboard history (including copied content from external apps):
Command + Shift + V

To copy the file path of a class:
Command + Shift + C, while inside the editor of that class, not even need to highlight the class name.

To copy the fully qualified name of a class (FQN):
Command + Alt + Shift + C

To delete a line:
Command + Y (y for yank)

To go to a line:
Command + G

To open a class:
Command + N (press it again to include non-project classes like JDK classes)

To organize imports:
Command + Alt + O

To bring up Version Control System popup (git, svn, etc):
Ctrl + V

To view all methods of the current class (press F12 again to include inherited methods):
Command + F12

To view type hierarchy (super-types, sub-types):
Ctrl + H

To jump to source file of another class:
Command + Click, or F4
Command + Shift + I, for a quick view in pop-up without openning it in editor

To go to next error/warning:
F2
Command + F1, to display the error details
Alt + Enter, to display recommended actions

To complete a statement (addthe semi-colon, going to the next line, etc):
Command + Shift + Enter

To auto-complete with text match (as opposed to code completion):
Alt + / (upward search), Alt + Shift + / (downward search)

To hide a tool window:
Shift + Escape
Command + window-number

To format the whole file:
Command + Alt + L

To format a selection:
Command + W to make a selection, then Command + Alt + L

To view and search for recent files:
Command + E, and type the first few letters to filter it in the pop-up
Command + Shift + E, for recent changed files

To switch between open files and tool windows:
Ctrl + Tab, or Ctrl + Shift + Tab, while holding Ctrl, repeatedly press Tab to change selection

To find usage of a type/class/method/variable:
Alt + F7

To generate constructor, getter, setter, toString, hashcode, equals, override method, inside editor window:
Ctrl + N
Command + O (select methods to override)
Command + I  (select methods to implement from interfaces)

To attach to a remote debugger: F9
To continue the debugger, skip all breakpoints: F9
To step over (go to next line): F8
To step into: F7
To set/unset a breakpoint: Command + F8
To view breakpoints: Command + Shift + F8

To increase indent of the current line:
Command + W, then Tab.  If pressing Tab without selection, it will just insert a tab at the cursor, which is not what you want unless the cursor is at the beginning of the line.

To decrease indent of the current line:
Shift + Tab.  No need to make a selection

To auto-format the current line:
Command + W, then Command + Alt + L.  Without a selection, it will just auto-indent the whole file

To auto-indent the current line:
Command + W, then Command + Alt + I.

To join next line into the current line:
Ctrl + Shift + J.  Useful when you need to get rid of the next few blank lines.  Command + Y will also delete a line, but you will need to move cursor to the blank line first.

To go to the beginning and end of the file:
Command + fn + Left
Command + fn + right
3 finger move on touch pad won't work

To go to the beginning and end of the screen:
Command + fn + Up
Command + fn + Down

To go to the beginning and end of the line:
Command + Left
Command + Right

To move the content up and down, while keeping the cursor in the current line:
Command + Up
Command + Down
2 fingers move on touch pad

Jump to navigation bar:
fn + Alt + Left

To add a user dictionary to customize spelling check, create a text file, one word per line, and name it choose-a-name.dic. Inside Intellij settings, search "dictionary", and add the directory containing choose-a-name.dic. Intellij will find all dictionary files (by extension .dic) in that directory.

10/05/2012

How to Configure hprof in JBoss AS7

These are the steps to configure hprof profiler in JBoss AS 7.x:

1, Identify the target JVM to profile.  In most cases, it's the standalone server JVM, but it can be other JVM such as domain process controller, host controller, or individual server.

2, For standalone server, edit $JBOSS_HOME/bin/standalone.conf, locate JBOSS_MODULES_SYSTEM_PKGS property, and append hprof classes to it, using , (comma) as the package separator.  This property tells JBoss Modules to make hprof classes accessible from any class loaders.

JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,sun.tools.hprof,com.sun.tools.hat.internal.parser,com.sun.demo.jvmti.hprof"

3, Still in standalone.conf file, uncomment JAVA_OPTS property, and append hprof options:

JAVA_OPTS="$JAVA_OPTS -agentlib:hprof=file=log.txt,thread=y,depth=3"

or using the old non-standard -X option:

JAVA_OPTS="$JAVA_OPTS -Xrunhprof:file=log.txt,thread=y,depth=3"

4, Start the standalone server, and the hprof data file (log.txt) will be created in the current directory.

One common error when enabling hprof in JBoss AS 7.x is a series of NoClassDefFoundError, which indicates JBOSS_MODULES_SYSTEM_PKGS property has not been properly configured as in step 2.

Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/demo/jvmti/hprof/Tracker
    at org.jboss.logmanager.LogManager$1.run(LogManager.java:65)
    at org.jboss.logmanager.LogManager$1.run(LogManager.java:52)
    at java.security.AccessController.doPrivileged(Native Method)
    at org.jboss.logmanager.LogManager.<init>(LogManager.java:52)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:355)
    at java.lang.Class.newInstance(Class.java:308)
    at java.util.logging.LogManager$1.run(LogManager.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.util.logging.LogManager.<clinit>(LogManager.java:157)
    at org.jboss.modules.Main.main(Main.java:278)
Caused by: java.lang.ClassNotFoundException: com.sun.demo.jvmti.hprof.Tracker from [Module "org.jboss.logmanager:main" from local module loader @1a28362]
    at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)
    at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
    at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)
    ... 14 more
Exception in thread "Thread-2" java.lang.NoClassDefFoundError: Could not initialize class java.util.logging.LogManager
    at java.util.logging.LogManager$Cleaner.run(LogManager.java:211)
Dumping Java heap ... allocation sites ... done.

Another related post: How to Configure hprof in GlassFish 3.x

10/03/2012

How to Create Management and Application Users in JBoss AS7

JBoss AS7 is secured by default, which means you will have to create users before accessing services and components, such as admin console and remote EJB.

 $JBOSS_HOME/bin/add-user.sh, or add-user.bat is the tool for such purpose. By default it runs in interactive mode, and prompt for user name, password, user type, role, realm, etc. It also has a silent mode (-s, or --silent).

For example, to create a management user, which can be used in admin console (http://localhost:9990):

$JBOSS_HOME/bin/add-user.sh -s -u admin -p "abc123***"

To create an application user, which can be used in remote EJB access:

$JBOSS_HOME/bin/add-user.sh -s -u app -p "abc123***" -a -realm ApplicationRealm --role app,user

To view help info:
$JBOSS_HOME/bin/add-user.sh --help

The user data is persisted in properties files in standalone/configuration and domain/configuration directories:

application-roles.properties
application-users.properties
mgmt-users.properties

Management users currently are not associated with any role, hence no mgmt-roles.properties. It would be nice to have role-based administration.