13 instances of one java application launch?

[kapow]

I have a branched LUX appliance that I have configured to run a java application. When I run the java app, ps tells me there are 13 processes with unique ids running all listening on the same port.

Why is that?

[JeremySuo]

Are you seeing these as Java JVM's or is this another process? If you are seeing something like the VMRD or similar process that is communicating with the grid controller to ensure the VM is up and available.

Thanks,

Jeremy

[kapow]

I'm seeing them as Java VM's. It's very strange as they are separate process numbers. Here's a ps after the start:

ps -d
PID TTY TIME CMD
1 ? 00:00:00 init
2 ? 00:00:00 migration/0
3 ? 00:00:00 ksoftirqd/0
4 ? 00:00:00 watchdog/0
5 ? 00:00:00 events/0
6 ? 00:00:00 khelper
7 ? 00:00:00 kthread
8 ? 00:00:00 xenwatch
9 ? 00:00:00 xenbus
18 ? 00:00:00 kblockd/0
45 ? 00:00:00 pdflush
46 ? 00:00:00 pdflush
48 ? 00:00:00 aio/0
658 ? 00:00:00 kseriod
47 ? 00:00:00 kswapd0
756 ? 00:00:00 kjournald
2144 ? 00:00:00 kjournald
2835 ? 00:00:00 vmad
2837 ? 00:00:00 vmad
2838 ? 00:00:00 vmad
2839 ? 00:00:00 vmad
2840 ? 00:00:00 vmad
2841 ? 00:00:00 vmad
2842 ? 00:00:00 vmad
2843 ? 00:00:00 vmad
2844 ? 00:00:00 vmad
2845 ? 00:00:00 vmad
2846 ? 00:00:00 vmad
2847 ? 00:00:00 vmad
2848 ? 00:00:00 vmad
2849 ? 00:00:00 vmad
2850 ? 00:00:00 vmad
2851 ? 00:00:00 vmad
2852 ? 00:00:00 vmad
2853 ? 00:00:00 vmad
2854 ? 00:00:00 vmad
2855 ? 00:00:00 vmad
2872 ? 00:00:00 ccad
2874 ? 00:00:00 ccad
2875 ? 00:00:00 ccad
2876 ? 00:00:00 ccad
2877 ? 00:00:00 ccad
2878 ? 00:00:00 ccad
2879 ? 00:00:00 ccad
2880 ? 00:00:00 ccad
2881 ? 00:00:00 ccad
2882 ? 00:00:00 ccad
2883 ? 00:00:00 ccad
2884 ? 00:00:00 ccad
2885 ? 00:00:00 ccad
2886 ? 00:00:00 ccad
2887 ? 00:00:00 ccad
2888 ? 00:00:00 ccad
2889 ? 00:00:00 ccad
2890 ? 00:00:00 ccad
2891 ? 00:00:00 ccad
2892 ? 00:00:00 ccad
2893 ? 00:00:00 ccad
2894 ? 00:00:00 ccad
2895 ? 00:00:00 ccad
3305 pts/0 00:00:00 su
3306 pts/0 00:00:00 bash
3559 pts/0 00:00:02 java
3620 pts/0 00:00:00 java
3621 pts/0 00:00:00 java
3622 pts/0 00:00:00 java
3623 pts/0 00:00:00 java
3624 pts/0 00:00:00 java
3625 pts/0 00:00:00 java
3626 pts/0 00:00:02 java
3627 pts/0 00:00:01 java
3628 pts/0 00:00:00 java
3629 pts/0 00:00:00 java
3630 pts/0 00:00:00 java
3631 pts/0 00:00:00 java
3633 pts/0 00:00:00 java
3634 pts/0 00:00:00 ps

Here's the details on one of the java processes:

admin 3637 3620 0 09:09 pts/0 00:00:00 /home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/bin/../jre/bin/java -Xmx1024M -XX:MaxPermSize=92m -Duser.country=US -Duser.region=US -Duser.language=en -Dcom.sun.management.jmxremote -Dsun.java2d.noddraw=true -server -Djava.ext.dirs=":/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/jre/lib/ext:/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/lib/patch:/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/lib:/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/Plugins:/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/lib/jdbc:/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/lib/thirdparty:/home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/lib/thirdparty/extra:" com.zerog.lax.LAX /home/admin/Kapow_Mashup_Server_6_3_SR1_Hotfix_6/bin/RoboServer.lax /tmp/env.properties.3559 "-service" "socket:50000"

This, of course, repeats 13 times.

[kapow]

Well, 14 times now. It seems to grow the longer the appliance is running.

And, btw, this is a single line execution of a .bin file. I do not notice this on CentOS 5 based appliances.

[JeremySuo]

With further investigation we have found this currently to be running on a LUX application with a FC3 base. Kapow is going to try and build it now on a CentOS 4.x or CentOS 5.x based LUX and see if that resolves the issue.

3tera any plans for creating a script to backport existing grids that have FC3 based components to a later release of CentOS or Fedora Core so the latest versions of JDK's and other apps can be used without having to upgrade the entire grids.

Thanks,

Jeremy

[PeterNic]

Kapow,

The FC3-based appliance uses an older version of the posix thread library -- linuxthreads (which used to be a the default -- and even only choice -- eons ago). When using linuxthreads, ps shows each thread as if it was a separate process -- which is why you also see multiple "processes" for vma and cca agents (those are single processes with multiple threads).

When you use NPTL threading, ps shows the processes correctly: one process - one entry, regardless of the number of threads in the process.

We do provide the Xen-enabled NPTL support in glibc for the CentOS 5-based appliances.

In any case, what you are observing -- multiple entries in ps for the multiple threads in a process -- is the expected behavior with linuxthreads. This is the *normal* and intended behavior when using the old-style threading library. Are you experiencing any failures or performance problems?

Jeremy, I am not sure I could parse your question; all 2.1.x grids provide LUX5 and LINUX5 appliances. We will probably discontinue the FC3-based LUX/LINUX this year (even though they seem to work great as a base for a variety of small appliances like HLB, IN, OUT, etc.)

Regards,
-- Peter

[kapow]

Great explanation. Thanks.

I upgraded this appliance to LUX5 and I'm seeing expected behavior. I suggest to anyone looking to create a custom appliance to run a java application to start with LUX5 instead of LUX.