Showstopper Java Problem with -Xmx setting

[Dmitry@Rivermine]

Hi, I'm going to submit a ticket regarding this with our vendor but I wanted to post here if anyone has encountered this before.

I've run into a very very strange problem with a LINUX5 cloned appliance and JVM on 2.2.2-e1688 hf2192 hf2142 hf1960 hf2061 hf2069 hf2081 grid.

Yesterday I found out that on my cloned appliance I couuld not run java 1.5.10 with -Xmx2048M parameter. The system would simply state

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

On "brick and mortar" servers this usually happens when there's not enough RAM in the machine. However, on my virtual appliance there's definitely enough as evident by "free" command. Numbers are in Kb.


total used free shared buffers cached
Mem: 4194444 107016 4087428 0 3752 30708
-/+ buffers/cache: 72556 4121888
Swap: 4194296 0 4194296

So I decided that something might have gone wrong during my clone and customization a while ago. I took out LINUX5 from catalog and instantiated it, gave it enough RAM, installed JVM and voila - runs ok. Next, I decided to clone LINUX5 again and set it up for my needs. I didn't need to install any custom libraries - only install some application servers and apache and Java. Nothing to be compiled to the /usr or /lib, etc.

Right after the work was done the JVM ran fine. However, this morning I'm getting the same error - unable to start JVM. This is after the application has been running for less than 24 hours.

The strace on "./java -Xmx2048m" gives the following output at the end.
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
close(3) = 0
mmap2(NULL, 2243952640, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)
write(1, "Error occurred during initializa"..., 43Error occurred during initialization of VM
) = 43
write(1, "Could not reserve enough space f"..., 46Could not reserve enough space for object heap) = 46
write(1, "\n", 1
) = 1
unlink("/tmp/hsperfdata_root/2084") = 0
write(2, "Could not create the Java virtua"..., 43Could not create the Java virtual machine.
) = 43
exit_group(1) = ?

Not sure if that helps or not. The "-Xmx2048M" settings to JVM tells it to "reserve" 2G of RAM from OS. This is actually a great show stopper for us. Please help!

[Chris]

Sounds like the usual java 2GB memory limit (which includes everything for java). I thought the 2.6 kernel (centos 5) was supposed to have some improvements for this, maybe not though.

Have you tried setting this up on a 64bit version of linux? (Linux64) ?

I would try setting the xmx to 1.5 gigs or so and see if it starts then to narrow down the problem.

[Dmitry@Rivermine]

Chris, would it make any difference that java has ran with 2G initially, then after 10 or so hours would not run again on the same application without any restarts or changes? Why would 64bit appliance make any difference? I will try it though. BTW this is not an issue on our 2.1.0 hf1960 hf2061 hf2069 hf2192 grid which is 32bit.

I'm reading up things on the 2G limit on the internet but most of the posts are several years old. They also state that 2.6 kernel solves this issue.

[Dmitry@Rivermine]

More strangness - after the application was rebuilt the JVM runs fine. I'm going to try to replicate this one an app that has problems with JVM but cleaning it and rebuilding volumes.

[Dmitry@Rivermine]

Another thing I found is that the actual value for -Xmx varies from appliance to appliance even if each appliance has 4G RAM allocated. Currently on one the setting -Xmx2658M breaks JVM, on another -Xmx2414M. Go figure...

[Dmitry@Rivermine]

Now, I have tried this with a 2G memory allocation to the appliance. Same thing. I can't get JVM to run with -Xmx2048M setting. But this works for "brick and mortar" servers. Go figure.

[Dmitry@Rivermine]

Here is an example of how 2 machines with relatively same configuration
are behaving differently.
First is the test machine provided to us.

[root@localhost bin]# uname -a
Linux localhost.localdomain 2.6.9-55.ELsmp #1 SMP Wed May 2 14:28:44 EDT
2007 i686 athlon i386 GNU/Linux
[root@localhost bin]# free
total used free shared buffers
cached
Mem: 4146496 513200 3633296 0 51432
400968
-/+ buffers/cache: 60800 4085696
Swap: 2040244 0 2040244
[root@localhost bin]# /opt/java/bin/java -Xmx2048M
Usage: java [-options] class [args...]
(to execute a class)
or java [-options] -jar jarfile [args...]
(to execute a jar file)

The JVM setting at which it doesn't run is -Xmx2645M

Now is the virtual machine called "xxxx" on our 64bit grid.


[root@xxxx bin]# uname -a
Linux xxxx 2.6.16.33-xenU #2 SMP Thu Aug 2 18:09:26
PDT 2007 i686 athlon i386 GNU/Linux
[root@xxxx bin]# free
total used free shared buffers
cached
Mem: 4194460 267496 3926964 0 40208
135980
-/+ buffers/cache: 91308 4103152
Swap: 0 0 0
[root@xxxx bin]# /opt/java/bin/java -Xmx2048M
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

The setting at which JVM doesn't run is -Xmx1569M

That's 1G difference and it's a big difference.

[Dmitry@Rivermine]

What looks like is going to work is 64bit appliance and 64bit version of JVM. Looks like it allows us to allocate memory as we need. I'll be converting all our custom appliances to 64bit ASAP. At this point the 32bit appliances on the 64bit grid are considered by me to be unusable for JVM.

[Chris]

Running on a 64bit OS should definitely fix this as it doesn't have the allocation limits that 32bit does. Curious to know how it works out, let me know once you get it all converted.

[LeoKalev]

One possible explanation of the difference between the "brick and mortar" behavior versus virtual appliance:
Your VM has no swap configured, while typical bare-metal installs have a swap file set up. Note that in the absence of a swap file, the Linux kernel will refuse exceedingly large memory allocations, even if there is actual RAM to fulfill them. This behavior can be tuned by modifying the virtual memory settings (in /proc/sys/vm). Alternatively, you can configure a small swap volume for your appliance.

[PeterNic]

BTW, an update

This was resolved within days with the customer off-forum. The conclusion was that the problem was in the JVM; the customer has moved to a 64-bit JVM where this problem does not exist.

-- Peter