Are you a sysadmin who is new to Java? Then you might find this post to be helpful.

Java has its own memory management system with garbage collection which is most of the time really nice, but you need to know some details how it works so you can administrate your JVM instances effectively.

How Java manages memory?

At the beginning Java JVM will allocate a block of memory from the OS to its heap which it will distribute to the program running inside the JVM. The amount is controlled with two command line arguments: -Xms tells how much memory JVM will allocate at the start and -Xmx what’s the maximum amount of memory which JVM can allocate from the OS. For example -Xms512m -Xmx1G will tell java to start with half gigabyte at the beginning and allow it to grow to one gigabyte.

As the Java program runs it allocates memory for the objects from the JVM heap. This will result the heap to grow until a GC (garbage collection) threshold is  reached. This will trigger the JVM to see which objects are no longer used (objects which are not referenced by any working object) and it will free this memory back to the heap. There are numerous ways how this can work in different GC implementations (Java has many of them) and they’re out of the scope of this article. The main point is that the Java heap usage will grow until about 80% usage when the GC occurs and then drop back to much lower level. If you use jconsole to watch the free memory you will see something like this:

The saw tooth like pattern is just normal life Java garbage collection and nothing to worry about. This however will make it difficult to know how much memory the program actually uses needs.

What happens when Java runs out of memory?

If JVM can’t free enough memory with a simple GC it will run a Full GC which will be a stop-the-world collection. This suspends the JVM execution until the collection is done.  A Full GC can be seen as a sudden drop on the amount of used memory, for example as seen in this image. The Full GC in this case took 0.8 seconds. It’s not much, but it did suspend the program execution for that time, so keep that in mind when designing your Java software and its real time requirements.

The Full GC will be able to free enough memory so that the program will continue, but if the JVM simply has not enough memory it will need to trigger another Full GC shortly. This can result in a GC storm where the program spends even more time doing even longer Full GC’s and finally running out of memory.  It’s not uncommon to see Full GC taking over two minutes in these situations and remember, the program is suspended during a Full GC! No need to say that this is bad, right?

However giving JVM too much memory is also bad. This will make the JVM happy as it doesn’t need to do Full GCs, but then the small GCs can take longer and if you eventually run into a Full GC situation it will take long. Very long. Thus you need to think how much memory your program will need and setup the JVM -Xms and -Xmx so that it has enough plus additional “gc breathing room” on top of that.

How the OS sees all this?

When the JVM starts and allocates the amount of memory specified in -Xms the OS will not immediately allocate all this memory but thanks to the modern virtual memory management the OS reserves this to be used later. You can see this in the VIRT column in top. Once the Java program starts to actually use this memory the OS will need to provide the memory and thus you can see the program RES column value to grow. VIRT means how much virtual memory has been allocated and mapped to the process (this includes the JVM heap memory plus JVM code and other libraries) and RES means how much memory from all VIRT memory is actually in the ram.

The Java in the image above has too big amount of memory. The program was started with 384MB heap (-Xms384m) but it was allowed to grow up to one gigabyte (-Xmx1G), but the program is actually using just 161 megabytes out of it.

However when the JVM GC runs and frees the memory back to the application, the memory is not given back to the OS. Thus you will see the RES value to grow up to VIRT, but never to actually decrease unless the OS chooses to swap some of the JVM memory out to disk. This can happen easily if you specify too big heap to the JVM which doesn’t get used and you should try to avoid this.

Top Tip for Top: You can press f to add and hide columns like SWAP. Notice that SWAP isn’t actually the amount of memory which has been swapped to disk. According to top manual: VIRT = SWAP + RES. Swap contains both the pages which has been swapped to disk and pages which hasn’t yet been actually used. See for more very usefull top commands by pressing ?.

How can I monitor all this?

The best way is to use JMX with some handy tool like jconsole. JConsole is a GUI utility which comes with all JDK distributions and can be found under the bin/ directory (jconsole.exe in windows). You can use the jconsole to connect into a running JVM and extract a lot of different metrics out of it and even tweak some settings on the fly.

JMX needs to be enabled, which can be done by adding these arguments to the JVM command line:

-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8892 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote=true

Notice that these settings disable authentication and ssl, so you should not do this unless your network is secured from the outside. You can also feed this data into monitor systems like Zabbix (my favourite), Cacti or Nagios, which I have found very helpful when debugging JVM performance.

Other way is to enable GC logging which can be done in Sun JVM with these command line parameters (these are reported to be working also with OpenJDK but I haven’t tested)

-XX:+PrintGCTimeStamps -XX:+PrintGC -Xloggc:/some/dir/cassandra.gc.log

These will print GC statistics to the log file, here’s an actual example:

17500.125: [GC 876226K->710063K(4193024K), 0.0195470 secs]
17569.086: [GC 877871K->711547K(4193024K), 0.0200440 secs]
17641.289: [GC 879355K->713210K(4193024K), 0.0201440 secs]
17712.079: [GC 881018K->714931K(4193024K), 0.0212350 secs]
17736.576: [GC 881557K->882170K(4193024K), 0.0419590 secs]
17736.620: [Full GC 882170K->231044K(4193024K), 0.8055450 secs]
17786.560: [GC 398852K->287047K(4193024K), 0.0244280 secs]

The first number is seconds since JVM startup, the second tells the GC type (normal vs. Full GC) and how much memory was freed.

Conclusion

• Java JVM will eat all the memory which you give to it (this is normal)
• You need to tune the JVM -Xms and -Xmx parameters to give it enough but not too much memory so your application works.
• The memory wont be released back to the OS until JVM exists, but the OS can swap the JVM memory out. Usually this is bad and you need to decrease the memory you give to the JVM.
• Use JMX to monitor the JVM memory usage to find suitable values.