Enterprise applications written in the Java language involve complex object relationships and utilize large numbers of objects. Although the Java language automatically manages memory associated with object life cycles, understanding the application usage patterns for objects is important. For more information, see WebSphere and Java tuning tips.
Understanding the effect of garbage collection is necessary to apply these management techniques.
See these sections for more information:
The Java garbage collection bottleneck
Examining Java garbage collection can help you understand how the application is utilizing memory. Because Java provides garbage collection, your application does not need to manage server memory. As a result, Java applications are more robust than applications written in languages that do not provide garbage collection. This robustness applies as long as the application is not over-utilizing objects.
Garbage collection normally consumes from 5% to 20% of total execution time of a properly functioning application. If you do not manage garbage collection, it can have a significant negative impact on application performance, especially when running on symmetric multiprocessing (SMP) server machines.
The i5/OS JVM uses concurrent (asynchronous) garbage collection. This type of garbage collection results in shorter pause times and allows application threads to continue processing requests during the garbage collection cycle.
Garbage collection in the i5/OS JVM is controlled by the heap size settings. The initial heap size is a threshold that triggers new garbage collection cycles. If the initial heap size is 10 MB, for example, then a new collection cycle is triggered as soon as the JVM detects that 10 MB have been allocated since the last collection cycle. Smaller heap sizes result in more frequent garbage collections than larger heap sizes. If the maximum heap size is reached, the garbage collector stops operating asynchronously, and user threads are forced to wait for collection cycles to complete. This situation has a significant negative impact on performance. A maximum heap size of 0 (*NOMAX) assures that garbage collection operates asynchronously at all times. For more information about tuning garbage collection with the JVM heap settings, see Java virtual machine tuning parameters.
You can use garbage collection to evaluate application performance health. Monitoring garbage collection when the server is under a fixed workload can help you determine if the application is creating several short-lived objects and can detect the presence of memory leaks.
You can monitor garbage collection statistics with any of these tools:
The -verbosegc JVM configuration setting
If you specify this setting, garbage collection generates verbose output.
Note: The -verbosegc format is not standardized between different JVMs or release levels.
The Dump Java Virtual Machine (DMPJVM) command
This command dumps JVM information for a specified job.
Heap Analysis Tools for Java(TM)
This tool is a component of the iDoctor for iSeries suite of performance monitoring tools. The Heap Analysis Tools component performs Java application heap analysis and object create profiling (size and identification) over time. This tool is sometimes called Java Watcher or Heap Analyzer. For more information, about iDoctor for iSeries, see iDoctor for iSeries.
Performance Explorer (PEX)
You can use a Performance Explorer (PEX) trace to determine how much CPU is being used by the garbage collector. For detailed instructions, see Tuning Garbage Collection for Java(TM) and WebSphere on iSeries.
To obtain meaningful statistics, run the application under a fixed workload until the application state is steady. It usually takes several minutes to reach a steady state.
Detecting large numbers of short-lived objects
You can also use these tools to monitor JVM object creation:
The DMPJVM (Dump Java Virtual Machine) command
The DMPJVM command dumps information about the JVM for a specified job.
The ANZJVM (Analyze Java Virtual Machine) command
The ANZJVM command collects information about the Java Virtual Machine (JVM) for a specified job. This command is available in i5/OS V5R2 and later.
The best result for the average time between garbage collections is at least 5 to 6 times the average duration of a single garbage collection cycle. If the average time is shorter, the application is spending more than 15% of its time in garbage collection.
If the information indicates a garbage collection bottleneck, there are two ways to clear the bottleneck. The most efficient way to optimize the application is to implement object caches and pools. Use a Java profiler to determine which objects to cache. If you can not optimize the application, you can add memory, processors, and clones. Additional memory allows each clone to maintain a reasonable heap size. Additional processors allow the clones to run in parallel.
Memory leaks in the Java language are a significant contributor to garbage collection bottlenecks. Memory leaks are more damaging than memory overuse, because a memory leak ultimately leads to system instability. Over time, there is typically an increase in paging and garbage collection times. Garbage collection times increase until the heap is too large to fit into memory, paging rates increase, and eventually garbage collections are forced into synchronous mode. As a result, threads that are waiting for memory allocation are stopped. From a client's point of view, the application stops processing requests. Clients might also receive java.lang.OutOfMemoryError exceptions.
Memory leaks occur when an unused object has references that are never freed. Memory leaks most commonly occur in collection classes, such as Hashtable because the table always has a reference to the object, even after real references are deleted.
High workload often causes applications to perform poorly after deployment in the production environment. This is especially true for leaking applications where the high workload accelerates the magnification of the leakage and the heap size grows too large for the garbage collector to manage.
Memory leak testing
The goal of memory leak testing is to magnify numbers. Memory leaks are measured in terms of the amount of bytes or kilobytes that garbage collection cannot collect. The delicate task is to differentiate these amounts between expected sizes of useful and unusable memory. This task is achieved more easily if the numbers are magnified, resulting in larger gaps and easier identification of inconsistencies. The following list contains important conclusions about memory leaks:
Long-running test
Memory leak problems can manifest only after a period of time. Therefore, memory leaks are found easily during long-running tests. Short running tests can lead to false alarms. It is sometimes difficult to know when a memory leak is occurring in the Java language, especially when memory usage has seemingly increased either abruptly or monotonically in a given period of time. The reason it is hard to detect a memory leak is that these kinds of increases can be valid or might be the intention of the developer. You can learn how to differentiate the delayed use of objects from completely unused objects by running applications for a longer period of time. Long-running application testing gives you higher confidence for whether the delayed use of objects is actually occurring.
Repetitive test
In many cases, memory leak problems occur by successive repetitions of the same test case. The goal of memory leak testing is to establish a big gap between unusable memory and used memory in terms of their relative sizes. By repeating the same scenario over and over again, the gap is multiplied in a very progressive way. This testing helps if the number of leaks caused by the execution of a test case is so minimal that it is hardly noticeable in one run.
You can use repetitive tests at the system level or module level. The advantage with modular testing is better control. When a module is designed to keep the private module without creating external side effects such as memory usage, testing for memory leaks is easier. First, the memory usage before running the module is recorded. Then, a fixed set of test cases are run repeatedly. At the end of the test run, the current memory usage is recorded and checked for significant changes.
Concurrency test
Some memory leak problems can occur only when there are several threads running in the application. Unfortunately, synchronization points are very susceptible to memory leaks because of the added complication in the program logic. Careless programming can lead to kept or unreleased references. The incident of memory leaks is often facilitated or accelerated by increased concurrency in the system. The most common way to increase concurrency is to increase the number of clients in the test driver.
Consider the following points when choosing which test cases to use for memory leak testing:
You can use these tools to detect memory leaks:
The DMPJVM (Dump Java Virtual Machine) command
The DMPJVM command dumps information about the JVM for a specified job.
The ANZJVM (Analyze Java Virtual Machine) command
The ANZJVM command collects information about the Java Virtual Machine (JVM) for a specified job. This command is available in i5/OS V5R2 and later.
Heap Analysis Tools for Java(TM)
This tool is a component of the iDoctor for iSeries suite of performance monitoring tools. The Heap Analysis Tools component performs Java application heap analysis and object create profiling (size and identification) over time. This tool is sometimes called Java Watcher or Heap Analyzer.
For best results, follow these guidelines:
The Java heap parameters also influence the behavior of garbage collection. Because a large heap takes longer to fill, the application runs longer before a garbage collection occurs. For more information about heap settings, see Java virtual machine tuning parameters.
Initial heap size
When tuning a production system where the working set size of the Java application is not understood, it is recommended that you set the initial heap size to 96MB per processor. The total heap size in an i5/OS JVM can be approximated as the sum of the amount of live (in use) heap space at the end of the last garbage collection plus the initial heap size.
The illustration represents three CPU profiles, each running a fixed workload with a varying initial Java heap size. In the middle profile, the initial size is set to 128MB. Four garbage collections occur. The total time in garbage collection is about 15% of the total run. When the initial heap size is doubled to 256MB, as in the top profile, the length of the work time increases between garbage collections. Only three garbage collections occur, but the length of each garbage collection is also increased. In the third profile, the heap size is reduced to 64MB and exhibits the opposite effect. With a smaller heap size, both the time between garbage collections and the time for each garbage collection are shorter.
This example shows that the total time in garbage collection is approximately the same in all cases. However, in most cases, setting a smaller initial heap size results in more total time spent in garbage collection, especially if the initial heap size is small compared to the pace of object allocation. If the initial heap size is too small the garbage collector runs almost continuously.
Run a series of test experiments that vary the initial Java heap settings. For example, run experiments with 128MB, 192MB, 256MB, and 320MB. During each experiment, monitor the total memory usage. When all of the runs are finished, compare these statistics:
If the application is not over-utilizing objects and has no memory leaks, the state of steady memory utilization is reached. Garbage collection also occurs less frequently and for short duration.
Note that unlike other JVM implementations, a large amount of heap free space is not generally a concern for the i5/OS JVM.
Maximum heap size
The maximum heap size can affect application performance. This value specifies the maximum amount of object space the garbage collected heap can consume. If the maximum heap size is too small, performance might degrade significantly, or the application might receive out of memory errors when the maximum heap size is reached. Due to the complexity of determining a correct value for the maximum heap size, a value of 0 (meaning there is no size limit) is recommended unless an absolute limit on the object space for the garbage collected heap size is required.
In a situation where an absolute limit for the garbage collected heap is required, the value specified should be large enough so that performance is not negatively affected. To determine an appropriate value, run your application under a heavy load with a maximum heap value of 0. Determine the maximum size of the garbage collected heap for the JVM using DMPJVM or iDoctor. The smallest acceptable value for the maximum heap size is 125 percent of the garbage collected heap size. This value is a reasonable estimate for your garbage collected heap working set size. You can specify a larger value for the maximum heap size without affecting performance, and it is recommended that you set the largest possible value based on the resource restrictions of the JVM or the limitations of the system configuration.
After you determine an appropriate value for the maximum heap size, you might need to set up or adjust the pool in which the JVM runs. By default, WebSphere Application Server - Express jobs run in the base system pool (storage pool 2 as shown by WRKSYSSTS), but you can specify a different pool. The maximum heap size should not be set larger than 125 percent of the size of the pool in which the JVM is running. It is recommended that you run the JVM in its own memory pool with the memory permanently assigned to that pool, if possible.
If the performance adjuster is set to adjust the memory pools (that is, the system value QPFRADJ is set to a value other than 0), it is recommended that you specify a minimum size for the pool using WRKSHRPOOL. The minimum size should be approximately equal to your garbage collected heap working set size. Setting a correct maximum heap size and properly configuring the memory pool can prevent a JVM with a memory leak from consuming system resources, but still offers excellent performance.
When a JVM must run in a shared pool, it is more difficult to determine an appropriate value for the maximum heap size. Other jobs running in the pool can cause the garbage collected heap pages to be aged out of the pool. If the garbage collected heap pages are aged out of the pool, the garbage collector must fault the pages back into the pool on the next garbage collection cycle because it needs to access all of the pages in the garbage collected heap. Because the i5/OS JVM does not stop all of the JVM threads to clean the heap, excessive page faulting causes the garbage collector to slow down and the garbage collected heap to grow. Instead the size of the heap is increased and threads continue to run.
This heap growth is an artificial inflation of the garbage collected heap working set size, and must be considered if you want to specify a maximum heap value. When a small amount of artificial inflation occurs, the garbage collector reduces the size of the heap over time if the space remains unused and the activity in the pool returns to a steady state. However, in a shared pool, you might experience problems if the maximum heap size is set incorrectly
If you want to determine the proper value for the maximum heap size, you must run multiple tests, because the appropriate value is different for each configuration or workload combination. If you want to prevent a run-away JVM, set the maximum heap size larger than you expect the heap to grow, but not so large that it affects the performance of the rest of the machine.
If you must set the maximum heap size to guarantee that the heap size does not exceed a given level, specify an initial heap size that is 80-90% smaller than the maximum heap size.