How often do you get out of heap space errors?

[jsturnbull]

Hello,

We are running JasperServer 3.0 on Tomcat 6.0.18 (Windows service installer) on a 2Gb RAM Windows 2003 server. We experience java.lang.OutOfMemoryError: Java heap space errors approximately every 7-10 days which obviously requires a restart of Tomcat. Our -Xms is set at 7768m and -Xmx is set at 1408m. We have in the region of 100 reports on the server. Usage of the server is not heavy.

I was wondering how this compares with other people's installations. I am looking into tuning the JVM garbage collection but it would be interesting to know what level of performance I should expect from JasperServer. The feeling is that the frequency of forced restarts we are currently experiencing is unacceptably high.

 

Thanks

James

 

 

[developerdude]

We are running a couple dozen reports on JS 3.0 with Tomcat 5.5.27 

 

Usage is not heavy at all - most of our reports are generated once a day then sent out via email or exported to FTP.

 

JAVA_OPTS="$JAVA_OPTS -Xms128m -Xmx512m -XX:PermSize=32m -XX:MaxPermSize=128m -Xss2m "

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled "

 

 

Running on Linux in a VM (VMWare). Sysadmin says VM is using all the mem given to it, but the command line doesn't seem to say that:

 

cat /proc/meminfo

 

MemTotal:      1899860 kB

MemFree:       1286048 kB

 

Anyway, a week or so ago we had an out of heap space memory exception and had to restart Tomcat. Once sysadmin gets back from other tasks and I can see for sure how much headroom I have,  I am wanting to bump up the settings.

 

[carminez]

I was having similar issues with my development environment server which has about the same memory as yours, also running Tomcat 6.0.18 on Windows 2003. It is my development server, so I can't guarantee on how much activity it gets but at times it was pretty decent. We are running Tomcat as a service.

After a lot of playing around, it seemed as though the PermGen was running out at 84MB and the settings weren't taking properly and memory never seemed to get freed up.

In the Tomcat configuration tool, in the Java tab, I ended up adding the following in the Java Options:

-XX:PermSize=128m
-XX:MaxPermSize=512m
-Xss2m
-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled

And then in the 3 textboxes below that we set this...

Initial memory pool: 128 MB
Maximum memory pool: 512 MB
Thread stack size:   2048 KB

Also created a simple jsp page and put it in webappsROOT folder so I could see what the memory usage was. See below code.

Ever since then everything has been running good. I have a bit more memory in my test and production environments, so increased the MaxPermSize and Maximim memory pool settings on those servers.

Code:

<%@ page import="java.lang.management.*, java.util.*" %><%   response.setContentType("text/plain");   Iterator iter = ManagementFactory.getMemoryPoolMXBeans().iterator();   while(iter.hasNext()){       MemoryPoolMXBean item = (MemoryPoolMXBean) iter.next();         MemoryUsage mu = item.getUsage();         long used      = mu.getUsed() / 1024 / 1024;         long committed = mu.getCommitted() / 1024 / 1024;         long max       = mu.getMax() / 1024 / 1024;         %>   MEMORY TYPE: <%=item.getName()%>   Used:        <%=used%>   Committed:   <%=used%>   Max:         <%=max%>   <%}%>  

[developerdude]

It is pretty common (at least in my experience) when taking servlets up and down in Tomcat for it to run out of Permgen memory. Tomcat doesn't do the best job it can when reloading classes, whether it is with regards to hot replacement or Permgen memory.

 

For the most part, once a system is setup properly you don't start and restart servlets enough to cause this problem - so you usually only see it in dev/staging systems, not production.

[anandharaj]

Ya, myself also is looking for Tomcat/Java expert who can give some "tips" on how to tune it. Based on my experience, OS will never show actual memory used by java, this is what make us hard to troubleshoot.

[swood]

Because of JasperReports compilation and dynamic use of bytecode, you need to have a fair amount of PermSpace and you need to tune the JVM parameters for garbage collection.

 

Here are the settings we suggest in the Install Guide:

 

rem standard Sun JDK 1.5 and 1.6 options
set JAVA_OPTS=%JAVA_OPTS% -Xss2m -XX:PermSize=32m -XX:MaxPermSize=128m -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled

rem for Sun JDK 1.5 only
rem set JAVA_OPTS=%JAVA_OPTS% -XX:+CMSPermGenSweepingEnabled
 

Other JVMs (OpenJDK, JRockit, IBM, ...) will have their own settings.

 

Maximum heap size depends the number of simultaneous reports being run  across broswer, web services and scheduled reports. JasperAnalysis and JasperServer Professional ad hoc cache results in the JVM. We are recommending 4GB as a minimum for production use to handle 100 simultaneous reporting requests, and 8GB if you are using JasperAnalysis and/or JasperServer Professional ad hoc. The more CPUs the better, as speed does reduce memory requirements.

 

Sherman

Jaspersoft

 

 

[jsturnbull]

Thanks all for your responses. I have been using jconsole to remotely monitor the heap space. What I have noticed is that very occassionally we get extreme spikes in memory usage. I am talking about going from less than 50Mb to over 1Gb in an instant. When I saw this occurring I was able to prevent out of heap space errors by clicking the Perform GC button on jconsole. There was a period of a couple of minutes where the memory kept jumping up followed by me hitting the GC button. I was wondering if this sort of behaviour could be caused by report execution? Even if a user put in absurd parameters into a report I'd still be amazed if the results required over 1Gb of memory.

[jsturnbull]

The source of our heap space errors appears to be web service calls. We have a PHP website that makes web service calls to run reports. The database is Oracle.

This is the stack trace we always get. It appears to be concerned with removing expired data sources.

02 Dec 2009 19:05:53,104 ERROR ManagementService,http-9080-3:1205 - caught Throwable exception: Java heap space
java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.ArrayList.ensureCapacity(Unknown Source)
    at java.util.ArrayList.add(Unknown Source)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.JdbcReportDataSourceServiceFactory$PooledDataSourcesCache.removeExpired(JdbcReportDataSourceServiceFactory.java:113)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.JdbcReportDataSourceServiceFactory.getPoolDataSource(JdbcReportDataSourceServiceFactory.java:173)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.JdbcReportDataSourceServiceFactory.createService(JdbcReportDataSourceServiceFactory.java:151)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.createDataSourceService(EngineServiceImpl.java:714)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:659)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:360)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.executeReport(EngineServiceImpl.java:792)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.domain.impl.ReportUnitRequest.execute(ReportUnitRequest.java:60)
    at com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.execute(EngineServiceImpl.java:266)
    at com.jaspersoft.jasperserver.ws.axis2.ManagementService.runReport(ManagementService.java:1120)
    at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
    at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
    at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
    at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
    at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
    at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
    at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453)
    at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
    at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.jaspersoft.jasperserver.war.common.UploadMultipartFilter.doFilter(UploadMultipartFilter.java:86)

 

[swood]

 

This could be related to Oracle JDBC drivers.

 

From some research I was doing:

 

http://www.oracle.com/technology/tech/java/sqlj_jdbc/pdf/memory%20management%20aug%202009.pdf

 

"The 7.3.4, 8i, and 9i Oracle JDBC drivers used more or less the minimal amount of memory possible. Unfortunately this resulted in an unacceptable performance penalty. The drivers were rearchitected in 10g to improve performance. Since memory is relatively cheap and typical physical memory sizes have grown dramatically since the 10g drivers were released, most users have benefited from improved performance. Some users, mostly those with very large scale applications, have seen performance problems due to large heap size, garbage collector thrashing, and even OutOfMemoryExceptions."

 

What version of the Oracle drivers are you using?

 

One thing I found with reporting was that JasperReports uses the default fetch size of 25 rows for JDBC. For doing big queries (large #s of rows), this should be upped for better performance. This can be done by putting the following line in <jasperserver>/WEB-INF/classes/jasperreports.properties

 

net.sf.jasperreports.jdbc.fetch.size=X

 

I have set X to be 200 to cover cases with wide result sets (a lot of columns). The thing to watch for is memory usage. As the result set is processed, memory may spike if you have large columns (VARCHAR, BLOB) or a lot of columns. This is true for Oracle 10g- I don't know how other JDBC drivers manage their memory.

 

Sherman

Jaspersoft

[jsturnbull]

Thanks for the advice. We have the latest 11g drivers. We are running reports against 2 versions of Oracle - 9i and 10g. We previously had the 9i drivers, however we get the heap space errors with both driver versions.

[swood]

What version of JasperServer are you running? There were some fixes in the JdbcReportDataSourceServiceFactory class in 3.5 related to clearing expired connections, which is where the problem appears to be.

 

Sherman

Jaspersoft

[jsturnbull]

Version 3. You're going to tell me to upgrade aren't you? /tools/fckeditor/editor/images/smiley/msn/teeth_smile.gif

[swood]

I'd say "you need to upgrade" is almost like saying "have you tried turning it off and on again" http://en.wikiquote.org/wiki/The_IT_Crowd

 

There have been a lot of related changes in this code since version 3, I am afraid.

 

 

Sherman

Jaspersoft

[jsturnbull]

We upgraded from version 3 to 3.7 a while back and I am glad to report that since then we have not experienced any more heap space errors.

[harvys1]

there is a strong problem with JasperServer and configuration of the garbage collector, and generate reports that large quantities of leaves, not free the memory used for this. Not if it's a bug or configuration is correct. I tried changing the virtual machine by modifying the parameters as they say in their various recommendations and I have been unable to release that memory ...