Excessive CPU usage

[synchro]

My JasperServer (4.2.1 on Ubuntu Lucid 64) has gone a bit nuts, with the /opt/jasperreports-server-cp-4.2.1/java/bin/java process consuming > 450% CPU. If I stop it and restart it, it carries on doing the same. It started spontaneously at midnight on July 1st (see attached graph).

As far as I can see it's not doing anything different (config has not been touched for > 1 year) - running two small reports perhaps once per day - but there seems to be no way of telling what it's doing since there is no status page or log file. How can I tell what it's doing? Are runaway processes like this common? Is this a known problem?

[synchro]

When trying to run the reports manually, I discovered that they're not working because its failing to open an X11 session. This strikes me as a bug since it's running headless and should have no need for an X11 environment. I don't know why this has suddenly appeared.

java.lang.InternalError: Can't connect to X11 window server using 'localhost:10.0' as the value of the DISPLAY variable.	sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)	sun.awt.X11GraphicsEnvironment.access$100(X11GraphicsEnvironment.java:52)	sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:155)	java.security.AccessController.doPrivileged(Native Method)	sun.awt.X11GraphicsEnvironment.(X11GraphicsEnvironment.java:131) 	java.lang.Class.forName0(Native Method) 	java.lang.Class.forName(Class.java:169) 	java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:68) 	net.sf.jasperreports.engine.util.JRStyledTextParser.(JRStyledTextParser.java:90) 	net.sf.jasperreports.engine.fill.JRBaseFiller.(JRBaseFiller.java:184) 	net.sf.jasperreports.engine.fill.JRVerticalFiller.(JRVerticalFiller.java:78) 	net.sf.jasperreports.engine.fill.JRVerticalFiller.(JRVerticalFiller.java:88) 	net.sf.jasperreports.engine.fill.JRVerticalFiller.(JRVerticalFiller.java:58) 	net.sf.jasperreports.engine.fill.JRFiller.createFiller(JRFiller.java:142) 	net.sf.jasperreports.engine.fill.JRFiller.fillReport(JRFiller.java:114) 	net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java:435) 	com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:773) 	com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:731) 	com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.fillReport(EngineServiceImpl.java:389) 	com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.executeReport(EngineServiceImpl.java:890) 	com.jaspersoft.jasperserver.api.engine.jasperreports.domain.impl.ReportUnitRequest.execute(ReportUnitRequest.java:57) 	com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.EngineServiceImpl.execute(EngineServiceImpl.java:320) 	com.jaspersoft.jasperserver.war.action.ViewReportAction.executeReport(ViewReportAction.java:495) 	com.jaspersoft.jasperserver.war.action.ViewReportAction.executeReport(ViewReportAction.java:485) 	com.jaspersoft.jasperserver.war.action.ViewReportAction.runReport(ViewReportAction.java:318) 	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 	java.lang.reflect.Method.invoke(Method.java:597) 	org.springframework.webflow.action.DispatchMethodInvoker.invoke(DispatchMethodInvoker.java:98) 	org.springframework.webflow.action.MultiAction.doExecute(MultiAction.java:123) 	org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) 	org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) 	org.springframework.webflow.action.EvaluateAction.doExecute(EvaluateAction.java:79) 	org.springframework.webflow.action.AbstractAction.execute(AbstractAction.java:188) 	org.springframework.webflow.execution.AnnotatedAction.execute(AnnotatedAction.java:145) 	org.springframework.webflow.execution.ActionExecutor.execute(ActionExecutor.java:51) 	org.springframework.webflow.engine.ActionState.doEnter(ActionState.java:101) 	org.springframework.webflow.engine.State.enter(State.java:194) 	org.springframework.webflow.engine.Transition.execute(Transition.java:227) 	org.springframework.webflow.engine.impl.FlowExecutionImpl.execute(FlowExecutionImpl.java:391) 	org.springframework.webflow.engine.impl.RequestControlContextImpl.execute(RequestControlContextImpl.java:214) 	org.springframework.webflow.engine.TransitionableState.handleEvent(TransitionableState.java:119) 	org.springframework.webflow.engine.Flow.handleEvent(Flow.java:555) 	org.springframework.webflow.engine.impl.FlowExecutionImpl.handleEvent(FlowExecutionImpl.java:386) 	org.springframework.webflow.engine.impl.RequestControlContextImpl.handleEvent(RequestControlContextImpl.java:210) 	org.springframework.webflow.engine.ViewState.handleEvent(ViewState.java:230) 	org.springframework.webflow.engine.ViewState.resume(ViewState.java:196) 	org.springframework.webflow.engine.Flow.resume(Flow.java:545) 	org.springframework.webflow.engine.impl.FlowExecutionImpl.resume(FlowExecutionImpl.java:259) 	org.springframework.webflow.executor.FlowExecutorImpl.resumeExecution(FlowExecutorImpl.java:163) 	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 	java.lang.reflect.Method.invoke(Method.java:597) 	org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) 	org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) 	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) 	org.springframework.security.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66) 	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) 	org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) 	$Proxy17.resumeExecution(Unknown Source) 	org.springframework.webflow.mvc.servlet.FlowHandlerAdapter.handle(FlowHandlerAdapter.java:183) 	org.springframework.webflow.mvc.servlet.FlowController.handleRequest(FlowController.java:174) 	org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) 	org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875) 	org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807) 	org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571) 	org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:501) 	javax.servlet.http.HttpServlet.service(HttpServlet.java:617) 	javax.servlet.http.HttpServlet.service(HttpServlet.java:717) 	com.jaspersoft.jasperserver.war.common.UploadMultipartFilter.doFilter(UploadMultipartFilter.java:83) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:394) 	com.jaspersoft.jasperserver.war.security.IPadSupportFilter.doFilter(IPadSupportFilter.java:77) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	com.jaspersoft.jasperserver.war.security.JSSwitchUserProcessingFilter.doFilterHttp(JSSwitchUserProcessingFilter.java:154) 	org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109) 	org.springframework.security.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.ui.ExceptionTranslationFilter.doFilterHttp(ExceptionTranslationFilter.java:101) 	org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.providers.anonymous.AnonymousProcessingFilter.doFilterHttp(AnonymousProcessingFilter.java:105) 	org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	com.jaspersoft.jasperserver.api.metadata.user.service.impl.MetadataAuthenticationProcessingFilter.doFilter(MetadataAuthenticationProcessingFilter.java:139) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	com.jaspersoft.jasperserver.war.util.RequestParameterAuthenticationFilter.doFilter(RequestParameterAuthenticationFilter.java:97) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.ui.basicauth.BasicProcessingFilter.doFilterHttp(BasicProcessingFilter.java:174) 	org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	com.jaspersoft.jasperserver.war.UserPreferencesFilter.doFilter(UserPreferencesFilter.java:184) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.ui.AbstractProcessingFilter.doFilterHttp(AbstractProcessingFilter.java:278) 	org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	com.jaspersoft.jasperserver.war.UserPreferencesFilter.doFilter(UserPreferencesFilter.java:184) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	com.jaspersoft.jasperserver.api.logging.filter.BasicLoggingFilter.doFilter(BasicLoggingFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.context.HttpSessionContextIntegrationFilter.doFilterHttp(HttpSessionContextIntegrationFilter.java:235) 	org.springframework.security.ui.SpringSecurityFilter.doFilter(SpringSecurityFilter.java:53) 	org.springframework.security.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:406) 	org.springframework.security.util.FilterChainProxy.doFilter(FilterChainProxy.java:185) 	org.springframework.security.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:99) 	com.jaspersoft.jasperserver.war.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:67) 	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236) 	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

This also doesn't explain the excessive CPU usage as that's happening all the time even when reports are not being generated.

Post Edited by synchro at 07/02/2012 09:06

Post Edited by synchro at 07/02/2012 09:10

[daisy_rmm]

We're seeing the exact same behaviour starting around midnight of July 1st. We initially thought it was a leap-second issue but subsequent restarts did not change the excessive cpu utilisation.

 

[daisy_rmm]

It looks like this is the java leap second bug

following the suggestions in https://bugzilla.mozilla.org/show_bug.cgi?id=769972, java is now down to 2% cpu utilisation.

 

[synchro]

I ended up finding leap seconds was the culprit for this too. Restarting jasper won't fix that; it appears that simply setting the date or rebooting will. More info.