shared JRSwapFileVirtualizer + maxSize

[DaveMcClellan]

Our application runs multiple Jasper Reports using a Quartz scheduler (under Spring) to start each report - so that we can schedule them. Some of these reports are complex and use one or more sub-reports. I recently had to add a JRSwapFileVirtualizer as one of the reports was running over a dataset that grew large in a hurry and we ran out of heap. I'm using a shared one, so I only have to create it once on app startup. The javadocs for the constructor say:

 

maxSize - the maximum size (in JRVirtualizable objects) of the paged in cache

 

Which I thought would be the number of cached pages not flushed to swap. I set it to 4, twice what the examples showed. It apparently also relates to number of simultaneous contexts which can use the SwapFile, as occasionally a report would fail on fill with the error:

The virtualizer is used by more contexts than its in-memory cache size 4

 

My question is - what is considered a context? A single call to

JasperFillManager.fillReport()

, or each report and sub-report it calls? That will help me set the maxSize, and control the number of simultaneous reports which Quartz will run.

[lucianc]

A virtualization context corresponds to an stand-alone report fill (subreports would use the master virtualization context).

 

The maxSize argument is the number of report pages that will be kept in memory by the virtualizer (if the virtualizer is shared, the maxSize refers to the number of in-memory pages across all report that use the virtualizer). Since a report needs to keep at least one page in memory, you'll need to set a maxSize value that is greater than the maximum number of reports which could share the virtualizer.

 

HTH,

Lucian

[DaveMcClellan]

The maxSize change you gave me's been working perfectly in our webapp running under Tomcat 5.5 on Linux. I set the max size and throttle the total simultaneous reports and we've run hundreds through a size-10 cache.

 

We're now seeing something odd with the cache when the web-app's running on a Windows 2003 server. Java 1.5 (012), same Tomcat 5.5.23, JasperReports 1.3.3, same code of ours I run on Linux. When running under 2003, the contexts don't appear to be releasing the virtualizer / cache after we run the report. I do this sequence:

Code:

   fill template Params Map, including a 
      put(JRParameter.REPORT_VIRTUALIZER, virtualizer)
   JasperPrint prtRpt = JasperFillManager.fillReport(     
       reportTemplate, templateParams, connection );
   for( de : delivery-list ) {
      create exporter of appropriate type (CSV, HTML, PDF, TXT , whatever)
      exporter.setParameter( 
          JRExporterParameter.JASPER_PRINT, prtRpt );
      ... set exporter params
      exporter.exportReport();
   }
   return status;

 

And that works fine on Linux; all the reports share the same cache. On Windows, the first maxSize reports run normally. The maxSize+1 report and all succeeding ones return this error (maxSize == 10 in my app):

The virtualizer is used by more contexts than its in-memory cache size 10

 

Apparently the context isn't being released when my local variable prtRpt goes out of scope, garbage collected, finalize(), or whatever signals the cache that the context isn't using it. On Windows only. Has anyone else heard of this issue? And is there a step I can add to explicitly release the cache from the context when I finish all my exports? Right now I'm working around it by created a JRFileVirtualizer() per report instead of a shared JRSwapFileVirtualizer(). But I'd like to understand why the context isn't being released (or releasing the cache), and I prefer using the shared swap-cache as it's less expensive long term and creates less temp files to clean up.

 

Thanks! - Dave

[rajil]

Hi. I am facing some problem while creating JasperPrint objects using Virtualizer.

 

Actually, i have total 5000 reports to generate. each having 2 pages. I am using Batch Export to generate report in a single file. To do this i am saving all 5000 JasperPrint objects into an Array List. and Export them to a single file.

 

I am running this under Tomcat4.1, JDK 1.4, Jasper 3.0.

 

Now, it consumes lots of memory. Arround 600 MB of HeapSize.

 

I tried using all 3 Virtualizer, but it goes out of memory after processing only 2700 JasperPrint Objects even after setting HeapSize to 1.6 GB.

But without using Virtualize its working fine.

 

Also, I tried in both ways, using seperate Virtualizer Object for all JasperPrints (MaxSize=1) as well as using the same Virtualizer Object for all (MaxSize=5001).

 

But in both cases it goes out of memory.

 

Can you suggest the solution.

[DaveMcClellan]

I use the JRConcurrentSwapFile + JRSwapFileVirtualizer on Linux-hosted copies of the webapp, but on Win 2003 hosts I have to use a per-report JRFileVirtualizer. I never got an answer on why the swap virtualizer is broken on Windows. The JRFileVirtualizer works ok since the application runs one report at a time (sometimes only seconds apart); I don't know that it would work as well in your batch scenario.

[lucianc]

DaveMcClellan wrote:

I use the JRConcurrentSwapFile + JRSwapFileVirtualizer on Linux-hosted copies of the webapp, but on Win 2003 hosts I have to use a per-report JRFileVirtualizer. I never got an answer on why the swap virtualizer is broken on Windows. The JRFileVirtualizer works ok since the application runs one report at a time (sometimes only seconds apart); I don't know that it would work as well in your batch scenario.

 

Sorry for not answering your question (for some reason I missed it or it got lost in my inbox).

 

I think that what causes the difference between Linux and Windows is the garbage collector strategy. As you know, objects do not get garbage collected immediately after they get out of scope (i.e. are not longer referenced), and there is no way to guarantee that one or more object will get garbage collected at a certain moment. What seems to happen in your Windows JVMs is that the first maxSize reports do not get garbage collected before the maxSize+1 report fill starts.

 

I think that this can be easily avoided by using a larger virtualizer maxSize value. 10 is a rather drastic value; the JR virtualization sample sets maxSize to 2 for demonstration purposes only. For a shared virtualizer deployed in a real word application I would recommend something of the order of hundreds.

 

And is there a step I can add to explicitly release the cache from the context when I finish all my exports?

 

There isn't, but there should be. Please post a bug/feature request for this here.

 

However, a quick solution is to do this yourself by doing this after you're done with a filed report:

Code:

JasperPrint prtRpt = ..fill the report..
.. export the report..

for (Iterator it = prtRpt.getPages().iterator(); it.hasNext(); ) {
  JRVirtualPrintPage page = (JRVirtualPrintPage) it.next();
  virtualizer.deregisterObject(page);
}

 

Regards,

Lucian

[lucianc]

rajil wrote:

Hi. I am facing some problem while creating JasperPrint objects using Virtualizer. Actually, i have total 5000 reports to generate. each having 2 pages. I am using Batch Export to generate report in a single file. To do this i am saving all 5000 JasperPrint objects into an Array List. and Export them to a single file. I am running this under Tomcat4.1, JDK 1.4, Jasper 3.0. Now, it consumes lots of memory. Arround 600 MB of HeapSize. I tried using all 3 Virtualizer, but it goes out of memory after processing only 2700 JasperPrint Objects even after setting HeapSize to 1.6 GB. But without using Virtualize its working fine. Also, I tried in both ways, using seperate Virtualizer Object for all JasperPrints (MaxSize=1) as well as using the same Virtualizer Object for all (MaxSize=5001). But in both cases it goes out of memory. Can you suggest the solution.

 

Report virtualizers were designed to handle very large reports, and do not scale well when used for a large number of very small reports.

 

What I don't understand is why you want to use virtualizers, since you say that without virtualizers it works fine. I'm missing the point here..

 

Regards,

Lucian

[rajil]

Hi. Thanks your reply.

 

I want to use virtualizer because i am using BATCH_EXPORT feature of Jasper.

 

I have lots of report to be exported in a single file. And there is only way to generate single file using multiple Jasper Print objects is to use BATCH_EXPORT.

 

Now. Without using virtualizer, it works fine. But it consumes lots of memory due to List of Jasper Print objects of each report resides in memory in order to use BATCH_EXPORT.

 

So. it would be better to use virtualizer using smaller amount of HeapSize to generate single file of Multiple JasperPrint objects using BATCH_EXPORT.

Or any other suggestion to generate single file from multiple JasperPrint object would also be appreciated.

 

But what i don't understand is why it goes out of memory if I use virtualizer.

[lucianc]

As I said before, JasperReports virtualizers do not scale well for many small reports. This happens because virtualizers (currently) keep at least one page per report in memory. If you think that this is a problem, please log a bug for it.

 

What I would suggest you to do is to use a kind of "virtualized" JasperPrint list implementation. This list would automatically and transparently serialize to temporary files JasperPrint objects added to it, and would deserialize them when the objects are retrieved from the list. This way, you would not keep the JasperPrint objects in memory, and you would be able to pass the list to the JasperReports exporters.

 

The list implementation would do something like:

 

Code:

package jr;

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;

import net.sf.jasperreports.engine.JRException;
import net.sf.jasperreports.engine.util.JRLoader;
import net.sf.jasperreports.engine.util.JRSaver;

public class SerializedList extends ArrayList {

	private static final String TEMP_FILE_PREFIX = "JasperPrint";
	
	public boolean add(Object e) {
		File tempFile = serializeToTempFile(e);
		return super.add(tempFile);
	}

	public Object get(int index) {
		Object e = super.get(index);
		if (!(e instanceof File)) {
			throw new RuntimeException("Found a non-File element in the list"«»);
		}
		File tempFile = (File) e;
		Object o = deserializeFromTempFile(tempFile);
		return o;
	}

	public void dispose() {
		for (int idx = 0; idx < size(); ++idx) {
			Object e = super.get(idx);
			if (e instanceof File) {
				File temp = (File) e;
				temp.delete();
			}
		}
	}
	
	protected File serializeToTempFile(Object o) {
		try {
			File tempFile = File.createTempFile(TEMP_FILE_PREFIX, null);
			JRSaver.saveObject(o, tempFile);
			return tempFile;
		} catch (IOException e) {
			throw new RuntimeException(e);
		} catch (JRException e) {
			throw new RuntimeException(e);
		}
	}
	
	protected Object deserializeFromTempFile(File temp) {
		try {
			Object o = JRLoader.loadObject(temp);
			return o;
		} catch (JRException e) {
			throw new RuntimeException(e);
		}
	}
	
}

 

Then, you would use it like this:

Code:

[code] SerializedList jasperPrintList = new SerializedList(); ... JasperPrint print = ..; jasperPrintList.add(print); ... exporter.setParameter(JRExporterParameter.JASPER_PRINT_LIST, jasperPrintList); ... jasperPrintList.dispose();

 

Regards,

Lucian

[rajil]

Well, Manual Serialization and De-serialization can be done.

-

But I found another problem. I am able to create and maintain JasperPrint Objects in memory and storing them in JasperPrintList. But BATCH EXPORT goes out of memory while exporting them.

-

There are around 1000+ jasper print objects and 9000+ pages. I can process even more pages and store them in List. But while BATCH EXPORT it goes out of memory. Is then any solution for this now?

-

I have tried following with 540 MB HeapSize (results of report run)

-

Result of 1st Run:

Number of Pages : 7456

Total Time Taken : 9 Minutes (6 Mins to Query + 3 Minutes to Export)

Generated File Size : 23.3 MB

Average Database CPU Usage : < 10%

Average Application CPU Usage: 40-60 %

-

Result of 2nd Run:

Number of Pages : 8000 (Number of Jasper Print Objects in memory = 952 )

Total Time Taken : 64 Minutes (9 Mins to Query + 56 Minutes to Export)

Generated File Size : 26.5 MB

Average Database CPU Usage : < 10%

Average Application CPU Usage: 40-60 %

-

More than 8000 Pages it goes out of Memory while Batch Export.

-

Also if I serialize the JasperPrint Objects. I will have to de-serialize them back to Print them using BatchExport in a List. But the BatchExport is not able to handle this many objects.

[lucianc]

rajil wrote:

But BATCH EXPORT goes out of memory while exporting them.

 

Which exporter are you using? Some exporters, such as the XLS ones, are known not to be scalable because they keep the entire resulting document in memory. Other exporters, such as the HTML or PDF ones, stream the output so there shouldn't be any memory problems (unless you export to an in-memory stream).

 

Also if I serialize the JasperPrint Objects. I will have to de-serialize them back to Print them using BatchExport in a List. But the BatchExport is not able to handle this many objects.

 

Are you using the approach I described in my previous post? In that case, you would not deserialize all the JasperPrint objects at the same time, but only one at a time.

 

Regards,

Lucian

[rajil]

Hi lucianc, Thank you for this solution.

-

It works fine. I generated report arround 40,000 pages with small amount of memory usage and the file size is arround 130 MB.

-

Thanks again.

[DaveMcClellan]

Lucian -

This week we upgraded from JR 1.33 to JR 3.0 (as part of a larger upgrade from Java 1.42 / Tomcat 5.0 to Java 1.5 / Tomcat 5.5). Most of our reports are running ok under the new setup, but the large (several thousand report) CSV-export report is failing with an OutOfMemory error at export time. I can run this same report to completion on the old server, using a 256Mb heap, but get an OutOfMemory error in the JRCsvExporter / JRGridLayout class in a 512Mb heap on the new server. I'm still using the JRSwapFileVirtualizer, so the report fills ok - it just doesn't export. Traceback is:

 

Code:

[2008-07-12 09:18:45,874] ERROR - JasperReportRunnerBS.runReport(705) | runReport(req,sched) : Trapped Exception class java.lang.OutOfMemoryError : Java heap space
java.lang.OutOfMemoryError: Java heap space
        at java.lang.String.<init>(String.java:208)
        at java.lang.StringBuffer.toString(StringBuffer.java:586)
        at net.sf.jasperreports.engine.export.JRGridLayout.createWrappers(JRGridLayout.java:924)
        at net.sf.jasperreports.engine.export.JRGridLayout.<init>(JRGridLayout.java:144)
        at net.sf.jasperreports.engine.export.JRCsvExporter.exportPage(JRCsvExporter.java:274)
        at net.sf.jasperreports.engine.export.JRCsvExporter.exportReportToWriter(JRCsvExporter.java:260)
        at net.sf.jasperreports.engine.export.JRCsvExporter.exportReport(JRCsvExporter.java:207)
        at com.ateb.obc.service.impl.JasperReportRunnerBS.fillAndDeliverRptTemplate(JasperReportRunnerBS.java:1489)

 

Did memory management change between 1.33, 2.05 and 3.0? You mentioned that some of the exporters try to load a whole page at a time to export, which would be bad for this line-oriented data set if the CSV exporter does so. Is there anything I can do other than going back to 1.33?

 

Thanks - Dave McClellan

[DaveMcClellan]

After a week and a half of running under the new environment (on Linux Java 5 / Tomcat 5.5), it appears that some report templates cause the same problem I reported earlier with the Windows JVM  - they do not release the JRSwapFileVirtualizer. The old server had been running for the better part of a year with a limit of 10 contexts. The same set  of reports running on the new server, with the maxSize moved up to 25, got the following error today, after running for only about 10 days:

[2008-07-21 12:30:00,188] ERROR - JasperReportRunnerBS.fillAndDeliverRptTemplate(1045) | Exception running report QNTXtrSiteFax: The virtualizer is used by more contexts than its in-memory cache size 25

Once the server gets this exception, no report afterwards will run. My guess is that some reports with some type of element (subreport, grouping, etc.) or some type of export (PDF, CSV,  Text, TIFF via JRGraphics2DExporter) result in a context not being released - even after garbage collection. This was not a problem in JR 1.3.3 / Java 1.4.2.  

It appears I'm going to have to use a JRFileVirtualizer for all of my virtualization. 

Note: I have to run the app I mentioned in my last post, with the large CSV export, in a special Java 1.4.2 / JasperReports 1.3.3 container.  It will not run in JR 2.05 or 3.0.0, and JR 1.3.3 cannot be run under Java 5 without getting the Perm Gen memory overflow.

[DaveMcClellan]

I've worked around the problem with the large (40,000+ rows) CSV export. Looking at the code for the JRCsvExporter, I can see it loads a full page at a time to export, rather than one row. iReport creates templates with the column names in the "columnheader" band and fields in the "detail" band. So originally I had resized all other bands to 0 after removing their elements, removed the margins and set "Ingore Pagination" in the report properties so that the "columnheader" would be output once - resulting in one very large page. The CSV exporter code doesn't appear to have changed significantly between 1.3.3 and 3.0.0, so the 1.42 JVM and 1.3.3 JR must have used less heap than the 1.5 JVM and 3.0.0 JR.

I've changed the template by re-enabling pagination, moving the CSV "column headers" out of the "columnheader" band and into the "title" band, and resizing the "columnheader" band to 0. This results in no visible page breaks, a single copy of the column headers in the first row of output, and all of the detail row rows being exported as they were before.

Are there plans to update JRCsvExporter to be less memory intensive for large-page reports (e.g., load the JRPrint a row at a time rather than a page?)

[lucian]

Are there plans to update JRCsvExporter to be less memory intensive for large-page reports (e.g., load the JRPrint a row at a time rather than a page?)

We are aware of this problem, but we don't currently have any feasible ideas on how to address it.  Large pages (usually obtained by filling reports with pagination ignored) do not scale not only when exporting (to CSV for instance), but also at fill time since virtualization works per report pages (and hence it is not effective when pagination is not performed).

Please post a bug on this here, so that we'll track this issue properly.

Regards,

Lucian

[lucianc]

 

After a week and a half of running under the new environment (on Linux Java 5 / Tomcat 5.5), it appears that some report templates cause the same problem I reported earlier with the Windows JVM  - they do not release the JRSwapFileVirtualizer. The old server had been running for the better part of a year with a limit of 10 contexts. The same set  of reports running on the new server, with the maxSize moved up to 25, got the following error today, after running for only about 10 days: [2008-07-21 12:30:00,188] ERROR - JasperReportRunnerBS.fillAndDeliverRptTemplate(1045) | Exception running report QNTXtrSiteFax: The virtualizer is used by more contexts than its in-memory cache size 25 Once the server gets this exception, no report afterwards will run. My guess is that some reports with some type of element (subreport, grouping, etc.) or some type of export (PDF, CSV,  Text, TIFF via JRGraphics2DExporter) result in a context not being released - even after garbage collection. This was not a problem in JR 1.3.3 / Java 1.4.2.

Please post this as well as a bug.  I don't think we'll be able to identify the cause without reproducing the issue, so please include any information that might help reproducing it.

A workaround is to explicitly dispose each JasperPrint object after you're done with it, as I explained above.

Regards,

Lucian

PS: Investigating why the CSV export's memory has increased significantly in newer versions is worthwhile as well, please report this too.

Post Edited by Lucian Chiriţă at 07/24/08 14:31

[rajil]

Hi;

Is there any solution to the previous error?  Is this a bug and reported?  I also am getting following

 -----------------

error:net.sf.jasperreports.engine.JRRuntimeException: The virtualizer is used by more contexts than its in-memory cache size 100
 at net.sf.jasperreports.engine.fill.JRAbstractLRUVirtualizer$Cache$LRUScanMap.removeLRU(JRAbstractLRUVirtualizer.java(Compiled Code))
 at org.apache.commons.collections.LRUMap.put(Unknown Source)
 at net.sf.jasperreports.engine.fill.JRAbstractLRUVirtualizer$Cache.put(JRAbstractLRUVirtualizer.java(Inlined Compiled Code))
 at net.sf.jasperreports.engine.fill.JRAbstractLRUVirtualizer.registerObject(JRAbstractLRUVirtualizer.java(Compiled Code))
 at net.sf.jasperreports.engine.base.JRVirtualPrintPage.<init>(JRVirtualPrintPage.java(Inlined Compiled Code))
 at net.sf.jasperreports.engine.fill.JRBaseFiller.newPage(JRBaseFiller.java(Inlined Compiled Code))
 at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReportStart(JRVerticalFiller.java(Inlined Compiled Code))
 at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReport(JRVerticalFiller.java(Compiled Code))
 at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java(Compiled Code))
 at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java(Compiled Code))
 at net.sf.jasperreports.engine.fill.JRFiller.fillReport(JRFiller.java(Compiled Code))
 at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java(Inlined Compiled Code))
 at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java(Compiled Code))
 

--------------

This problem is found under Windows 2003, WebSphere 6 Network Eddition, JDK 1.4.2 and JasperReports 3.0

Same problem is found under Tomcat4.1 with above all configuration. 

I didn't find any post related to above error except this and is there any solution to above Exception??

[Teodor Danciu]

Hi,

 

I don't think the virtualizer problem was logged as a bug yet. Please create a bug tracker here:

http://jasperforge.org/projects/jasperreports/tracker

but please make sure you provide sample files that would enable us to reproduce the problem in order to solve it.

 

Thank you,

Teodor

 

[rajil]

Hi;

 

Actually these problems reported at client side and i am not able to re-produce the same on our side. I am able to run the same code successfully at my side but 2 of our clients facing problem.

 

Let me re-produce the problem here and will report bug then-after with proper data and source.

 

Thanks for the reply.