Out of memory exception

[felipinVzla]

Hi, i'm using the jasperReports 1.2.5 and when fill a report of 40000 records results in a out of memory exception. The report use a DataSource not a DataBase connection. This problem is happening running the fill in my machine. Also this implementation take the rows in segments of 5000, using the methods of JRAbstractBeanDataSource class.

 

I'm look the sample in demo/samples/virtualizer and the problem still.

 

The jdk is 1.5.06 and the tomcat 5.5.17.

 

And i want to know if the exportReport() method of JRAbstractExporter kepp the report in memory to do the export. Because exporting a excel a report of 65000 records don't work and repeat the out of memory exception. To the excel export i'm using JExcelApiExporter class. This problem is happening doing the report request to a server.

 

Thanks...

[teodord]

Hi,

 

Do you mean you have an enhanced custom data source able to fetch records in multiple steps (data pagination) in order to minimize memory consumption?

If you do, that's good. Maybe you could lower the number of records read in a fetch operation to see if this helps. 5000 records at a time might be too much, depending on the size of the records.

 

If you have used a virtualizer, then the problem might not be related to JasperReports itself, but rather to the way the XLS exporting libraries work. They seem to store the generated XLS document in-memory, even we then save it into a file or send it to an output stream.

So we optimized memory consumption in JasperReports by using a report virtualizer, but we cannot do the same for the exporters, because we rely on third-party libraries such as POI or JExcelAPI which do not seem to have similar mechanisms.

 

And then a third issue. It depends on how you deliver the XLS content to the browser, assuming you are in a Web application. Some browsers need to know the length of the binary content they receive beforehand and this is why our XLS servlets output first to an in-memory byte array output stream, just to be able to measure the resulting document and only after that they output to the response output stream. Fortunately, there is a parameter available to turn this kind of buffering off and save up memory.

 

I hope this helps.

Teodor

[lucianc]

Also note that the XLS exportes do keep the entire report in memory when IS_ONE_PAGE_PER_SHEET is not set (and it is unset by default). This does not happen when the flag is set or for other exporters.

 

Regards,

Lucian

[Bosley]

I have the same problem.

 

I'm using POI to export some data from an DB.

 

The problem is that our users (stupid users ) want to extract ~ 400.000 rows of data. I have managed to write in xls file only 10.000 rows and in different workbooks. However, at 200.000 rows Tomcat is throwing an "Out of memory " exception . Tomcat instance can have up to 256 M RAM.

 

I don't want to raise Tomcat virtual memory.

[greghilton]

teodord wrote:

So we optimized memory consumption in JasperReports by using a report virtualizer, but we cannot do the same for the exporters, because we rely on third-party libraries such as POI or JExcelAPI which do not seem to have similar mechanisms.

I was running into the same issue, so I took a look at JRXlsAbstractExporter. I believe I have come up with an approach that optimizes memory consumption even when IS_ONE_PAGE_PER_SHEET is false.

 

The approach basically consists of making two passes over the pages in exportReportToStream. The first pass simply calculates all the xCut values. On the second pass, exportPage is called once per page rather than once at the end. The pre-calculated xCut values are passed to exportPage, and then on to the JRGridLayout constructor. JRGridLayout need only calculate the yCut values for that page. It can use the pre-calculated xCut values. This allows each page to be exported as it is hit rather than all elements on all pages being collected and exported at one time.

 

This seems to work well for me, however, there may be side-effects that I haven't considered. I can provide the modified source code if this is something that interests you.

[felipinVzla]

greghilton wrote:

teodord wrote: So we optimized memory consumption in JasperReports by using a report virtualizer, but we cannot do the same for the exporters, because we rely on third-party libraries such as POI or JExcelAPI which do not seem to have similar mechanisms. I was running into the same issue, so I took a look at JRXlsAbstractExporter. I believe I have come up with an approach that optimizes memory consumption even when IS_ONE_PAGE_PER_SHEET is false.   The approach basically consists of making two passes over the pages in exportReportToStream. The first pass simply calculates all the xCut values. On the second pass, exportPage is called once per page rather than once at the end. The pre-calculated xCut values are passed to exportPage, and then on to the JRGridLayout constructor. JRGridLayout need only calculate the yCut values for that page. It can use the pre-calculated xCut values. This allows each page to be exported as it is hit rather than all elements on all pages being collected and exported at one time.   This seems to work well for me, however, there may be side-effects that I haven't considered. I can provide the modified source code if this is something that interests you.

 

Ok, good, please i like you provide me the modified source code.

 

Many regards... :)

[lucianc]

Greg, this is a very interesting idea. Please upload your code on the patch area of JasperReports so that we can properly track it.

 

Thank you,

Lucian

[greghilton]

Ok, see patch artifact 1873.

 

Regards,

Greg