Out of memory Error

[sudha_ramaiah]

Hi,

We are generating 3 different formats of reports PDF,Excel and HTML using the same design (JRXML).We got out of memory error for huge volume of data.Based on teh forum comments we implemented virtualization in our code and we could see the temporory files getting generated.After using virtulaization,we did not get out of memory error for PDF and it works fine for records ~=59000 records with the paginaiton size of 10.But for excel reports,it takes more time than the PDf files and my webserver(apache) gets timed out.We are writing the report content in the form of byte[] in the stream to open the download box.We sometimes get "Caused by: ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error" also.In case of html files also,it is taking more time and the webserver gets timed out and sometimes the JVM crashes.

We tried increasing the permanent size of memeory and that's how we made PDF altealst to work.Our memory setting is -Xms512m -Xmx1028m -XX:MaxPermSize=128m.We tried increasing the time out period of webserver but did not help.

 

Can anyone help me in getting teh Excel and HTML report also in case of huge volume of data?

 

Thanks,

Sudha.

[jmoody]

I have a similar use case, and am experiencing similar results. I'm exporting large reports (~600,000 rows x 15-20 columns) to CSV and Excel. Initially, before using virtualization, it would run out of memory while filling. After taking advantage of the swap file virtualizer (very cool feature), it no longer runs out of memory during the fill stage, it now locks up while exporting.

 

I'm also experiencing what appears to be socket level errors when the export is running.

 

The memory allocated to the process appears to grow substantially during export, the processor use goes to roughly zero, and the page faults jump to about 1200 per second. The system may be thrashing, which I suspect could cause socket errors. At least that's my working hypothesis.

 

So, I suspect the issue for us both is the increased memory use during export to Excel/HTML (which I thought virtualization fixed). Can anyone confirm that and perhaps suggest a solution?

 

Thank you,

John

[christiaan_se]

Hi,

exporting to different formats use other libraries to accomplish this. Depending on how these libraries deal with memory everything is loaded into memory or not. Virtualization only deals with the reports part (filling the report and creating a representation of the report). For instance, the JExcelApi holds all data in memory before creating the excel file. I recently did some modification on the library in order to deal with large quantities, not sure whether this will be included in the library itself.

 

kind regards,

Christiaan

[jmoody]

Well that would explain I think both Sudha's and my problem. I hope your work makes it into the library!

 

Thanks, Christiaan.