Jump to content
Changes to the Jaspersoft community edition download ×

Using JRXmlDataSource with LOTS of data


2004 IR Help

Recommended Posts

By: Andrey - andfedotov

Using JRXmlDataSource with LOTS of data

2005-07-12 01:33

When I use xml with lots of data as an datasourse for report filling, method JasperFillManager.fillReport takes too much time (about 30 sec per page).

My xml is about 2,5 Mb large and have about 150 000 strings of data.

I examed report filling process using YourKit Java Profiler and found that 73% of report filling time takes method CachedXPathAPI.selectSingleNode that is being called from net.sf.jasperreports.engine.data.JRXmlDataSource.getFieldValue(JRField).

As is described in overview of CachedXPathAPI class "functions of theese class tend to be a little slow, since a number of objects must be created for each evaluation".

So, does anybody know how to avoid this problem and probably to use some other class for getting data from a xml...

Thanks a lot!

P.S. Please answer as soon as it possible cause my project really need the approach for solving this problem...

 

 

 

 

By: Andrey - andfedotov

Re

2005-07-12 02:40

I found this note in JRXmlDataSource:

* Note on performance. Due to the fact that all the XPath expression are interpreted the

* data source performance is not great. For the cases where more speed is required,

* consider implementing a custom data source that directly accesses the Document through the DOM API.

 

In my project I'm dealing with lots of data and use xml as a datasource for filling process. And I have a problem with perfomance of JRXmlDataSouce when xml is 2Mb or more. So, could you give me an advise how to resolve a perfomance problem (and may be you have some patch for this class...).

I'll be grateful to recieve any suggestion from you..

 

Thanks in advance,

Andrey Fedotov

 

 

 

 

By: jorge - sirion_oef

RE: Using JRXmlDataSource with LOTS of data

2005-07-12 14:56

like you read in the note, make a custom data source (lok at the examples) that use a fast xml parser. I recommend you:

 

* tinyxml, it is small, and can be used on movile devices, as the autor says

 

* Jdom, is for dummys (but powerful), is based in java style, an is very very simple.

 

 

 

 

By: Andrey - andfedotov

RE: Using JRXmlDataSource with LOTS of data

2005-07-13 02:31

So, is there's no prepared solution of this problem?..

Sounds strange...

Surely I'll make a patch by myself, but this means that everybody have to do it by their own (and it's when this problem is common known (even the author of a JRXmlDatasource wrote about it in the header))...

Am I wrong? Hope so... and hope to get a ready patch :-)

 

 

 

 

By: jorge - sirion_oef

RE: Using JRXmlDataSource with LOTS of data

2005-07-13 15:30

you can make that object (maybe JRXmlFastDatasource) and put that patch in the patches section. All of us have to put our effort in make jasperreports better day by day, this could be your contribution

 

 

 

 

By: Tbozu - tbozu

RE: Using JRXmlDataSource with LOTS of data

2005-07-13 23:49

Does this means that JRXmlDataSource can support a lot data? coz i am looking at 100,00 ~ 200,000 records at time.i always end up with java.lang.outofmemory error as the datasource object is too big when retrieving records of that amount. With JRXmlDataSource, u only need to specify the xml file and the jasper will retrieve the records one by one instead of loading it all into memory rite?

 

 

 

 

By: Piyush - piyushrules

RE: Using JRXmlDataSource with LOTS of data

2005-07-14 01:05

Hi all,

I'm also facing with the same problem as tbozu. I have 1,00,000+ records in my Oracle database. When I retrieve them and display them directly there is an out of memory error. On the contrary, when i save the output (PDF and HTML) on to the file, i'm able to save it. I was able to open the PDF file using the Acrobat Reader. Thou' I was not able to open the HTML format in the brower, when I checked the .HTML file on a notepad, I found all the data had been retrieved properly. So the problem is with the browser. Is there a work around for this ??

 

 

 

 

By: jorge - sirion_oef

RE: Using JRXmlDataSource with LOTS of data

2005-07-14 15:12

the main topic that andreys begins was: the use of xml datasource with lots of data, not only lots of data.

 

you can watch the response that teoddor gives in other topic about lots of data

 

http://sourceforge.net/forum/forum.php?thread_id=1317715&forum_id=113529

 

 

 

 

By: Darren Davis - ddavis539

RE: Using JRXmlDataSource with LOTS of data

2005-07-14 21:10

I'm producing a simple report and a pretty complicated report using xml files that contain records ranging from 1 ... 1000 for testing purposes. Using the default JRXMLDataSource, I was getting out of memory errors for the file containing 1000 records, and very slow processing for the files containing > 250 records.

 

I spent the past day implementing a custom datasource for Jasper, using the SAX2 implemenatation provided by xerces. I basically created java objects to represent my report data, and use the events to populate these records and stuff them into ArrayList objects.

 

I've done some benchmarking on the new datasource and the results are amazing.

 

For example, to process the 100 record file took 114005 ms for the simple report and 330994 ms for the complicated report with the other datasource.

 

With the new datasource running on the same machine, the simple report took 6448 ms and the complicated one took 41754. in other words from 5 1/2 minutes down to 41 seconds.

 

I'm also not seeing the memory issues anymore, because I can cleanup my list objects between runs manually.

 

The downside is my implementation takes away the flexibility of XML, but for this application the data format is unlikely to change. I'm going to continue to benchmark, but I'm very happy with these results.

 

The custom data source was relatively easy to implement, and the samples that come with Jasper have an easy to follow example.

 

 

 

 

 

 

By: Tbozu - tbozu

RE: Using JRXmlDataSource with LOTS of data

2005-07-19 01:23

can u provide me with the implementation of the custom data source u said Darren?

Link to comment
Share on other sites

  • Replies 0
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...