We are using Japser to generate reports in our application
We have upgraded from 5.0.1 to 6.4 to achive better performance but this is what we are seeing after few test runs.
Jasper version : 6.4
Server : tomcat 7 , 3GB RAM
Number of records : 850k
I think Jaspersoft took back on 6.4.0 because it's no longer available at sourceforge. It may be that there were some problem with it. Try using 6.3.1.
Also, JasperReports 6.x is better with Oracle JDK 8 - try upgrading your jdk.
Are you running JasperReports on Tomcat? If so, that's the problem. Try running from the command line.
Anyways, 3GB for a web app is too small - I've found that JasperReports works better with at least 8GB memory.
You also didn't mention your datasource. Try using csv and don't use something like SpringFramework because it drains memory.