There are fundamental issues trying to define Return Values from a Dataset Run object used in a table element within a main report.
I have created a post describing the problem here: http://community.jaspersoft.com/comment/814111
This feature just does not work, you cannot correctly define a return value form the dataset. From my understanding the flow should be as follows:
Sub dataset variables -> (Return Values mapping) -> main report variables
Whereas the current forms are written as though the data flow is from the main report to the dataset, i.e:
Main dataset variable -> (Return Values mapping) -> sub-dataset variable