This issue was addressed several times: http://community.jaspersoft.com/jasperreports-library/issues/4594 , http://community.jaspersoft.com/questions/530836/ignore-pagination-and-virtualizers
However, I can not see a solution in the discussions.
Did you try to use band filters? With them, you could achive the same xls-view without pagination:
But I do not know how it works in conjunction with file virtualizer (but i don't think that should make problems)