Historically, we have published many reports from Studio v6.3.1 to our Jasper Server 6.3.0 and everything has worked fine. We have recently upgraded Jasper Server to 6.4.0 and still everything worked fine.
We then installed Studio v6.4.0 on several machines, but none of these can use the "Open in Editor" feature on ANY jrxml within the Repository Editor (although everything else still seems to work fine, e.g. "run report unit" still generates the report perfectly). But when you try "Open in Editor", the "Opening" popup appears for a few milliseconds and immediately disappears, and that's it. Nothing else happens, and no error messages are generated.
When we try Studio 6.3.1 (which is still installed on other machines) on to the same 6.4.0 server and jrxml, then the "Open in Editor" feature works perfectly. So the problem appears to be within Studio 6.4.0 alone.
We have tried "Studio 6.4.0 + Server 6.4.0" with various "JasperReports Library Versions" set in the "JasperReports Server Access Configuration" of Studio - namely 6.3.0, 6.3.1, 6.4.0 (current); "test connection" works, but "Open in Editor" still fails. So it doesn't seem to be related to the library version setting.
Final note: we are working on MS Windows throughout, in case that makes any difference.