There is no documentation on how to do this.
For example, we have a SOAP data source, QueryExecuter in a JAR and other supporting JARS.
I added the JARs into the MyReports project (Configure Build Path).
I changed the Properties on the MyReports project (MyReports -> Properties -> Jaspersoft Studio group -> Properties -> Use Workspace Settings -> Configure Workspace Settings) and added the QueryExecuter properties ie.
net.sf.jasperreports.query.executer.factory.soap = com.jaspersoft.jasperserver.api.engine.jasperreports.service.impl.SoapQueryExecuterFactory
In the Data Set and Query Dialog, soap is not listed and validation fails.
Using Studio version: Jaspersoft Studio Professional 5.3.0.v201310030846