Does the integration of CAS with Jasper works with selfsigned certificate?

I do not think it will, Java and new broswers are pretty picky about not allowing self signed certs. Sometimes you can get around this by adding your cert to the keysotres of both Jasper and your CAS server, see https://wiki.jasig.org/display/casum/ssl+troubleshooting+and+reference+g... and http://community.jaspersoft.com/documentation/tibco-jasperreports-server...

In any case using self signed certs in production is not a good practice :)