[#11896] - AWS Multi-tenant AMI broken

Deployed TIBCO Jaspersoft for AWS with Multi-Tenancy (Hourly) on AWS in ap-southeast-2 region in a clustered configuration with an autoscaling group and an external database. (ami-0d4c85dc753fd469a)

The server would not operated properly after boot, responding with 403 error codes. The application was not able to initialise the spring context successfully in Tomcat. Encountered the following 3 issues:

Issue 1: Unable to auto-detect AWS settings for the SQS message queue
The error being logged is:

"Cannot resolve reference to bean 'sqsEndpoint' while setting bean property 'endpoint'; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 7): Property or field 'region' cannot be found on object of type 'com.jaspersoft.ji.replication.discovery.BrokerInfo' - maybe not public?"

This error comes from this spring config: /var/lib/tomcat8/webapps/jasperserver-pro/WEB-INF/applicationContext-jms.xml which is one of the modified files copied over by the jrs_init.sh when clusterenabled is defined by this line:

/bin/cp -fR /usr/share/jrs_dist/clusterConfig/* /var/lib/tomcat8/webapps/jasperserver-pro/WEB-INF/

the bean in the file that is the problem is this:

<beans:bean id="sqsEndpoint" class="java.lang.String">
<beans:constructor-arg value="#{'sqs.'.concat(AutodetectBrokerInfo.brokerInfo.region)}"/>
</beans:bean>

<beans:constructor-arg value="#{'sqs.'.concat(AutodetectBrokerInfo.brokerInfo.region)}"/

The AutodetectBrokerInfo bean is defined in the same file here:

<beans:bean name="AutodetectBrokerInfo" class="com.jaspersoft.ji.replication.discovery.AwsAutodetectBrokerInfo" init-method="init">
<beans:property name="fallBackBrokerInfo" ref="brokerDescription"/>
<beans:property name="port" value="61616"/>
<beans:property name="scheme" value="tcp"/>
<beans:property name="awsEc2MetadataClient" ref="awsEc2MetadataClient2"/>
</beans:bean>

I believe this class is actually broken but without source code cannot confirm.

I have worked around this by hardcoding the SQS endpoint for my region by replacing the bean definition with this:

<beans:bean id="sqsEndpoint" class="java.lang.String">
<!--<beans:constructor-arg value="#{'sqs.'.concat(AutodetectBrokerInfo.brokerInfo.region)}"/> -->
<beans:constructor-arg value="sqs.ap-southeast-2.amazonaws.com"/>
</beans:bean>

This enabled me to fall victim to issue 2

Issue 2: Permissions on tomcat directory
Once fixing the above and restarting tomcat, many errors where being logged about being unable to create directories, eg:

2019-06-09 12:17:00,770 ERROR BrokerService,localhost-startStop-1:624 - Failed to start Apache ActiveMQ ([localhost, null], java.lang.RuntimeException: java.io.IOException: Failed to create directory 'activemq-data/localhost/KahaDB')

Another example:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'accessService' defined in ServletContext resource [/WEB-INF/applicationContext-events-logging.xml]: Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'broker-1': Invocation of init method failed; nested exception is java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Failed to create directory 'activemq-data/localhost/KahaDB'

I resolved these errors by changing the permission on /usr/share/tomcat8 from owner root, group tomcat to owner tomcat, group tomcat

Now the server starts and seems to be working fine

However on startup this error is still logged - issue 3

Issue 3:

java.net.URISyntaxException: Expected scheme-specific part at index 4: tcp:
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.failExpecting(URI.java:2854)
at java.net.URI$Parser.parse(URI.java:3057)
at java.net.URI.<init>(URI.java:673)
at com.jaspersoft.ji.replication.discovery.AwsAutodetectBrokerInfo.init(AwsAutodetectBrokerInfo.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1640)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1581)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.config.BeanExpressionContext.getObject(BeanExpressionContext.java:56)
at org.springframework.context.expression.BeanExpressionContextAccessor.read(BeanExpressionContextAccessor.java:40)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:199)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:85)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:78)
at org.springframework.expression.spel.ast.CompoundExpression.getValueRef(CompoundExpression.java:49)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:82)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:93)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:89)
etc etc etc..

I believe this is related to Issue 1, with that class being broken

I am working around these errors by using a user-data script to apply my fixes on boot, but it would be better to have a working AMI (unless I am missing something and there is something wrong with my setup but I have tried configuring myself and using the cloudformation templates)