JaspersoftETL v6.0.1 Change Log

Spark Job designer

1. The Spark Streaming framework is supported in Talend Real-time Big Data Platform and Talend Data Fabric. With these products, users can create Spark Streaming Jobs to handle data being generated in motion.

   • New messaging components such as tKafkaInput, tKafkaOutput, tKinesisInput, and tKinesisOutput, are available to the Spark Streaming Jobs.

   • The Spark window operations and the Spark checkpoints are supported by the Spark Streaming Jobs.

   • The tRestWebServiceOutput is added to allow users to send message streams to a given RESTful Webservice in a Spark Streaming Job.

   • In Talend Administration Center, the Big Data Streaming Conductor module has been created to gather the script generation, deployment and execution phases of Streaming Jobs in an execution task.

2. A wide range of component families are available in Spark Batch and Spark Streaming Jobs for accomplishing many different specific tasks, such as:

   • The full-features tMap component and other Processing components

   • Native connectors to Cassandra

   • Components for Spark SQL and Elasticsearch

   • Native RDBMS components

   • Spark Tuning properties

3. Hierarchical Avro files:

   • The tAvroInput component in a Spark Job can now handle the hierarchical Avro schema.

   • In a Spark Batch Job, the tAvroOutput component allows users to define a hierarchical key/value schema for the Avro file to be written.

4. In addition to the Spark components used for data integration, the new Machine learning components allow users to leverage Spark to make predictive analyses.

Discovery and reusability in Hadoop cluster metadata

1. An import wizard for Hadoop configuration has been created to import the configurations of various Hadoop services directly from different sources.

   • This wizard supports importing configurations directly from the Ambari service or the Cloudera manager service.

   • When users import configurations from local files, this wizard supports all of the distributions already officially supported by Talend.

2. Users can create environmental context out of the metadata of a connection to a Hadoop service such as HDFS or Hive. Then this context can be used to define the parameters of the Hadoop connection to be created.

Upgraded support for NoSQL and Hadoop distributions

1. New versions of the following Hadoop distributions are supported:

   • Cloudera 5.4 (YARN)

   • MapR 4.1.0 (YARN)

   • Hortonworks Data Platform V2.2

   • EMR Apache 2.4.0 and EMR 4.0.0 (Apache 2.6.0)

2. Cassandra:

   Cassandra CQL3 and the Datastax API are now available to the Cassandra components.

3. MongoDB:

   • MongoDB 3.0.X. is supported

   • The MongoDB components now support more operations such as "Bulk write" or "Write concern" and have been enabled to work with different authentication mechanisms including Kerberos and SCRAM-SHA1.

Hadoop components enhancements

1. The support of the Tez framework has been added to the Hive and the Pig components.

2. With HiveServer 2 in the Standalone mode, the Hive components now support the Kerberos authentication.

3. From the Hive components, the SSL connection is now available to HiveServer2 in the Standalone mode of the following distributions:

   • Hortonworks Data Platform 2.0 +

   • Cloudera CDH4 +

   • Pivotal HD 2.0 +

4. Support for Kerberos has been added to the HBase components and the HBase connection in the Hive and Pig components.

5. The tHDFSOutputRaw component has been created to write data into more formats than the Text and the Sequence formats.

New messaging components in the Data Integration Jobs

The following Kafka components have been created to help users visually create a topic and consume or publish messages:

• tKafkaCreateTopic

• tKafkaInput

• tKafkaOutput

Memory Test Execution

Users can now monitor the JVM memory consumption and CPU usage via curve graphs during Job execution

Enhanced look and feel and user experience

The Studio has undergone a number of changes to enhance the look and feel and user experience.

1. The Studio user interface and component icons have been redesigned for enhanced look and feel.

2. The Studio login process has been streamlined:

   • For the first startup, fewer steps are required.

   • After the first start up, users can directly open the same project without seeing the login dialog.

3. Users can now drag and drop an output component from an input one on the design workspace.

4. Enhanced component search:

   • Users can now search for a component by providing a descriptive phrase as the search keywords, without having to know the component name. The maximum number of entries in the search result is configurable.

   • The Palette now shows recently used components to ease component search in Job design. The maximum number of components shown on the Recently Used list is configurable.

Java 8 support

The Studio now supports Java 8 only.

Components updates

Databases

Teradata SCD

Teradata SCD (Slowly Changing Dimensions) is now supported through the new component tTeradataSCD.

Vertica

VerticaELT and SQL Templates

1. Vertica ELT is now supported through the following new components:

   • tELTVerticaInput

   • tELTVerticaMap

   • tELTVerticaOutput

2. SQL Templates for Vertica are now supported.

VerticaBulk

tVerticaBulkExec and tVerticaOutputBulkExec now support bulk-loading data to a Vertica database based on the defined schema columns.

Batch mode

The following components now support batch mode when using an existing connection:

Oracle

tOracleSP now supports mapping the source database type to XMLTYPE.

PostgreSQL

1. PostgreSQL v9.x.x is now supported.

Cloud components

Salesforce

1. tSalesforceWaveBulkExec and tSalesforceWaveOutputBulkExec, which let you load data to Salesforce Analytics Cloud, have been added to the Studio.

2. tSalesforceWaveBulkExec and tSalesforceWaveOutputBulkExec support reusing an existing connection and retrieving the upload status.

Netsuite

1. tNetsuiteInput and tNetsuiteOutput, which let you read/write data from/to NetSuite, have been added to the Studio.

2. tNetsuiteOutput supports retrieving data rows with errors via a Reject row connection.

Marketo

1. tMarketoInput, which supports three lead selector types when retrieving lead records from Marketo.

2. tMarketoOutput, which supports providing the status (CREATED, UPDATED, FAILED) and the optional error text.

Others

1. Four tRedshift* load/unload components, which let you load/unload data from/to Amazon S3.

2. Seven tGoogleDrive* components, which let you create, upload, download, copy, delete, move and list files and folders on Google Drive.

3. Six tAmazonAurora* components, which let you read/write data from/to Amazon Aurora databases.

4. tS3Put now supports server-side encryption.

Messaging components

1. Static transport discovery mechanism is now supported for the ActiveMQ server in tMomXXX components.

2. tMomInput and tMomRollback now support backouting the messages to backout queue when the backout count reaches the configured threshold.