JasperETL: How to upload very large files with tS3Put

Issue:

You can upload smaller files to Amazon S3 using tS3Put. If you try to upload larger files more than 10GB, you get an error saying "max size exceeded".

Using Java code you can upload files greater than 10GB. How can you do it in JasperETL?

Resolution

This feature has been added in Talend v6.3.1. It will be released with JasperETL the release after JRS release v6.3.0 in 2017. This JasperETL version number may have a number higher than v6.3.1 [e.g. v6.4.0].

Note that S3 PUT operation is limited to a max file size of 5 GB. However, using the Multipart upload API allows objects up to 5 TB to be uploaded. See http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html for more details. Existing S3 components such as tS3Put or tS3Connection do not provide an option to load such large files. We would need a new component/component parameters to fully utilize the Multipart upload API. Talend has added this feature in 6.3.1 release. https://jira.talendforge.org/browse/TDI-35666

Ref. Case #01459958