2.2.1: Fix vulnerabilities in dependencies

Summary

This release fixes the following vulnerabilities in dependencies:

• CVE-2024-47554 in commons-io:commons-io:jar:2.11.0:compile
• CVE-2024-47561 in org.apache.avro:avro:jar:1.11.3:compile
• CVE-2024-47535 in io.netty:netty-common:jar:4.1.109.Final:compile
• CVE-2024-23454 in org.apache.hadoop:hadoop-common:jar:3.3.6:provided
• CVE-2024-12798 and CVE-2024-12801 in ch.qos.logback:logback-core:jar:1.2.13:compile
• CVE-2024-7254 in com.google.protobuf:protobuf-java:jar:3.24.3:provided
• CVE-2024-51504 in org.apache.zookeeper:zookeeper:jar:3.9.2:compile

Security

• #230: Fixed vulnerabilities in dependencies

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated org.apache.maven.plugins:maven-deploy-plugin:3.1.1 to 3.1.3

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Updated com.exasol:exasol-jdbc:24.1.2 to 24.2.1
• Updated com.exasol:spark-connector-common-java:2.0.5 to 2.0.10
• Updated org.apache.avro:avro:1.11.3 to 1.11.4

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.3 to 7.1.2
• Updated com.exasol:hamcrest-resultset-matcher:1.6.1 to 1.7.0
• Updated com.exasol:test-db-builder-java:3.5.1 to 3.6.0
• Updated org.apache.logging.log4j:log4j-1.2-api:2.20.0 to 2.24.3
• Updated org.apache.logging.log4j:log4j-api:2.20.0 to 2.24.3
• Updated org.apache.logging.log4j:log4j-core:2.20.0 to 2.24.3
• Updated org.mockito:mockito-core:5.5.0 to 5.15.2
• Updated org.mockito:mockito-junit-jupiter:5.5.0 to 5.15.2

Spark Exasol Connector With S3

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:2.0.5 to 2.0.10
• Updated org.apache.avro:avro:1.11.3 to 1.11.4
• Updated software.amazon.awssdk:s3:2.20.115 to 2.29.51

Test Dependency Updates

• Updated com.amazonaws:aws-java-sdk-s3:1.12.639 to 1.12.780
• Updated com.exasol:exasol-testcontainers:6.6.3 to 7.1.2
• Updated com.exasol:hamcrest-resultset-matcher:1.6.1 to 1.7.0
• Updated com.exasol:test-db-builder-java:3.5.1 to 3.6.0
• Updated org.junit-pioneer:junit-pioneer:2.1.0 to 2.3.0
• Updated org.junit.jupiter:junit-jupiter-api:5.10.0 to 5.11.4
• Updated org.junit.jupiter:junit-jupiter:5.10.0 to 5.11.4
• Updated org.mockito:mockito-junit-jupiter:5.5.0 to 5.15.2
• Updated org.testcontainers:junit-jupiter:1.19.0 to 1.20.4
• Updated org.testcontainers:localstack:1.19.0 to 1.20.4

2.2.0: Fix Exasol-JDBC compatibility, CVE CVE-2024-25638

Summary

Fixed issue with Exasol-JDBC parameter change.
Fixed CVE-2024-25638 in dnsjava:dnsjava:jar:2.1.7:provided (hadoop-client dependency).

Security

• #228: Fix vulnerability in dnsjava:dnsjava:jar:2.1.7:provided

Bugfixes

• #227: Spark Connector is not compatible with exasol-jdbc V24+

Dependency Updates

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Updated com.exasol:exasol-jdbc:7.1.20 to 24.1.2
• Updated com.exasol:spark-connector-common-java:2.0.1 to 2.0.5

Spark Exasol Connector With S3

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:2.0.1 to 2.0.5

2.1.6: Spark 3.5 support

Summary

Added support of spark 3.5.

Features

• #221: Add support for spark 3.5
• #219: Updated dependency to fix CVE-2023-52428

2.1.5: Fix CVE-2024-21634 and CVE-2023-33546

Summary

This release fixes CVE-2024-21634 in transitive test dependency software.amazon.ion:ion-java.

Security

• #214: Fixed CVE-2024-21634 in software.amazon.ion:ion-java

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated org.apache.maven.plugins:maven-surefire-plugin:3.1.2 to 3.2.3
• Added org.apache.maven.plugins:maven-toolchains-plugin:3.1.0
• Updated org.codehaus.mojo:versions-maven-plugin:2.16.1 to 2.16.2

Spark Exasol Connector With JDBC

Plugin Dependency Updates

• Added org.apache.maven.plugins:maven-toolchains-plugin:3.1.0

Spark Exasol Connector With S3

Test Dependency Updates

• Updated com.amazonaws:aws-java-sdk-s3:1.12.518 to 1.12.639

Plugin Dependency Updates

• Added org.apache.maven.plugins:maven-toolchains-plugin:3.1.0

2.1.4: PK back in config, CVE fixed in transitive deps.

Summary

Project keeper added to Maven configuration.
Vulnerability fixed in transitive dependency.

Features

• #211: Brought PK plugin back to the Maven config
• #212: Fix vulnerability in org.eclipse.parsson:parsson

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated com.exasol:error-code-crawler-maven-plugin:1.3.0 to 1.3.1
• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.4.0 to 3.4.1
• Updated org.codehaus.mojo:versions-maven-plugin:2.16.0 to 2.16.1
• Updated org.jacoco:jacoco-maven-plugin:0.8.10 to 0.8.11
• Updated org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184 to 3.10.0.2594

Spark Exasol Connector With JDBC

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.2 to 6.6.3

Spark Exasol Connector With S3

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.2 to 6.6.3

2.1.3: More flexibility for AWS Credentials specification in spark-connector-s3

Summary

In addition to explicit AWS Credentials specification we now support environment variables and EC2 instance profiles.
Fixes CVE-2023-39410 in apache avro (transitive dependency).

Features

• 192: Add support for AWS IAM Profile Credentials for s3 connector.

Dependency Updates

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Added org.apache.avro:avro:1.11.3

Spark Exasol Connector With S3

Compile Dependency Updates

• Added org.apache.avro:avro:1.11.3

Test Dependency Updates

• Added org.junit-pioneer:junit-pioneer:2.1.0

2.1.2: Transaction commit fixes, security fixes in dependencies

Summary

Fixes CVE-2023-42503 in org.apache.commons:commons-compress.
Fixes logic in transaction commit during the DB import.

Features

• 194: Wrong name of Exasol JDBC format in documentation
• 197: Committing transaction in the finally handler
• 198: Remove unused Statement variable
• 206: Fix CVE-2023-42503 and update dependencies

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Updated org.apache.maven.plugins:maven-enforcer-plugin:3.3.0 to 3.4.0

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:2.0.0 to 2.0.1

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.1 to 6.6.2
• Updated com.exasol:hamcrest-resultset-matcher:1.6.0 to 1.6.1
• Updated com.exasol:test-db-builder-java:3.4.2 to 3.5.1
• Updated org.mockito:mockito-core:5.4.0 to 5.5.0
• Updated org.mockito:mockito-junit-jupiter:5.4.0 to 5.5.0
• Updated org.scalatestplus:scalatestplus-mockito_2.13:1.0.0-M2 to 1.0.0-SNAP5
• Updated org.scalatest:scalatest_2.13:3.2.9 to 3.3.0-SNAP4

Spark Exasol Connector With S3

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:2.0.0 to 2.0.1
• Updated org.apache.hadoop:hadoop-aws:3.3.4 to 3.3.6

Test Dependency Updates

• Updated com.exasol:exasol-testcontainers:6.6.1 to 6.6.2
• Updated com.exasol:hamcrest-resultset-matcher:1.6.0 to 1.6.1
• Updated com.exasol:test-db-builder-java:3.4.2 to 3.5.1
• Updated org.mockito:mockito-junit-jupiter:5.4.0 to 5.5.0
• Updated org.testcontainers:junit-jupiter:1.18.3 to 1.19.0
• Updated org.testcontainers:localstack:1.18.3 to 1.19.0

2.1.1: JDBC connection cache drop, `magic` s3 hadoop committer

Summary

S3 connector uses faster hadoop committer, fix possible race condition in RDD operations

Features

• #163: Switched to magic hadoop committer for intermediate s3 files
• #200: Dropped JDBC connection cache in ExasolConnectionManager

2.1.0: Added filter pushdown and column selection for `S3` variant

Summary

This release adds predicate pushdown and column selection for the S3 variant. Similarly, uses the latest API changes from the spark-connector-common-java library.

Features

• #190: Added predicate pushdown and column selection for S3 variant

Dependency Updates

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:1.1.1 to 2.0.0

Plugin Dependency Updates

• Updated com.diffplug.spotless:spotless-maven-plugin:2.37.0 to 2.38.0

Spark Exasol Connector With S3

Compile Dependency Updates

• Updated com.exasol:spark-connector-common-java:1.1.1 to 2.0.0
• Updated software.amazon.awssdk:s3:2.20.103 to 2.20.115

Test Dependency Updates

• Updated com.amazonaws:aws-java-sdk-s3:1.12.506 to 1.12.518
• Added com.exasol:java-util-logging-testing:2.0.3
• Updated org.junit.jupiter:junit-jupiter-api:5.9.3 to 5.10.0
• Updated org.junit.jupiter:junit-jupiter:5.9.3 to 5.10.0

2.0.0: Support S3 intermediate storage

Summary

In this release we added support to use AWS S3 bucket as an intermediate storage layer when accessing Exasol database from Spark cluster.

With this release, we separated the connector into two variants, S3 and JDBC. We recommend to use this new S3 variant instead of JDBC variant. It improves the stability of the connector.

Please refer to the user guide for updated API changes and usage instructions.

Features

• #149: Added s3 import query generator and runner
• #150: Added S3 intermediate storage layer
• #159: Added cleanup process to remove intermediate data after job finish
• #160: Add support for writing to Exasol database using S3 as intermediate storage
• #168: Refactored to add module setup

Security

• #151: Fixed vulnerability CVE-2023-26048 coming with jetty-util transitive dependency

Bugs

• #176: Fixed artifact upload of sha256sum files
• #188: Fixed Maven Central deployment bug

Refactoring

• #155: Unified user options
• #158: Refactored common options class
• #164: Validated that write directory is empty
• #171: Refactored artifact packaging and releasing for module setup
• #174: Refactored Github .github/workflow/ action files
• #183: Updated user guide and prepared for release

Dependency Updates

Spark Exasol Connector Parent POM

Plugin Dependency Updates

• Added com.exasol:error-code-crawler-maven-plugin:1.3.0
• Added io.github.zlika:reproducible-build-maven-plugin:0.16
• Added org.apache.maven.plugins:maven-clean-plugin:2.5
• Added org.apache.maven.plugins:maven-compiler-plugin:3.11.0
• Added org.apache.maven.plugins:maven-deploy-plugin:3.1.1
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.3.0
• Added org.apache.maven.plugins:maven-install-plugin:2.4
• Added org.apache.maven.plugins:maven-site-plugin:3.3
• Added org.apache.maven.plugins:maven-surefire-plugin:3.1.2
• Added org.basepom.maven:duplicate-finder-maven-plugin:2.0.1
• Added org.codehaus.mojo:flatten-maven-plugin:1.5.0
• Added org.codehaus.mojo:versions-maven-plugin:2.16.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.6.2
• Added org.jacoco:jacoco-maven-plugin:0.8.10
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0

Spark Exasol Connector With JDBC

Compile Dependency Updates

• Added com.exasol:error-reporting-java8:1.0.1
• Added com.exasol:exasol-jdbc:7.1.20
• Added com.exasol:spark-connector-common-java:1.1.1
• Added com.exasol:sql-statement-builder-java8:4.5.4

Test Dependency Updates

• Added com.exasol:exasol-testcontainers:6.6.1
• Added com.exasol:hamcrest-resultset-matcher:1.6.0
• Added com.exasol:test-db-builder-java:3.4.2
• Added org.apache.logging.log4j:log4j-1.2-api:2.20.0
• Added org.apache.logging.log4j:log4j-api:2.20.0
• Added org.apache.logging.log4j:log4j-core:2.20.0
• Added org.mockito:mockito-core:5.4.0
• Added org.mockito:mockito-junit-jupiter:5.4.0
• Added org.scalatestplus:scalatestplus-mockito_2.13:1.0.0-M2
• Added org.scalatest:scalatest_2.13:3.2.9

Plugin Dependency Updates

• Added com.diffplug.spotless:spotless-maven-plugin:2.37.0
• Added com.exasol:error-code-crawler-maven-plugin:1.3.0
• Added io.github.evis:scalafix-maven-plugin_2.13:0.1.4_0.9.31
• Added io.github.zlika:reproducible-build-maven-plugin:0.16
• Added net.alchim31.maven:scala-maven-plugin:4.8.1
• Added org.apache.maven.plugins:maven-clean-plugin:2.5
• Added org.apache.maven.plugins:maven-compiler-plugin:3.11.0
• Added org.apache.maven.plugins:maven-deploy-plugin:3.1.1
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.3.0
• Added org.apache.maven.plugins:maven-failsafe-plugin:3.1.2
• Added org.apache.maven.plugins:maven-gpg-plugin:3.1.0
• Added org.apache.maven.plugins:maven-install-plugin:2.4
• Added org.apache.maven.plugins:maven-jar-plugin:3.3.0
• Added org.apache.maven.plugins:maven-javadoc-plugin:3.5.0
• Added org.apache.maven.plugins:maven-resources-plugin:2.6
• Added org.apache.maven.plugins:maven-shade-plugin:3.4.1
• Added org.apache.maven.plugins:maven-site-plugin:3.3
• Added org.apache.maven.plugins:maven-source-plugin:3.2.1
• Added org.apache.maven.plugins:maven-surefire-plugin:3.1.2
• Added org.basepom.maven:duplicate-finder-maven-plugin:2.0.1
• Added org.codehaus.mojo:flatten-maven-plugin:1.5.0
• Added org.codehaus.mojo:versions-maven-plugin:2.16.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.6.2
• Added org.jacoco:jacoco-maven-plugin:0.8.10
• Added org.scalatest:scalatest-maven-plugin:2.2.0
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0
• Added org.sonatype.plugins:nexus-staging-maven-plugin:1.6.13

Spark Exasol Connector With S3

Compile Dependency Updates

• Added com.exasol:spark-connector-common-java:1.1.1
• Added org.apache.hadoop:hadoop-aws:3.3.4
• Added org.scala-lang:scala-library:2.13.11
• Added org.wildfly.openssl:wildfly-openssl:2.2.5.Final
• Added software.amazon.awssdk:s3:2.20.103

Test Dependency Updates

• Added com.amazonaws:aws-java-sdk-s3:1.12.506
• Added com.exasol:exasol-testcontainers:6.6.1
• Added com.exasol:hamcrest-resultset-matcher:1.6.0
• Added com.exasol:test-db-builder-java:3.4.2
• Added org.junit.jupiter:junit-jupiter-api:5.9.3
• Added org.junit.jupiter:junit-jupiter:5.9.3
• Added org.mockito:mockito-junit-jupiter:5.4.0
• Added org.testcontainers:junit-jupiter:1.18.3
• Added org.testcontainers:localstack:1.18.3

Plugin Dependency Updates

• Added com.exasol:error-code-crawler-maven-plugin:1.3.0
• Added io.github.zlika:reproducible-build-maven-plugin:0.16
• Added org.apache.maven.plugins:maven-clean-plugin:2.5
• Added org.apache.maven.plugins:maven-compiler-plugin:3.11.0
• Added org.apache.maven.plugins:maven-deploy-plugin:3.1.1
• Added org.apache.maven.plugins:maven-enforcer-plugin:3.3.0
• Added org.apache.maven.plugins:maven-failsafe-plugin:3.1.2
• Added org.apache.maven.plugins:maven-gpg-plugin:3.1.0
• Added org.apache.maven.plugins:maven-install-plugin:2.4
• Added org.apache.maven.plugins:maven-jar-plugin:2.4
• Added org.apache.maven.plugins:maven-javadoc-plugin:3.5.0
• Added org.apache.maven.plugins:maven-resources-plugin:2.6
• Added org.apache.maven.plugins:maven-shade-plugin:3.4.1
• Added org.apache.maven.plugins:maven-site-plugin:3.3
• Added org.apache.maven.plugins:maven-source-plugin:3.2.1
• Added org.apache.maven.plugins:maven-surefire-plugin:3.1.2
• Added org.basepom.maven:duplicate-finder-maven-plugin:2.0.1
• Added org.codehaus.mojo:flatten-maven-plugin:1.5.0
• Added org.codehaus.mojo:versions-maven-plugin:2.16.0
• Added org.itsallcode:openfasttrace-maven-plugin:1.6.2
• Added org.jacoco:jacoco-maven-plugin:0.8.10
• Added org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184
• Added org.sonatype.ossindex.maven:ossindex-maven-plugin:3.2.0
• Added org.sonatype.plugins:nexus-staging-maven-plugin:1.6.13