🎉 New destination: Cassandra #7186

itaseskii · 2021-10-19T19:45:41Z

What

The MR adds new destination connector for Apache Cassandra

How

Using the Java Datastax driver https://docs.datastax.com/en/developer/java-driver/4.13/

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

CLAassistant · 2021-10-19T19:45:45Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ itaseskii
❌ itaseski

itaseski seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

itaseskii · 2021-10-19T20:08:44Z

Hi Airbyte team :) I would like to submit my Cassandra destination connector for the Airbyte platform. Unfortunately I have the same issue with the acceptance tests as the Elasticsearch connector #7005 (comment). My unit and integration tests run without a problem but the build gets stuck on the acceptance tests, from initial investigation it seems that the running container won't exit after the write command has been executed and that locks the gradle build.

itaseskii · 2021-10-19T21:14:57Z

Also, can you please remove the 'itaseski' user from the CLA issue? I have already signed the agreement with my Github acc.

marcosmarxm · 2021-10-20T02:13:16Z

thanks for the contribution @itaseskii I added this to our sprint to review. About the acceptance tests any news to solve I'll let you know. If possible run ./gradlew format to format all files.

alexandr-shegeda · 2021-10-20T11:56:56Z

airbyte-integrations/connectors/destination-cassandra/Dockerfile

+
+RUN tar xf ${APPLICATION}.tar --strip-components=1
+
+LABEL io.airbyte.version=1.0.0


please, change the version to 0.1.0

changed to 0.1.0

alexandr-shegeda · 2021-10-20T12:08:13Z

...sandra/src/main/java/io/airbyte/integrations/destination/cassandra/CassandraDestination.java

+    private static final Logger LOGGER = LoggerFactory.getLogger(CassandraDestination.class);
+
+    public static void main(String[] args) throws Exception {
+        new IntegrationRunner(new CassandraDestination()).run(args);


it would be great to add logs around the connector initialization

Suggested change

new IntegrationRunner(new CassandraDestination()).run(args);

LOGGER.info("starting destination: {}", CassandraDestination.class);

new IntegrationRunner(new CassandraDestination()).run(args);

LOGGER.info("completed destination: {}", CassandraDestination.class);

Since run() is already logging the info that the integration/destination has started and finished isn't that enough? :)

alexandr-shegeda · 2021-10-20T12:29:10Z

...ra/src/main/java/io/airbyte/integrations/destination/cassandra/CassandraMessageConsumer.java

+            var streamConfig =
+                cassandraStreams.get(AirbyteStreamNameNamespacePair.fromRecordMessage(messageRecord));
+            Preconditions.checkNotNull(streamConfig, "Unrecognized destination stream");
+            //var formattedNode = CassandraNameTransformer.formatJsonPath(messageRecord.getData());


please, remove commented line if not needed

alexandr-shegeda · 2021-10-20T12:33:15Z

...gration/java/io/airbyte/integrations/destination/cassandra/CassandraContainerInitializr.java

+        if (cassandraContainer == null) {
+            cassandraContainer = new ConfiguredCassandraContainer();
+            // max heap size
+            // cassandraContainer.addEnv("MAX_HEAP_SIZE", "2048");


please, remove commented line if not needed

…aseskii/airbyte into feature/destination-cassandra

itaseskii · 2021-10-24T22:23:51Z

Hey @marcosmarxm :) I have formatted the code with ./gradlew format and I have managed to fix my acceptance tests. The reason the jvm was hanging was because of a missing executor service shutdown. I have also done some general code improvements and refactoring. As an attachment I am sending you screenshot of the passing acceptance tests.

VitaliiMaltsev · 2021-10-25T09:12:17Z

airbyte-integrations/connectors/destination-cassandra/src/main/resources/spec.json

+      "address": {
+        "title": "Address",
+        "description": "Address to connect to.",
+        "type": "string",


please add an example here

* add cassandra destination connector * refactor and docs. * delete test dockerfile * revert Dockerfile rm change * refactor & fix acceptance tests & format * revert stream peek * remove get pip * add address example * improved copy and code refactor * add docker-compose and improved docs Co-authored-by: itaseski <[email protected]>

itaseski added 4 commits October 18, 2021 11:33

add cassandra destination connector

84feb3c

refactor and docs.

7eb87c9

delete test dockerfile

e599f69

revert Dockerfile rm change

5235bcb

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 19, 2021

octavia-squidington-iii added the community label Oct 19, 2021

Merge branch 'master' into feature/destination-cassandra

819e149

itaseskii mentioned this pull request Oct 19, 2021

Destination: Cassandra airbytehq/connector-contest#64

Closed

marcosmarxm added the hacktober label Oct 19, 2021

marcosmarxm self-assigned this Oct 20, 2021

marcosmarxm added the gl label Oct 20, 2021

alexandr-shegeda reviewed Oct 20, 2021

View reviewed changes

itaseski added 4 commits October 24, 2021 22:37

refactor & fix acceptance tests & format

63fed1f

Merge branch 'feature/destination-cassandra' of https://github.com/it…

65da9cc

…aseskii/airbyte into feature/destination-cassandra

revert stream peek

5d78952

remove get pip

80e2008

VitaliiMaltsev reviewed Oct 25, 2021

View reviewed changes

VitaliiMaltsev approved these changes Oct 27, 2021

View reviewed changes

itaseskii requested a review from alexandr-shegeda October 27, 2021 08:54

itaseskii added 2 commits October 29, 2021 00:37

add address example

0595a6d

improved copy and code refactor

d54a3d8

add docker-compose and improved docs

d619155

marcosmarxm merged commit f53fd5e into airbytehq:master Nov 5, 2021

jrhizor mentioned this pull request Nov 6, 2021

Bump Airbyte version from 0.30.31-alpha to 0.30.32-alpha #7700

Merged

karinakuz added connectors/destination/cassandra connectors/destinations-database labels Jan 17, 2022

marcosmarxm removed the hacktober label Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New destination: Cassandra #7186

🎉 New destination: Cassandra #7186

itaseskii commented Oct 19, 2021 •

edited

Loading

CLAassistant commented Oct 19, 2021 •

edited

Loading

itaseskii commented Oct 19, 2021 •

edited

Loading

itaseskii commented Oct 19, 2021

marcosmarxm commented Oct 20, 2021 •

edited

Loading

alexandr-shegeda Oct 20, 2021

itaseskii Oct 24, 2021

alexandr-shegeda Oct 20, 2021

itaseskii Oct 24, 2021

alexandr-shegeda Oct 20, 2021

itaseskii Oct 24, 2021

alexandr-shegeda Oct 20, 2021

itaseskii Oct 24, 2021

itaseskii commented Oct 24, 2021

VitaliiMaltsev Oct 25, 2021

itaseskii Oct 28, 2021


		RUN tar xf ${APPLICATION}.tar --strip-components=1

		LABEL io.airbyte.version=1.0.0

🎉 New destination: Cassandra #7186

🎉 New destination: Cassandra #7186

Conversation

itaseskii commented Oct 19, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Oct 19, 2021 • edited Loading

itaseskii commented Oct 19, 2021 • edited Loading

itaseskii commented Oct 19, 2021

marcosmarxm commented Oct 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itaseskii commented Oct 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itaseskii commented Oct 19, 2021 •

edited

Loading

CLAassistant commented Oct 19, 2021 •

edited

Loading

itaseskii commented Oct 19, 2021 •

edited

Loading

marcosmarxm commented Oct 20, 2021 •

edited

Loading