working on lineage es integration #14181

07Himank · 2023-11-30T06:28:00Z

I worked on ... because ... [ WIP ] working on lineage es integration

Fixes #8680
Fixes #12609
Fixes #11795
Fixes #13119
Fixes #11515
Fixes #14048

Type of change:

[ x] Improvement

…ta/OpenMetadata into lineageEsIntegration

github-actions · 2023-12-12T11:09:58Z

The Java checkstyle failed.

Please run mvn googleformatter:format@reformat-sources in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-01-08T13:32:22Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-01-08T14:49:50Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-01-08T15:44:58Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-01-08T16:52:20Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

github-actions · 2024-01-08T17:02:05Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java

harshach · 2024-01-08T19:21:04Z

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java

+    relationshipDetails.put("toEntity", toDetails);
+    if (lineageDetails != null) {
+      relationshipDetails.put(
+          "pipeline",


this could be pipeline or stored procedure as an edge cc @pmbrull

@07Himank need to address this, it can be pipeline or stored procedure here

@harshach just key name is pipeline but it works for both pipeline and stored procedure .. do you want to me change the key name?

...metadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java

harshach · 2024-01-08T19:22:45Z

...service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchClient.java

+    if (depth <= 0) {
+      return;
+    }
+    es.org.elasticsearch.action.search.SearchRequest searchRequest =


can we move this into import

@harshach We have two SearchRequest one is internal that why we use it this way.

...service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchClient.java

github-actions · 2024-01-09T05:58:44Z

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

…ta/OpenMetadata into lineageEsIntegrationi

sonarqubecloud · 2024-01-09T12:03:01Z

Quality Gate passed for 'open-metadata-ui'

The SonarCloud Quality Gate passed, but some issues were introduced.

4 New issues
0 Security Hotspots
0.0% Coverage on New Code
1.2% Duplication on New Code

See analysis details on SonarCloud

sonarqubecloud · 2024-01-09T12:14:57Z

Quality Gate passed for 'open-metadata-ingestion'

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

07Himank added 2 commits November 30, 2023 11:52

working on lineage es integration

f880e28

os change

2ca0d14

07Himank added the safe to test Add this label to run secure Github workflows on PRs label Nov 30, 2023

07Himank requested a review from a team as a code owner November 30, 2023 06:28

07Himank marked this pull request as draft November 30, 2023 06:28

github-actions bot added the backend label Nov 30, 2023

karanh37 and others added 8 commits December 8, 2023 13:00

Merge branch 'main' into lineageEsIntegration

7e463b7

Merge branch 'main' into lineageEsIntegration

6c70458

working

aa2cf35

working on lineage es integration

c0d3269

Merge branch 'main' into lineageEsIntegration

9db0635

add curremt lineage, special character handling, delete lineage

e7c79ab

Merge branch 'lineageEsIntegration' of https://github.com/open-metada…

f716d18

…ta/OpenMetadata into lineageEsIntegration

remove test code

4a1942d

07Himank and others added 2 commits December 13, 2023 17:01

update lineage index work done

8cc9868

fix: add lineage ui changes

83347d7

github-actions bot added the UI UI specific issues label Dec 14, 2023

karanh37 and others added 12 commits December 14, 2023 18:05

fix: lineage add edit

1dd5fa3

fix: center on load

dd6ee2a

fix: add remove edges

79f5357

Merge branch 'main' into lineageEsIntegration

9499b6d

fix: column lineage issues

563c07e

use reactflow controls

6467ddb

add collapse

06451f2

added upstream and downstream depth diffrently

5f8c9ba

resolved conflicts

d0d703b

depth and collapse

f91521c

fix loading of dialog

f47f7cf

add includeDeleted in API

6e51586

add pipeline status color

26757ab

fix unit tests

9437778

Merge branch 'main' into lineageEsIntegration

f1e3f6f

cleanup

6a3aaad

harshach reviewed Jan 8, 2024

View reviewed changes

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java Outdated Show resolved Hide resolved

harshach reviewed Jan 8, 2024

View reviewed changes

openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/LineageRepository.java Outdated Show resolved Hide resolved

harshach reviewed Jan 8, 2024

View reviewed changes

...metadata-service/src/main/java/org/openmetadata/service/resources/search/SearchResource.java Outdated Show resolved Hide resolved

harshach reviewed Jan 8, 2024

View reviewed changes

...service/src/main/java/org/openmetadata/service/search/elasticsearch/ElasticSearchClient.java Outdated Show resolved Hide resolved

cleanup

795d23e

07Himank and others added 4 commits January 9, 2024 14:21

added opensearch code and changed endpoint

1075013

Merge branch 'lineageEsIntegration' of https://github.com/open-metada…

015b7ac

…ta/OpenMetadata into lineageEsIntegrationi

fix: lineage url

b7ccc3c

depth issue resolved

0da8c79

07Himank changed the title ~~[ WIP ] working on lineage es integration~~ working on lineage es integration Jan 9, 2024

07Himank marked this pull request as ready for review January 9, 2024 11:47

chirag-madlani approved these changes Jan 9, 2024

View reviewed changes

harshach approved these changes Jan 9, 2024

View reviewed changes

harshach merged commit 44af8b3 into main Jan 9, 2024
23 checks passed

harshach deleted the lineageEsIntegration branch January 9, 2024 16:34

karanh37 mentioned this pull request Jan 11, 2024

Show service type in lineage node #12613

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

working on lineage es integration #14181

working on lineage es integration #14181

07Himank commented Nov 30, 2023 •

edited by karanh37

Loading

github-actions bot commented Dec 12, 2023

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

harshach Jan 8, 2024

harshach Jan 9, 2024

07Himank Jan 10, 2024

harshach Jan 8, 2024

07Himank Jan 10, 2024

github-actions bot commented Jan 9, 2024

sonarqubecloud bot commented Jan 9, 2024

sonarqubecloud bot commented Jan 9, 2024

working on lineage es integration #14181

working on lineage es integration #14181

Conversation

07Himank commented Nov 30, 2023 • edited by karanh37 Loading

Type of change:

github-actions bot commented Dec 12, 2023

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

github-actions bot commented Jan 8, 2024

harshach Jan 8, 2024

Choose a reason for hiding this comment

harshach Jan 9, 2024

Choose a reason for hiding this comment

07Himank Jan 10, 2024

Choose a reason for hiding this comment

harshach Jan 8, 2024

Choose a reason for hiding this comment

07Himank Jan 10, 2024

Choose a reason for hiding this comment

github-actions bot commented Jan 9, 2024

sonarqubecloud bot commented Jan 9, 2024

Quality Gate passed for 'open-metadata-ui'

sonarqubecloud bot commented Jan 9, 2024

Quality Gate passed for 'open-metadata-ingestion'

07Himank commented Nov 30, 2023 •

edited by karanh37

Loading