update vpc flow with flint-s3 based DDL assets and dashboard #1691

YANG-DB · 2024-04-15T04:31:27Z

Description

Add Flint S3 datasource support for VPC flow logs based integration

include live query (week's worth data) MV
include pre-aggregated MV queries
include pre-aggregated MV window based queries

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: YANGDB <[email protected]>

RyanL1997

Hi @YANG-DB , thanks for putting these together. Just left some minor reviews.

RyanL1997 · 2024-04-15T16:56:24Z

.../integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql

+WITH (
+    auto_refresh = false
+)
+-- Enable for Auto refresh --


I think we still need to go with the auto refresh here. The empty index issue has been address by using different checkpoints when creating new MV.

RyanL1997 · 2024-04-15T16:58:17Z

...tors/integrations/__data__/repository/aws_vpc_flow/assets/create_table_parquet_vpc-1.0.0.sql

@@ -0,0 +1,75 @@
+ --- DDL VPC create table statement definition


should we include the integration name in the file name? I think for all of the pending PRs, we are just named it like create_table-1.0.0

Signed-off-by: YANGDB <[email protected]>

Swiddis · 2024-04-15T21:37:45Z

Also closes opensearch-project/opensearch-catalog#143

Swiddis · 2024-04-15T23:56:56Z

.../integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql

@@ -0,0 +1,37 @@
+-- 30 Min Aggregation MV of VPC connections / bytes / packets


I think the file gets flattened at some point of install, so comments will cause issues

.../integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql

.../integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_60min_connections_mv-1.0.0.sql

server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_week_mv-1.0.0.sql

..._data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql

..._data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql

Swiddis

^ Meant to request changes instead of approving

…assets/vpc_agg_30min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

…assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

…assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

…assets/vpc_live_week_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

…assets/vpc_agg_60min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

Signed-off-by: YANGDB <[email protected]>

Swiddis

Tentatively approving assuming this runs, I haven't tested it locally but I don't see any obvious issues. Seems like active changes are still happening.

add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]>

Signed-off-by: YANGDB <[email protected]>

# Conflicts: # server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/create_mv_vpc-1.0.0.sql

Signed-off-by: YANGDB <[email protected]>

opensearch-trigger-bot · 2024-04-18T06:26:55Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/dashboards-observability/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/dashboards-observability/backport-2.x
# Create a new branch
git switch --create backport/backport-1691-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 71592c46ac18457ffe60853dd1a30c7435a63943
# Push it to GitHub
git push --set-upstream origin backport/backport-1691-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/dashboards-observability/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-1691-to-2.x.

…arch-project#1691) * Cypress13 testing frame work for OIDC and SAML (opensearch-project#1579) * Cypress13 testing frame work for OIDC and SAML --------- Signed-off-by: Ryan Liang <[email protected]> * Correct the workflow version Signed-off-by: Ryan Liang <[email protected]> * Update yarn lock Signed-off-by: Ryan Liang <[email protected]> --------- Signed-off-by: Ryan Liang <[email protected]>

…rch-project#1691) * update vpc flow with flint-s3 based DDL assets and dashboard Signed-off-by: YANGDB <[email protected]> * update MV to use auto sync refresh Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_week_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_60min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * remove comments from MV queries Signed-off-by: YANGDB <[email protected]> * update to remove HIVE as not supported yet in current EMR version Signed-off-by: YANGDB <[email protected]> * update refresh sync rate Signed-off-by: YANGDB <[email protected]> * update watermark_delay Signed-off-by: YANGDB <[email protected]> * add live only workflow & dashboard add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]> * add live only workflow & dashboard add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]> * add live all only workflow & dashboard Signed-off-by: YANGDB <[email protected]> * correct table name hard coded issue Signed-off-by: YANGDB <[email protected]> * add vpc table creation based on CSV format Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Co-authored-by: Simeon Widdis <[email protected]>

…and dashboard (#1721) * update vpc flow with flint-s3 based DDL assets and dashboard (#1691) * update vpc flow with flint-s3 based DDL assets and dashboard Signed-off-by: YANGDB <[email protected]> * update MV to use auto sync refresh Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_week_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_60min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * remove comments from MV queries Signed-off-by: YANGDB <[email protected]> * update to remove HIVE as not supported yet in current EMR version Signed-off-by: YANGDB <[email protected]> * update refresh sync rate Signed-off-by: YANGDB <[email protected]> * update watermark_delay Signed-off-by: YANGDB <[email protected]> * add live only workflow & dashboard add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]> * add live only workflow & dashboard add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]> * add live all only workflow & dashboard Signed-off-by: YANGDB <[email protected]> * correct table name hard coded issue Signed-off-by: YANGDB <[email protected]> * add vpc table creation based on CSV format Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> * Change the vpc flow's refresh interval to 15mins and watermark delay to 1 min Signed-off-by: Ryan Liang <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Signed-off-by: Ryan Liang <[email protected]> Co-authored-by: YANGDB <[email protected]> Co-authored-by: Simeon Widdis <[email protected]>

…and dashboard (opensearch-project#1721) * update vpc flow with flint-s3 based DDL assets and dashboard (opensearch-project#1691) * update vpc flow with flint-s3 based DDL assets and dashboard Signed-off-by: YANGDB <[email protected]> * update MV to use auto sync refresh Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_30min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_live_week_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/vpc_agg_60min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]> * remove comments from MV queries Signed-off-by: YANGDB <[email protected]> * update to remove HIVE as not supported yet in current EMR version Signed-off-by: YANGDB <[email protected]> * update refresh sync rate Signed-off-by: YANGDB <[email protected]> * update watermark_delay Signed-off-by: YANGDB <[email protected]> * add live only workflow & dashboard add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]> * add live only workflow & dashboard add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]> * add live all only workflow & dashboard Signed-off-by: YANGDB <[email protected]> * correct table name hard coded issue Signed-off-by: YANGDB <[email protected]> * add vpc table creation based on CSV format Signed-off-by: YANGDB <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> * Change the vpc flow's refresh interval to 15mins and watermark delay to 1 min Signed-off-by: Ryan Liang <[email protected]> --------- Signed-off-by: YANGDB <[email protected]> Signed-off-by: Ryan Liang <[email protected]> Co-authored-by: YANGDB <[email protected]> Co-authored-by: Simeon Widdis <[email protected]> (cherry picked from commit e97df5f)

update vpc flow with flint-s3 based DDL assets and dashboard

bde5c8f

Signed-off-by: YANGDB <[email protected]>

YANG-DB requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, rupal-bq, mengweieric, vamsimanohar, Swiddis, penghuo, seankao-az, anirudha, paulstn and sumukhswamy as code owners April 15, 2024 04:31

RyanL1997 reviewed Apr 15, 2024

View reviewed changes

RyanL1997 added the backport 2.x label Apr 15, 2024

update MV to use auto sync refresh

503a8ab

Signed-off-by: YANGDB <[email protected]>

Swiddis mentioned this pull request Apr 15, 2024

Update pattern for multiple mview suffixes #1693

Merged

6 tasks

Swiddis added enhancement New feature or request integrations Used to denote items related to the Integrations project labels Apr 15, 2024

Swiddis mentioned this pull request Apr 15, 2024

Fix typo with VPC filters #1638

Closed

6 tasks

ps48 approved these changes Apr 15, 2024

View reviewed changes

RyanL1997 approved these changes Apr 15, 2024

View reviewed changes

Swiddis reviewed Apr 15, 2024

View reviewed changes

Swiddis approved these changes Apr 15, 2024

View reviewed changes

Swiddis requested changes Apr 16, 2024

View reviewed changes

YANG-DB and others added 2 commits April 15, 2024 18:32

Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/…

351442a

…assets/vpc_agg_30min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/…

027abf9

…assets/vpc_window-agg_60min_dest_ip_total-bytes_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

YANG-DB and others added 4 commits April 15, 2024 18:32

Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/…

2ea2ce4

…assets/vpc_window-agg_60min_dest_ip_cardinality_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/…

790d73b

…assets/vpc_live_week_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

Update server/adaptors/integrations/__data__/repository/aws_vpc_flow/…

54084e8

…assets/vpc_agg_60min_connections_mv-1.0.0.sql Co-authored-by: Simeon Widdis <[email protected]> Signed-off-by: YANGDB <[email protected]>

remove comments from MV queries

183fb8c

Signed-off-by: YANGDB <[email protected]>

YANG-DB added ux-integration ux related integration issues and removed enhancement New feature or request labels Apr 16, 2024

YANG-DB added 4 commits April 16, 2024 12:45

update to remove HIVE as not supported yet in current EMR version

a7887c1

Signed-off-by: YANGDB <[email protected]>

update refresh sync rate

945f531

Signed-off-by: YANGDB <[email protected]>

Merge remote-tracking branch 'origin/main' into flint-vpc-integration

db4e843

update watermark_delay

1f0455e

Signed-off-by: YANGDB <[email protected]>

Swiddis approved these changes Apr 16, 2024

View reviewed changes

YANG-DB added 5 commits April 16, 2024 17:25

add live only workflow & dashboard

a6c6b58

add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]>

add live only workflow & dashboard

d014b5c

add pre-agg + live workflow & dashboard add refresh-workflow for the pre-agg queries Signed-off-by: YANGDB <[email protected]>

add live all only workflow & dashboard

3158988

Signed-off-by: YANGDB <[email protected]>

correct table name hard coded issue

998c022

Signed-off-by: YANGDB <[email protected]>

Merge remote-tracking branch 'origin/main' into flint-vpc-integration

924aa5c

# Conflicts: # server/adaptors/integrations/__data__/repository/aws_vpc_flow/assets/create_mv_vpc-1.0.0.sql

YANG-DB removed the backport 2.x label Apr 17, 2024

RyanL1997 mentioned this pull request Apr 18, 2024

[Integrations] Add default refresh interval for all the integrations and correct the version on main #1717

Merged

6 tasks

add vpc table creation based on CSV format

0006110

Signed-off-by: YANGDB <[email protected]>

YANG-DB merged commit 71592c4 into opensearch-project:main Apr 18, 2024
9 of 19 checks passed

ps48 added the backport 2.x label Apr 18, 2024

opensearch-trigger-bot bot added the backport-failed label Apr 18, 2024

RyanL1997 mentioned this pull request Apr 18, 2024

[Manual Backport 2.x] update vpc flow with flint-s3 based DDL assets and dashboard #1721

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update vpc flow with flint-s3 based DDL assets and dashboard #1691

update vpc flow with flint-s3 based DDL assets and dashboard #1691

YANG-DB commented Apr 15, 2024

RyanL1997 left a comment

RyanL1997 Apr 15, 2024

RyanL1997 Apr 15, 2024

Swiddis commented Apr 15, 2024

Swiddis Apr 15, 2024

Swiddis left a comment

Swiddis left a comment

opensearch-trigger-bot bot commented Apr 18, 2024

		@@ -0,0 +1,75 @@
		--- DDL VPC create table statement definition

		@@ -0,0 +1,37 @@
		-- 30 Min Aggregation MV of VPC connections / bytes / packets

update vpc flow with flint-s3 based DDL assets and dashboard #1691

update vpc flow with flint-s3 based DDL assets and dashboard #1691

Conversation

YANG-DB commented Apr 15, 2024

Description

Check List

RyanL1997 left a comment

Choose a reason for hiding this comment

RyanL1997 Apr 15, 2024

Choose a reason for hiding this comment

RyanL1997 Apr 15, 2024

Choose a reason for hiding this comment

Swiddis commented Apr 15, 2024

Swiddis Apr 15, 2024

Choose a reason for hiding this comment

Swiddis left a comment

Choose a reason for hiding this comment

Swiddis left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Apr 18, 2024