Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MINOR - Reconfigure app metrics #15460

Merged
merged 3 commits into from
Mar 7, 2024
Merged

Conversation

pmbrull
Copy link
Collaborator

@pmbrull pmbrull commented Mar 5, 2024

Describe your changes:

Rel to https://github.com/open-metadata/openmetadata-collate/issues/355

  • Keeping all jvm_* metrics
  • Removing latency histogram, and just keeping percentiles
    http_latency_requests_seconds{quantile="0.99",} 0.200802304
    http_latency_requests_seconds{quantile="0.9",} 0.108527616
    http_latency_requests_seconds_count 7.0
    http_latency_requests_seconds_sum 0.562955292
    
  • Removing unnecessary web analytics metrics (handled now separated via telemetry app)
  • Cleaned up the http_server_requests histogram. Now we only have
    http_server_requests_sec_bucket{method="GET",le="0.01",} 1.0
    http_server_requests_sec_bucket{method="GET",le="0.1",} 5.0
    http_server_requests_sec_bucket{method="GET",le="1.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="2.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="5.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="10.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="20.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="60.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="300.0",} 7.0
    http_server_requests_sec_bucket{method="GET",le="+Inf",} 7.0
    http_server_requests_sec_count{method="GET",} 7.0
    http_server_requests_sec_sum{method="GET",} 0.562788041
    
  • Adding JDBI queries histogram & latency:
    jdbi_requests_seconds_bucket{le="0.01",} 944.0
    jdbi_requests_seconds_bucket{le="0.1",} 944.0
    jdbi_requests_seconds_bucket{le="1.0",} 944.0
    jdbi_requests_seconds_bucket{le="2.0",} 944.0
    jdbi_requests_seconds_bucket{le="5.0",} 944.0
    jdbi_requests_seconds_bucket{le="10.0",} 944.0
    jdbi_requests_seconds_bucket{le="20.0",} 944.0
    jdbi_requests_seconds_bucket{le="60.0",} 944.0
    jdbi_requests_seconds_bucket{le="300.0",} 944.0
    jdbi_requests_seconds_bucket{le="+Inf",} 944.0
    jdbi_requests_seconds_count 944.0
    jdbi_requests_seconds_sum 0.0
    
    jdbi_latency_requests_seconds{quantile="0.99",} 0.0
    jdbi_latency_requests_seconds{quantile="0.9",} 0.0
    jdbi_latency_requests_seconds_count 944.0
    jdbi_latency_requests_seconds_sum 0.0
    

I worked on ... because ...

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@pmbrull pmbrull requested a review from a team as a code owner March 5, 2024 13:49
@github-actions github-actions bot added backend Ingestion safe to test Add this label to run secure Github workflows on PRs labels Mar 5, 2024
@@ -263,7 +263,7 @@ eventMonitoringConfiguration:
eventMonitor: ${EVENT_MONITOR:-prometheus} # Possible values are "prometheus", "cloudwatch"
batchSize: ${EVENT_MONITOR_BATCH_SIZE:-10}
pathPattern: ${EVENT_MONITOR_PATH_PATTERN:-["/api/v1/tables/*", "/api/v1/health-check"]}
latency: ${EVENT_MONITOR_LATENCY:-[]} # For value p99=0.99, p90=0.90, p50=0.50 etc.
latency: ${EVENT_MONITOR_LATENCY:-[0.99, 0.90]} # For value p99=0.99, p90=0.90, p50=0.50 etc.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keeping these defaults here. Seem like sensible values

Timer.builder("latency_requests")
.description("Request latency in seconds.")
private static final double[] latencyBuckets = new double[]
{.01, .1, 1, 2, 5, 10, 20, 60, 300};
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LMK if you see the need on having these as a config cc @tutte

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static is ok, we can remove the 300 as the HTTP should kill every request beyond 60s

Copy link
Contributor

github-actions bot commented Mar 5, 2024

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

Copy link

cypress bot commented Mar 5, 2024

20 failed tests on run #33331 ↗︎

20 703 12 0 Flakiness 0

Details:

Format
Project: openmetadata Commit: e8bedff7e1
Status: Errored Duration: 15:22 💡
Started: Mar 5, 2024 2:34 PM Ended: Mar 5, 2024 3:49 PM
Failed  Service/ServiceIngestion.spec.ts • 3 failed tests • cypress-ci-mysql-pr-8157724417-1

View Output Video

Test Artifacts
Superset Ingestion > Create & Ingest Superset service Screenshots Video
Superset Ingestion > Update schedule options and verify Screenshots Video
Superset Ingestion > Delete Superset service Screenshots Video
Failed  Pages/Glossary.spec.js • 7 failed tests • cypress-ci-mysql-pr-8157724417-1

View Output Video

Test Artifacts
Glossary page should work properly > Approval Workflow for Glossary Term Screenshots Video
Glossary page should work properly > Update glossary term Screenshots Video
Glossary page should work properly > Request Tags workflow for Glossary Screenshots Video
Glossary page should work properly > Assets Tab should work properly Screenshots Video
Glossary page should work properly > Remove Glossary term from entity should work properly Screenshots Video
Glossary page should work properly > Delete glossary term should work properly Screenshots Video
Glossary page should work properly > Delete glossary should work properly Screenshots Video
Failed  Pages/Customproperties.spec.js • 1 failed test • cypress-ci-mysql-pr-8157724417-1

View Output Video

Test Artifacts
Custom Properties should work properly > Custom properties for glossary and glossary terms > Add update and delete String, Integer, Markdown custom properties for glossary term Screenshots Video
Failed  Pages/DataQualityAndProfiler.spec.js • 1 failed test • cypress-ci-mysql-pr-8157724417-1

View Output Video

Test Artifacts
Data Quality and Profiler should work properly > Update profiler setting modal Screenshots Video
Failed  Pages/Teams.spec.js • 1 failed test • cypress-ci-mysql-pr-8157724417-1

View Output Video

Test Artifacts
Teams flow should work properly > Update description for created team Screenshots Video

The first 5 failed specs are shown, see all 9 specs in Cypress Cloud.

Review all test suite changes for PR #15460 ↗︎

Copy link

sonarqubecloud bot commented Mar 7, 2024

Quality Gate Passed Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

@pmbrull pmbrull merged commit 5b24277 into open-metadata:main Mar 7, 2024
18 of 21 checks passed
pmbrull added a commit that referenced this pull request Mar 21, 2024
* MINOR - Reconfigure app metrics

* Format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Ingestion safe to test Add this label to run secure Github workflows on PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants