Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARK-25280 Expose metrics for Prometheus #25

Merged
merged 3 commits into from
Nov 15, 2020
Merged

SPARK-25280 Expose metrics for Prometheus #25

merged 3 commits into from
Nov 15, 2020

Conversation

AnDyXX
Copy link
Contributor

@AnDyXX AnDyXX commented Nov 10, 2020 •
edited
Loading

Be default Prometheus metrics are disable. Will be exposed in out environments by changes in deployment file.
Locally for development purposes can be enabled by adding following environment variable to run configuration:

PROMETHEUS_METRICS=true

Example of metrics:

http://localhost:8080/prometheus

# HELP tomcat_sessions_active_current_sessions  
# TYPE tomcat_sessions_active_current_sessions gauge
tomcat_sessions_active_current_sessions{application="not specified",} 0.0
# HELP hikaricp_connections_acquire_seconds Connection acquire time
# TYPE hikaricp_connections_acquire_seconds summary
hikaricp_connections_acquire_seconds_count{application="not specified",pool="HikariPool-1",} 0.0
hikaricp_connections_acquire_seconds_sum{application="not specified",pool="HikariPool-1",} 0.0
# HELP hikaricp_connections_acquire_seconds_max Connection acquire time
# TYPE hikaricp_connections_acquire_seconds_max gauge
hikaricp_connections_acquire_seconds_max{application="not specified",pool="HikariPool-1",} 0.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads{application="not specified",} 28.0
# HELP tomcat_cache_hit_total  
# TYPE tomcat_cache_hit_total counter
tomcat_cache_hit_total{application="not specified",} 0.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage{application="not specified",} 0.0
# HELP hikaricp_connections_active Active connections
# TYPE hikaricp_connections_active gauge
hikaricp_connections_active{application="not specified",pool="HikariPool-1",} 0.0
# HELP tomcat_servlet_request_seconds  
# TYPE tomcat_servlet_request_seconds summary
tomcat_servlet_request_seconds_count{application="not specified",name="default",} 0.0
tomcat_servlet_request_seconds_sum{application="not specified",name="default",} 0.0
tomcat_servlet_request_seconds_count{application="not specified",name="dispatcherServlet",} 1.0
tomcat_servlet_request_seconds_sum{application="not specified",name="dispatcherServlet",} 0.0
# HELP tomcat_threads_busy_threads  
# TYPE tomcat_threads_busy_threads gauge
tomcat_threads_busy_threads{application="not specified",name="http-nio-8080",} 1.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{application="not specified",id="mapped",} 0.0
jvm_buffer_total_capacity_bytes{application="not specified",id="direct",} 16384.0
jvm_buffer_total_capacity_bytes{application="not specified",id="mapped - 'non-volatile memory'",} 0.0
# HELP tomcat_sessions_expired_sessions_total  
# TYPE tomcat_sessions_expired_sessions_total counter
tomcat_sessions_expired_sessions_total{application="not specified",} 0.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds{application="not specified",} 1.60499588782E9
# HELP hikaricp_connections_idle Idle connections
# TYPE hikaricp_connections_idle gauge
hikaricp_connections_idle{application="not specified",pool="HikariPool-1",} 10.0
# HELP hikaricp_connections_max Max connections
# TYPE hikaricp_connections_max gauge
hikaricp_connections_max{application="not specified",pool="HikariPool-1",} 10.0
# HELP jvm_gc_live_data_size_bytes Size of old generation memory pool after a full GC
# TYPE jvm_gc_live_data_size_bytes gauge
jvm_gc_live_data_size_bytes{application="not specified",} 0.0
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage{application="not specified",} 0.0
# HELP number_of_developers  
# TYPE number_of_developers gauge
number_of_developers{application="not specified",} 0.0
# HELP tomcat_sessions_created_sessions_total  
# TYPE tomcat_sessions_created_sessions_total counter
tomcat_sessions_created_sessions_total{application="not specified",} 0.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{application="not specified",id="mapped",} 0.0
jvm_buffer_count_buffers{application="not specified",id="direct",} 2.0
jvm_buffer_count_buffers{application="not specified",id="mapped - 'non-volatile memory'",} 0.0
# HELP tomcat_global_received_bytes_total  
# TYPE tomcat_global_received_bytes_total counter
tomcat_global_received_bytes_total{application="not specified",name="http-nio-8080",} 0.0
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds{application="not specified",} 15.184
# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files{application="not specified",} 128.0
# HELP tomcat_cache_access_total  
# TYPE tomcat_cache_access_total counter
tomcat_cache_access_total{application="not specified",} 0.0
# HELP tomcat_global_sent_bytes_total  
# TYPE tomcat_global_sent_bytes_total counter
tomcat_global_sent_bytes_total{application="not specified",name="http-nio-8080",} 0.0
# HELP tomcat_threads_config_max_threads  
# TYPE tomcat_threads_config_max_threads gauge
tomcat_threads_config_max_threads{application="not specified",name="http-nio-8080",} 200.0
# HELP tomcat_threads_current_threads  
# TYPE tomcat_threads_current_threads gauge
tomcat_threads_current_threads{application="not specified",name="http-nio-8080",} 10.0
# HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC
# TYPE jvm_gc_memory_promoted_bytes_total counter
jvm_gc_memory_promoted_bytes_total{application="not specified",} 3.9186944E7
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{application="not specified",id="mapped",} 0.0
jvm_buffer_memory_used_bytes{application="not specified",id="direct",} 16384.0
jvm_buffer_memory_used_bytes{application="not specified",id="mapped - 'non-volatile memory'",} 0.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads{application="not specified",} 24.0
# HELP number_of_maintainers  
# TYPE number_of_maintainers gauge
number_of_maintainers{application="not specified",} 1.0
# HELP hikaricp_connections_timeout_total Connection timeout total count
# TYPE hikaricp_connections_timeout_total counter
hikaricp_connections_timeout_total{application="not specified",pool="HikariPool-1",} 0.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{application="not specified",area="nonheap",id="CodeHeap 'non-nmethods'",} 1315584.0
jvm_memory_used_bytes{application="not specified",area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 2176256.0
jvm_memory_used_bytes{application="not specified",area="heap",id="G1 Eden Space",} 1048576.0
jvm_memory_used_bytes{application="not specified",area="nonheap",id="Metaspace",} 5.741452E7
jvm_memory_used_bytes{application="not specified",area="heap",id="G1 Survivor Space",} 217464.0
jvm_memory_used_bytes{application="not specified",area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.2156928E7
jvm_memory_used_bytes{application="not specified",area="nonheap",id="Compressed Class Space",} 6483120.0
jvm_memory_used_bytes{application="not specified",area="heap",id="G1 Old Gen",} 3.5188736E7
# HELP jdbc_connections_min Minimum number of idle connections in the pool.
# TYPE jdbc_connections_min gauge
jdbc_connections_min{application="not specified",name="dataSource",} 10.0
# HELP tomcat_servlet_error_total  
# TYPE tomcat_servlet_error_total counter
tomcat_servlet_error_total{application="not specified",name="default",} 0.0
tomcat_servlet_error_total{application="not specified",name="dispatcherServlet",} 0.0
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads{application="not specified",} 31.0
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total{application="not specified",} 0.0
# HELP logback_events_total Number of error level events that made it to the logs
# TYPE logback_events_total counter
logback_events_total{application="not specified",level="info",} 16.0
logback_events_total{application="not specified",level="trace",} 0.0
logback_events_total{application="not specified",level="error",} 0.0
logback_events_total{application="not specified",level="warn",} 1.0
logback_events_total{application="not specified",level="debug",} 14.0
# HELP tomcat_sessions_active_max_sessions  
# TYPE tomcat_sessions_active_max_sessions gauge
tomcat_sessions_active_max_sessions{application="not specified",} 0.0
# HELP hikaricp_connections_pending Pending threads
# TYPE hikaricp_connections_pending gauge
hikaricp_connections_pending{application="not specified",pool="HikariPool-1",} 0.0
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{application="not specified",area="nonheap",id="CodeHeap 'non-nmethods'",} 5840896.0
jvm_memory_max_bytes{application="not specified",area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 1.22908672E8
jvm_memory_max_bytes{application="not specified",area="heap",id="G1 Eden Space",} -1.0
jvm_memory_max_bytes{application="not specified",area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{application="not specified",area="heap",id="G1 Survivor Space",} -1.0
jvm_memory_max_bytes{application="not specified",area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.22908672E8
jvm_memory_max_bytes{application="not specified",area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{application="not specified",area="heap",id="G1 Old Gen",} 4.294967296E9
# HELP tomcat_sessions_alive_max_seconds  
# TYPE tomcat_sessions_alive_max_seconds gauge
tomcat_sessions_alive_max_seconds{application="not specified",} 0.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes{application="not specified",} 9896.0
# HELP jvm_gc_pause_seconds Time spent in GC pause
# TYPE jvm_gc_pause_seconds summary
jvm_gc_pause_seconds_count{action="end of minor GC",application="not specified",cause="G1 Evacuation Pause",} 16.0
jvm_gc_pause_seconds_sum{action="end of minor GC",application="not specified",cause="G1 Evacuation Pause",} 0.069
jvm_gc_pause_seconds_count{action="end of minor GC",application="not specified",cause="Metadata GC Threshold",} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",application="not specified",cause="Metadata GC Threshold",} 0.006
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
# TYPE jvm_gc_pause_seconds_max gauge
jvm_gc_pause_seconds_max{action="end of minor GC",application="not specified",cause="G1 Evacuation Pause",} 0.009
jvm_gc_pause_seconds_max{action="end of minor GC",application="not specified",cause="Metadata GC Threshold",} 0.006
# HELP tomcat_servlet_request_max_seconds  
# TYPE tomcat_servlet_request_max_seconds gauge
tomcat_servlet_request_max_seconds{application="not specified",name="default",} 0.0
tomcat_servlet_request_max_seconds{application="not specified",name="dispatcherServlet",} 0.0
# HELP jvm_gc_max_data_size_bytes Max size of old generation memory pool
# TYPE jvm_gc_max_data_size_bytes gauge
jvm_gc_max_data_size_bytes{application="not specified",} 4.294967296E9
# HELP jdbc_connections_max Maximum number of active connections that can be allocated at the same time.
# TYPE jdbc_connections_max gauge
jdbc_connections_max{application="not specified",name="dataSource",} 10.0
# HELP hikaricp_connections_usage_seconds Connection usage time
# TYPE hikaricp_connections_usage_seconds summary
hikaricp_connections_usage_seconds_count{application="not specified",pool="HikariPool-1",} 2.0
hikaricp_connections_usage_seconds_sum{application="not specified",pool="HikariPool-1",} 0.107
# HELP hikaricp_connections_usage_seconds_max Connection usage time
# TYPE hikaricp_connections_usage_seconds_max gauge
hikaricp_connections_usage_seconds_max{application="not specified",pool="HikariPool-1",} 0.105
# HELP number_of_applications  
# TYPE number_of_applications gauge
number_of_applications{application="not specified",} 0.0
# HELP tomcat_global_request_max_seconds  
# TYPE tomcat_global_request_max_seconds gauge
tomcat_global_request_max_seconds{application="not specified",name="http-nio-8080",} 0.0
# HELP hikaricp_connections_min Min connections
# TYPE hikaricp_connections_min gauge
hikaricp_connections_min{application="not specified",pool="HikariPool-1",} 10.0
# HELP tomcat_global_error_total  
# TYPE tomcat_global_error_total counter
tomcat_global_error_total{application="not specified",name="http-nio-8080",} 0.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count{application="not specified",} 8.0
# HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the young generation memory pool after one GC to before the next
# TYPE jvm_gc_memory_allocated_bytes_total counter
jvm_gc_memory_allocated_bytes_total{application="not specified",} 2.55852544E8
# HELP hikaricp_connections_creation_seconds_max Connection creation time
# TYPE hikaricp_connections_creation_seconds_max gauge
hikaricp_connections_creation_seconds_max{application="not specified",pool="HikariPool-1",} 0.0
# HELP hikaricp_connections_creation_seconds Connection creation time
# TYPE hikaricp_connections_creation_seconds summary
hikaricp_connections_creation_seconds_count{application="not specified",pool="HikariPool-1",} 0.0
hikaricp_connections_creation_seconds_sum{application="not specified",pool="HikariPool-1",} 0.0
# HELP jdbc_connections_idle Number of established but idle connections.
# TYPE jdbc_connections_idle gauge
jdbc_connections_idle{application="not specified",name="dataSource",} 10.0
# HELP tomcat_sessions_rejected_sessions_total  
# TYPE tomcat_sessions_rejected_sessions_total counter
tomcat_sessions_rejected_sessions_total{application="not specified",} 0.0
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m{application="not specified",} 6.9794921875
# HELP tomcat_global_request_seconds  
# TYPE tomcat_global_request_seconds summary
tomcat_global_request_seconds_count{application="not specified",name="http-nio-8080",} 0.0
tomcat_global_request_seconds_sum{application="not specified",name="http-nio-8080",} 0.0
# HELP hikaricp_connections Total connections
# TYPE hikaricp_connections gauge
hikaricp_connections{application="not specified",pool="HikariPool-1",} 10.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files{application="not specified",} 10240.0
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{application="not specified",area="nonheap",id="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_committed_bytes{application="not specified",area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 2555904.0
jvm_memory_committed_bytes{application="not specified",area="heap",id="G1 Eden Space",} 5.24288E7
jvm_memory_committed_bytes{application="not specified",area="nonheap",id="Metaspace",} 5.933056E7
jvm_memory_committed_bytes{application="not specified",area="heap",id="G1 Survivor Space",} 1048576.0
jvm_memory_committed_bytes{application="not specified",area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.2189696E7
jvm_memory_committed_bytes{application="not specified",area="nonheap",id="Compressed Class Space",} 7249920.0
jvm_memory_committed_bytes{application="not specified",area="heap",id="G1 Old Gen",} 7.0254592E7
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{application="not specified",state="runnable",} 11.0
jvm_threads_states_threads{application="not specified",state="new",} 0.0
jvm_threads_states_threads{application="not specified",state="terminated",} 0.0
jvm_threads_states_threads{application="not specified",state="blocked",} 0.0
jvm_threads_states_threads{application="not specified",state="waiting",} 11.0
jvm_threads_states_threads{application="not specified",state="timed-waiting",} 6.0
# HELP jdbc_connections_active Current number of active connections that have been allocated from the data source.
# TYPE jdbc_connections_active gauge
jdbc_connections_active{application="not specified",name="dataSource",} 0.0

@AnDyXX AnDyXX requested a review from a user November 10, 2020 08:12
alewir
alewir reviewed Nov 12, 2020
View reviewed changes
...tore-metadata-service/src/main/java/com/lgi/appstore/metadata/api/stb/StbAppsController.java Outdated
@@ -37,6 +38,7 @@
import javax.validation.Valid;
import java.util.Optional;

//@Timed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this stay commented out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I will remove this.

Left from my experiments with metrics

kubagruszka
kubagruszka requested changes Nov 12, 2020
View reviewed changes
...re-metadata-service/src/main/java/com/lgi/appstore/metadata/api/metrics/MetricsProvider.java Outdated
* limitations under the License.
*/
@Component
public class MetricsProvider {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm skepticall on the purpose and implementation of those metrics. If will have multiple instances of ASMS they all will make redundant calls and unnecessary stress the database.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK,

Will remove these.

@AnDyXX
Copy link
Contributor Author

AnDyXX commented Nov 13, 2020

@kubagruszka @alewir I made changes according to your comments. Could review PR again?

@AnDyXX AnDyXX merged commit 96aadf2 into master Nov 15, 2020
@AnDyXX AnDyXX deleted the feature/SPARK-25280 branch November 15, 2020 11:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alewir alewir alewir approved these changes

@kubagruszka kubagruszka kubagruszka approved these changes

@karolostaszewski karolostaszewski Awaiting requested review from karolostaszewski

@djasiok djasiok Awaiting requested review from djasiok

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AnDyXX @alewir @kubagruszka