Added in-flight cancellation of SearchShardTask based on resource consumption #4575

ketanv3 · 2022-09-23T08:37:55Z

Description

This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance.

This PR is intended to add the core framework for in-flight cancellation of search requests. The actual implementations of ResourceUsageTrackers (which decide if a task should be cancelled), along with the node stats API will be added in a later PR.

Issues Resolved

#1181

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma [email protected]

github-actions · 2022-09-23T09:04:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/3334/
CommitID: 4456edd

codecov-commenter · 2022-09-23T09:06:43Z

Codecov Report

Merging #4575 (6424470) into main (5d7d83e) will increase coverage by 0.15%.
The diff coverage is 77.92%.

@@             Coverage Diff              @@
##               main    #4575      +/-   ##
============================================
+ Coverage     70.60%   70.75%   +0.15%     
- Complexity    57603    57794     +191     
============================================
  Files          4675     4687      +12     
  Lines        276925   277290     +365     
  Branches      40347    40376      +29     
============================================
+ Hits         195517   196193     +676     
+ Misses        65156    64779     -377     
- Partials      16252    16318      +66

Impacted Files	Coverage Δ
...ain/java/org/opensearch/cluster/ClusterModule.java	`100.00% <ø> (ø)`
...rg/opensearch/common/settings/ClusterSettings.java	`91.89% <ø> (ø)`
...ain/java/org/opensearch/tasks/CancellableTask.java	`82.35% <ø> (-5.89%)`	⬇️
...backpressure/settings/SearchShardTaskSettings.java	`51.42% <51.42%> (ø)`
...kpressure/settings/SearchBackpressureSettings.java	`67.34% <67.34%> (ø)`
...arch/backpressure/settings/NodeDuressSettings.java	`70.00% <70.00%> (ø)`
...n/java/org/opensearch/common/util/TokenBucket.java	`74.28% <74.28%> (ø)`
...search/backpressure/SearchBackpressureService.java	`77.31% <77.31%> (ø)`
.../opensearch/tasks/TaskResourceTrackingService.java	`80.00% <81.81%> (+0.23%)`	⬆️
...in/java/org/opensearch/tasks/TaskCancellation.java	`89.65% <89.65%> (ø)`
... and 502 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Bukhtawar

Thanks @ketanv3 I have shared few thoughts on how we could break down the core logic into separate classes for better maintainability.

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureSettings.java

server/src/main/java/org/opensearch/search/backpressure/TaskCancellation.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureManager.java

Bukhtawar · 2022-09-23T16:30:09Z

server/src/main/java/org/opensearch/common/util/MovingAverage.java

+    public synchronized double record(long value) {
+        long delta = value - observations[(int) (count % observations.length)];
+        observations[(int) (count % observations.length)] = value;
+
+        count++;
+        sum += delta;
+        average = (double) sum / Math.min(count, observations.length);
+        return average;
+    }


Can we use a queue and see if we can avoid the synchronized block by using CAS if possible

I actually plan to benchmark CAS vs synchronized approaches before committing to either one of them, especially since the operations are pretty simple and quick to execute.

If there are major gains with CAS then it makes sense to go with it. Otherwise, I would prefer keeping it simple and readable.

I did not observe any major gains with queue + CAS approach as followed in indexing back-pressure (here). With higher thread contention, it performed slightly poorly.

# JMH version: 1.35 # VM version: JDK 17.0.3, OpenJDK 64-Bit Server VM, 17.0.3+7 Benchmark (1 thread - no contention) Mode Cnt Score Error Units MovingAverageBenchmark.timeMovingAverage avgt 5 25.669 ± 1.884 ns/op MovingAverageBenchmark.timeMovingAverageQueue avgt 5 25.213 ± 0.383 ns/op Benchmark (4 threads - low contention) Mode Cnt Score Error Units MovingAverageBenchmark.timeMovingAverage avgt 5 217.714 ± 6.676 ns/op MovingAverageBenchmark.timeMovingAverageQueue avgt 5 223.088 ± 3.651 ns/op Benchmark (16 threads - high contention) Mode Cnt Score Error Units MovingAverageBenchmark.timeMovingAverage avgt 5 785.830 ± 13.446 ns/op MovingAverageBenchmark.timeMovingAverageQueue avgt 5 792.442 ± 64.234 ns/op

It is also worth noting that the current implementation of moving average in shard indexing back-pressure has subtle race-condition bugs. Two or more concurrent threads could reach this point and remove excessive elements from the queue leading to incorrect results, or even NPE in the worst case. Another problem is lack of causal ordering – an older thread may overwrite a new thread's average at this point.

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureManager.java

github-actions · 2022-09-24T13:43:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3359/
CommitID: caad78d

github-actions · 2022-09-24T14:35:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/3360/
CommitID: ef01ba3

github-actions · 2022-09-24T15:01:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/3361/
CommitID: 9d0f2f0

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

server/src/main/java/org/opensearch/tasks/TaskCancellation.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

github-actions · 2022-10-04T14:09:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/3756/
CommitID: 1f33317

github-actions · 2022-10-04T14:39:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/3758/
CommitID: 6445b4a

nssuresh2007

LGTM!

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

server/src/main/java/org/opensearch/common/Streak.java

Bukhtawar · 2022-10-09T06:50:08Z

server/src/main/java/org/opensearch/common/util/MovingAverage.java

+     */
+    public synchronized double record(long value) {
+        long delta = value - observations[(int) (count % observations.length)];
+        observations[(int) (count % observations.length)] = value;
+
+        count++;
+        sum += delta;
+        average = (double) sum / Math.min(count, observations.length);
+        return average;
+    }


Can we use CAS here with a do-while loop

Using CAS backed by a queue or a ring-buffer, it may be possible to track the moving average similar to how it's done in indexing backpressure (though the current implementation has subtle race-condition bugs which I have highlighted in an above comment).

To successfully implement this, we need to ensure the last 'n' items, running sum, and count of inserted items are updated atomically; which may not be possible with CAS alone. We need to treat the entire state (even the backing queue/buffer) as immutable and create a new copy with every update (similar concept to CopyOnWriteArrayList).

Our use-case is write heavy (on task completion) with infrequent reads (on search backpressure service iteration), creating copies may be very expensive especially for larger window sizes. I'm still inclined to use the current approach.

Benchmark results comparing the existing approach v/s CAS backed by an immutable ring-buffer.

Using synchronized (current approach):

10 observations 100 observations 1000 observations

1 thread 14.123 ns/op 14.258 ns/op 14.544 ns/op

4 threads 368.087 ns/op 364.879 ns/op 378.378 ns/op

16 threads 1400.703 ns/op 1506.456 ns/op 1809.835 ns/op

Using compare-and-set backed by an immutable ring-buffer (implementation reference):

10 observations 100 observations 1000 observations

1 thread 27.438 ns/op 112.920 ns/op 1082.057 ns/op

4 threads 798.966 ns/op 1077.040 ns/op 4675.375 ns/op

16 threads 5820.276 ns/op 8374.579 ns/op 36474.605 ns/op

Key observations:

Synchronized approach doesn't have to clone the backing buffer, so, the time complexity doesn't grow proportional to the window size. This is a major drawback with the CAS approach as repeated array copy is expensive and creates a lot of GC overhead too.

Even with smaller window sizes, the performance of CAS becomes poorer as the number of threads (and the contention) grows.

Synchronized approach performed between 2–20x better for these measurements.

Bukhtawar · 2022-10-09T06:51:47Z

server/src/main/java/org/opensearch/common/util/TokenBucket.java

+        synchronized (this) {
+            refill();
+
+            if (tokens >= n) {
+                tokens -= n;
+                return true;
+            }
+
+            return false;


Can we use CAS here as well?

This should be doable. Will make the changes.

Bukhtawar · 2022-10-09T06:53:13Z

server/src/test/java/org/opensearch/common/util/TokenBucketTests.java

+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.LongSupplier;
+
+public class TokenBucketTests extends OpenSearchTestCase {


We might need multi-threaded tests for guaranteeing thread safety on all new utilities we are building

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

server/src/main/java/org/opensearch/search/backpressure/trackers/NodeResourceUsageTracker.java

Bukhtawar · 2022-10-09T11:34:35Z

server/src/main/java/org/opensearch/search/backpressure/trackers/TaskResourceUsageTracker.java

+    public long incrementCancellations() {
+        return cancellations.incrementAndGet();
+    }
+
+    public long getCancellations() {
+        return cancellations.get();
+    }


Can we delegate this to TaskCancellation instead

Here we are tracking the total cancellations done by a task resource usage tracker.

On the other hand, TaskCancellation is a wrapper for a single cancellable task along with the list of cancellation reasons and callbacks. The callbacks in it are responsible for calling incrementCancellations() on the relevant trackers.

Snippet for reference:

TaskCancellation getTaskCancellation(CancellableTask task) { List<TaskCancellation.Reason> reasons = new ArrayList<>(); List<Runnable> callbacks = new ArrayList<>(); for (TaskResourceUsageTracker tracker : taskResourceUsageTrackers) { Optional<TaskCancellation.Reason> reason = tracker.cancellationReason(task); if (reason.isPresent()) { reasons.add(reason.get()); callbacks.add(tracker::incrementCancellations); } } return new TaskCancellation(task, reasons, callbacks); }

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureState.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

github-actions · 2022-10-10T12:40:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4247/
CommitID: 64b3123

github-actions · 2022-10-10T12:58:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4246/
CommitID: 489d4f7

github-actions · 2022-10-10T13:04:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4248/
CommitID: 8d46e74

github-actions · 2022-10-10T23:24:49Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/4341/
CommitID: 69a2ff8

Bukhtawar

Thanks @ketanv3 please add links to follow ups for

Multi-threaded tests
Coordinator cancellation at parent

ketanv3 · 2022-10-13T05:48:04Z

Links for pending tasks:

[Search Backpressure] Add multi-threaded tests for newly added common/utils #4765
[Search Backpressure] Generalize TaskCancellation to support parent task cancellation at the coordinator #4766

github-actions · 2022-10-13T14:06:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4582/
CommitID: 7f8ae89

…sumption This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. Signed-off-by: Ketan Verma <[email protected]>

github-actions · 2022-10-14T05:39:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4679/
CommitID: 2421979

github-actions · 2022-10-14T05:48:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4680/
CommitID: 3c19186

github-actions · 2022-10-14T06:18:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4685/
CommitID: bfbebef

github-actions · 2022-10-14T06:53:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/4691/
CommitID: 6282515

Signed-off-by: Ketan Verma <[email protected]>

github-actions · 2022-10-14T07:31:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/4696/
CommitID: 6424470

…on resource consumption (opensearch-project#4575) This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. Signed-off-by: Ketan Verma <[email protected]>

…ource consumption (#5039) * [Backport 2.x] Added in-flight cancellation of SearchShardTask based on resource consumption (#4575) This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. * [Backport 2.x] Added resource usage trackers for in-flight cancellation of SearchShardTask (#4805) 1. CpuUsageTracker: cancels tasks if they consume too much CPU 2. ElapsedTimeTracker: cancels tasks if they consume too much time 3. HeapUsageTracker: cancels tasks if they consume too much heap * [Backport 2.x]Added search backpressure stats API Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <[email protected]>

…ource consumption (opensearch-project#5039) * [Backport 2.x] Added in-flight cancellation of SearchShardTask based on resource consumption (opensearch-project#4575) This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. * [Backport 2.x] Added resource usage trackers for in-flight cancellation of SearchShardTask (opensearch-project#4805) 1. CpuUsageTracker: cancels tasks if they consume too much CPU 2. ElapsedTimeTracker: cancels tasks if they consume too much time 3. HeapUsageTracker: cancels tasks if they consume too much heap * [Backport 2.x]Added search backpressure stats API Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <[email protected]>

…ource consumption (#5039) * [Backport 2.x] Added in-flight cancellation of SearchShardTask based on resource consumption (#4575) This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. * [Backport 2.x] Added resource usage trackers for in-flight cancellation of SearchShardTask (#4805) 1. CpuUsageTracker: cancels tasks if they consume too much CPU 2. ElapsedTimeTracker: cancels tasks if they consume too much time 3. HeapUsageTracker: cancels tasks if they consume too much heap * [Backport 2.x]Added search backpressure stats API Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <[email protected]> (cherry picked from commit 7c521b9)

…ource consumption (#5039) (#5058) * [Backport 2.x] Added in-flight cancellation of SearchShardTask based on resource consumption (#4575) This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. * [Backport 2.x] Added resource usage trackers for in-flight cancellation of SearchShardTask (#4805) 1. CpuUsageTracker: cancels tasks if they consume too much CPU 2. ElapsedTimeTracker: cancels tasks if they consume too much time 3. HeapUsageTracker: cancels tasks if they consume too much heap * [Backport 2.x]Added search backpressure stats API Added search backpressure stats to the existing node/stats API to describe: 1. the number of cancellations (currently for SearchShardTask only) 2. the current state of TaskResourceUsageTracker Signed-off-by: Ketan Verma <[email protected]> (cherry picked from commit 7c521b9) Co-authored-by: Ketan Verma <[email protected]>

…sumption (opensearch-project#4575) * Added in-flight cancellation of SearchShardTask based on resource consumption This feature aims to identify and cancel resource intensive SearchShardTasks if they have breached certain thresholds. This will help in terminating problematic queries which can put nodes in duress and degrade the cluster performance. Signed-off-by: Ketan Verma <[email protected]>

ketanv3 marked this pull request as ready for review September 23, 2022 09:10

ketanv3 requested review from a team and reta as code owners September 23, 2022 09:10

Bukhtawar reviewed Sep 23, 2022

View reviewed changes

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 4456edd to caad78d Compare September 24, 2022 13:17

ketanv3 requested a review from Bukhtawar September 24, 2022 13:46

ketanv3 force-pushed the feature/inflight-cancellation-core branch from caad78d to ef01ba3 Compare September 24, 2022 14:09

ketanv3 force-pushed the feature/inflight-cancellation-core branch from ef01ba3 to 9d0f2f0 Compare September 24, 2022 14:37

nssuresh2007 reviewed Sep 25, 2022

View reviewed changes

nssuresh2007 reviewed Sep 26, 2022

View reviewed changes

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java Show resolved Hide resolved

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java Show resolved Hide resolved

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 9d0f2f0 to 1f33317 Compare October 4, 2022 13:43

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 1f33317 to 6445b4a Compare October 4, 2022 14:13

ketanv3 requested review from nssuresh2007 and Bukhtawar and removed request for Bukhtawar and nssuresh2007 October 4, 2022 20:08

nssuresh2007 approved these changes Oct 7, 2022

View reviewed changes

Bukhtawar reviewed Oct 9, 2022

View reviewed changes

ketanv3 force-pushed the feature/inflight-cancellation-core branch 3 times, most recently from 64b3123 to 8d46e74 Compare October 10, 2022 12:38

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 8d46e74 to 69a2ff8 Compare October 10, 2022 22:57

Bukhtawar approved these changes Oct 11, 2022

View reviewed changes

ketanv3 mentioned this pull request Oct 13, 2022

[Search Backpressure] Add multi-threaded tests for newly added common/utils #4765

Open

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 2421979 to 3c19186 Compare October 14, 2022 05:18

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 3c19186 to bfbebef Compare October 14, 2022 05:51

ketanv3 force-pushed the feature/inflight-cancellation-core branch from bfbebef to 6282515 Compare October 14, 2022 06:19

Addressed PR comments

6424470

Signed-off-by: Ketan Verma <[email protected]>

ketanv3 force-pushed the feature/inflight-cancellation-core branch from 6282515 to 6424470 Compare October 14, 2022 07:01

Bukhtawar merged commit d708860 into opensearch-project:main Oct 14, 2022

ketanv3 mentioned this pull request Nov 2, 2022

[Backport 2.x] In-flight cancellation of SearchShardTask based on resource consumption #5039

Merged

6 tasks

PritLadani mentioned this pull request Nov 9, 2022

Add search backpressure cancellation at the coordinator level #5173

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added in-flight cancellation of SearchShardTask based on resource consumption #4575

Added in-flight cancellation of SearchShardTask based on resource consumption #4575

ketanv3 commented Sep 23, 2022

github-actions bot commented Sep 23, 2022

codecov-commenter commented Sep 23, 2022 •

edited

Loading

Bukhtawar left a comment

Bukhtawar Sep 23, 2022

ketanv3 Sep 23, 2022

ketanv3 Oct 2, 2022 •

edited

Loading

github-actions bot commented Sep 24, 2022

github-actions bot commented Sep 24, 2022

github-actions bot commented Sep 24, 2022

github-actions bot commented Oct 4, 2022

github-actions bot commented Oct 4, 2022

nssuresh2007 left a comment

Bukhtawar Oct 9, 2022

ketanv3 Oct 10, 2022 •

edited

Loading

ketanv3 Oct 10, 2022

Bukhtawar Oct 9, 2022

ketanv3 Oct 10, 2022

Bukhtawar Oct 9, 2022

Bukhtawar Oct 9, 2022

ketanv3 Oct 10, 2022

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 10, 2022

github-actions bot commented Oct 10, 2022

Bukhtawar left a comment

ketanv3 commented Oct 13, 2022

github-actions bot commented Oct 13, 2022

github-actions bot commented Oct 14, 2022

github-actions bot commented Oct 14, 2022

github-actions bot commented Oct 14, 2022

github-actions bot commented Oct 14, 2022

github-actions bot commented Oct 14, 2022

	10 observations	100 observations	1000 observations
1 thread	14.123 ns/op	14.258 ns/op	14.544 ns/op
4 threads	368.087 ns/op	364.879 ns/op	378.378 ns/op
16 threads	1400.703 ns/op	1506.456 ns/op	1809.835 ns/op

	10 observations	100 observations	1000 observations
1 thread	27.438 ns/op	112.920 ns/op	1082.057 ns/op
4 threads	798.966 ns/op	1077.040 ns/op	4675.375 ns/op
16 threads	5820.276 ns/op	8374.579 ns/op	36474.605 ns/op

Added in-flight cancellation of SearchShardTask based on resource consumption #4575

Added in-flight cancellation of SearchShardTask based on resource consumption #4575

Conversation

ketanv3 commented Sep 23, 2022

Description

Issues Resolved

Check List

github-actions bot commented Sep 23, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Sep 23, 2022 • edited Loading

Codecov Report

Bukhtawar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ketanv3 Oct 2, 2022 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Sep 24, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 24, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 24, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2022

Gradle Check (Jenkins) Run Completed with:

nssuresh2007 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ketanv3 Oct 10, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 10, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 10, 2022

Gradle Check (Jenkins) Run Completed with:

Bukhtawar left a comment

Choose a reason for hiding this comment

ketanv3 commented Oct 13, 2022

github-actions bot commented Oct 13, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 14, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 14, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 14, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 14, 2022

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 14, 2022

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Sep 23, 2022 •

edited

Loading

ketanv3 Oct 2, 2022 •

edited

Loading

ketanv3 Oct 10, 2022 •

edited

Loading