Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] [Detections] Improves custom query rule upgrade test #114454

Merged
merged 6 commits into from
Oct 13, 2021
Merged

[Security Solution] [Detections] Improves custom query rule upgrade test #114454

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b36822c
improves upgrade test
MadameSheema Oct 11, 2021
cfc70c2
Merge branch 'master' into improves-upgrade-test
kibanamachine Oct 11, 2021
7109ef3
fixes type check issues
MadameSheema Oct 11, 2021
c1eedcd
Merge branch 'improves-upgrade-test' of github.com:MadameSheema/kiban…
MadameSheema Oct 11, 2021
934b325
Merge branch 'master' into improves-upgrade-test
kibanamachine Oct 12, 2021
627b95a
Merge branch 'master' into improves-upgrade-test
kibanamachine Oct 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions x-pack/plugins/security_solution/cypress/screens/alerts_detection_rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,3 +103,7 @@ export const RULES_DELETE_CONFIRMATION_MODAL = '[data-test-subj="allRulesDeleteC
export const MODAL_CONFIRMATION_BTN = '[data-test-subj="confirmModalConfirmButton"]';

export const RULE_DETAILS_DELETE_BTN = '[data-test-subj="rules-details-delete-rule"]';

export const ALERT_DETAILS_CELLS = '[data-test-subj="dataGridRowCell"]';

export const SERVER_SIDE_EVENT_COUNT = '[data-test-subj="server-side-event-count"]';
6 changes: 6 additions & 0 deletions x-pack/plugins/security_solution/cypress/tasks/common.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

import { esArchiverResetKibana } from './es_archiver';
import { RuleEcs } from '../../common/ecs/rule';
import { LOADING_INDICATOR } from '../screens/security_header';

const primaryButton = 0;

Expand Down Expand Up @@ -155,3 +156,8 @@ export const deleteCases = () => {
};

export const scrollToBottom = () => cy.scrollTo('bottom');

export const waitForPageToBeLoaded = () => {
cy.get(LOADING_INDICATOR).should('exist');
cy.get(LOADING_INDICATOR).should('not.exist');
};
149 changes: 131 additions & 18 deletions x-pack/plugins/security_solution/cypress/upgrade_integration/custom_query_rule.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,138 @@
* 2.0.
*/

import { RULE_NAME } from '../screens/alerts_detection_rules';

import { ALERT_DETAILS_CELLS, SERVER_SIDE_EVENT_COUNT } from '../screens/alerts_detection_rules';
import {
goToManageAlertsDetectionRules,
waitForAlertsIndexToBeCreated,
waitForAlertsPanelToBeLoaded,
} from '../tasks/alerts';
import { waitForRulesTableToBeLoaded } from '../tasks/alerts_detection_rules';
import { loginAndWaitForPageWithoutDateRange } from '../tasks/login';

import { ALERTS_URL } from '../urls/navigation';

describe('After an upgrade, the cusom query rule', () => {
it('Displays the rule', function () {
loginAndWaitForPageWithoutDateRange(ALERTS_URL);
waitForAlertsPanelToBeLoaded();
waitForAlertsIndexToBeCreated();
goToManageAlertsDetectionRules();
ADDITIONAL_LOOK_BACK_DETAILS,
ABOUT_DETAILS,
ABOUT_RULE_DESCRIPTION,
CUSTOM_QUERY_DETAILS,
DEFINITION_DETAILS,
getDetails,
INDEX_PATTERNS_DETAILS,
RISK_SCORE_DETAILS,
RULE_NAME_HEADER,
RULE_TYPE_DETAILS,
RUNS_EVERY_DETAILS,
SCHEDULE_DETAILS,
SEVERITY_DETAILS,
TIMELINE_TEMPLATE_DETAILS,
} from '../screens/rule_details';

import { waitForPageToBeLoaded } from '../tasks/common';
import { waitForRulesTableToBeLoaded, goToRuleDetails } from '../tasks/alerts_detection_rules';
import { loginAndWaitForPage } from '../tasks/login';

import { DETECTIONS_RULE_MANAGEMENT_URL } from '../urls/navigation';

const EXPECTED_NUMBER_OF_ALERTS = '1';

const alert = {
rule: 'Custom query rule for upgrade',
severity: 'low',
riskScore: '7',
reason:
'file event with process test, file The file to test, by Security Solution on security-solution.local created low alert Custom query rule for upgrade.',
hostName: 'security-solution.local',
username: 'test',
processName: 'The file to test',
fileName: 'The file to test',
sourceIp: '127.0.0.1',
destinationIp: '127.0.0.2',
};

const rule = {
customQuery: '*:*',
name: 'Custom query rule for upgrade',
description: 'My description',
index: ['auditbeat-*'],
severity: 'Low',
riskScore: '7',
timelineTemplate: 'none',
runsEvery: '10s',
lookBack: '179999990s',
timeline: 'None',
};

describe('After an upgrade, the custom query rule', () => {
before(() => {
loginAndWaitForPage(DETECTIONS_RULE_MANAGEMENT_URL);
waitForRulesTableToBeLoaded();
cy.get(RULE_NAME).should('have.text', 'Custom query rule for upgrade');
goToRuleDetails();
waitForPageToBeLoaded();
});

it('Has the expected alerts number', () => {
cy.get(SERVER_SIDE_EVENT_COUNT).contains(EXPECTED_NUMBER_OF_ALERTS);
});

it('Displays the rule details', () => {
cy.get(RULE_NAME_HEADER).should('contain', `${rule.name}`);
cy.get(ABOUT_RULE_DESCRIPTION).should('have.text', rule.description);
cy.get(ABOUT_DETAILS).within(() => {
getDetails(SEVERITY_DETAILS).should('have.text', rule.severity);
getDetails(RISK_SCORE_DETAILS).should('have.text', rule.riskScore);
});
cy.get(DEFINITION_DETAILS).within(() => {
getDetails(INDEX_PATTERNS_DETAILS).should('have.text', rule.index.join(''));
getDetails(CUSTOM_QUERY_DETAILS).should('have.text', rule.customQuery);
getDetails(RULE_TYPE_DETAILS).should('have.text', 'Query');
getDetails(TIMELINE_TEMPLATE_DETAILS).should('have.text', rule.timeline);
});
cy.get(SCHEDULE_DETAILS).within(() => {
getDetails(RUNS_EVERY_DETAILS).should('have.text', rule.runsEvery);
getDetails(ADDITIONAL_LOOK_BACK_DETAILS).should('have.text', rule.lookBack);
});
});

it('Displays the alert details', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there are no explicit assertions in this test (just .contains and .then), what is the behavior here when an element is not present?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using here the contains instead of the have.text assertion, because the elements are returning something like elementText Row1 Column1. I tried different ways to get the text I wanted but was not able to do it.

Contains yields to the element that has the text we are expecting, then, we scroll left using the keyword. We have to scroll, because not all the elements are displayed on the screen making them not accessible for Cypress.

In case that the element is not present, the test fails (see below screenshot).

Screenshot 2021-10-13 at 15 23 39

I would be more than happy to run a pair-programming session to find a better solution :)

cy.get(ALERT_DETAILS_CELLS).first().focus();
cy.get(ALERT_DETAILS_CELLS).first().type('{rightarrow}');
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.rule)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.severity)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.riskScore)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.reason)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.hostName)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.username)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.processName)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.fileName)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS)
.contains(alert.sourceIp)
.then(($el) => {
cy.wrap($el).type('{rightarrow}');
});
cy.get(ALERT_DETAILS_CELLS).contains(alert.destinationIp);
});
});
1 change: 1 addition & 0 deletions x-pack/plugins/security_solution/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
"cypress:open": "yarn cypress open --config-file ./cypress/cypress.json",
"cypress:open:ccs": "yarn cypress:open --config integrationFolder=./cypress/ccs_integration",
"cypress:open-as-ci": "node ../../../scripts/functional_tests --config ../../test/security_solution_cypress/visual_config.ts",
"cypress:open:upgrade": "yarn cypress:open --config integrationFolder=./cypress/upgrade_integration",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should document this new script in our README.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. The overall README of the upgrade tests is still in progress.

The nature of these upgrade tests is completely different from the ones we are use to write. Everything is originated on the elastic stack repo, it creates a cloud deployment, generates the test data needed, and then upgrades to a different version. All of these, are done using gradle tasks.

Once all those steps are finished, it executes our upgrade cypress tests.

The best practices here are going to be slightly different from the ones we tend to use. I.E. We are not going to be able to clean the data between tests, since the time cost of spin-up an instance and upgrade it is huge.

At this point, I don't know what is the best way of writing these types of tests (as well as the data creation) for us and what should be the best practices. This is why I'm adding new tests, in order to explore and find it.

"cypress:run": "yarn cypress:run:reporter --browser chrome --spec './cypress/integration/**/*.spec.ts'; status=$?; yarn junit:merge && exit $status",
"cypress:run:firefox": "yarn cypress:run:reporter --browser firefox --spec './cypress/integration/**/*.spec.ts'; status=$?; yarn junit:merge && exit $status",
"cypress:run:reporter": "yarn cypress run --config-file ./cypress/cypress.json --reporter ../../../node_modules/cypress-multi-reporters --reporter-options configFile=./cypress/reporter_config.json",
Expand Down