Skip to content

Commit

Permalink
[DOCS] Updates Custom Expectations Overview (#4237)
Browse files Browse the repository at this point in the history
* linking and verbiage

* fixes to linting doc
  • Loading branch information
austiezr authored Feb 18, 2022
1 parent 217fd8f commit bae6310
Show file tree
Hide file tree
Showing 2 changed files with 35 additions and 25 deletions.
26 changes: 16 additions & 10 deletions docs/contributing/style_guides/code_style.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,29 @@ This style guide will be enforced for all incoming PRs. However, certain legacy

* **Docstrings are highly recommended.** We use the Sphinx’s [Napoleon extension](http://www.sphinx-doc.org/en/master/ext/napoleon.html) to build documentation from Google-style docstrings.

* **Lint your code.** Our CI system will check using `black`, `isort`, `flake8` and `pyupgrade`. - Linting with `isort` MUST occur from a virtual environment that has all required packages installed, and pre-commit uses the virtual environment from which it was installed, whether or not that environment is active when making the commit. So, **before running ``pre-commit install`` ensure you have activated a virtual environment that has all development requirements installed.**
### Linting

````console
Our CI system will check using `black`, `isort`, `flake8` and `pyupgrade`.

Linting with `isort` MUST occur from a virtual environment that has all required packages installed, and pre-commit uses the virtual environment from which it was installed, whether or not that environment is active when making the commit.

This means you must ensure you have activated a virtual environment that has all development requirements installed **before running `pre-commit install`**.

```console
pre-commit uninstall
# ACTIVATE ENV, e.g.: conda activate pre_commit_env OR source pre_commit_env/bin/activate
pip install -r requirements-dev.txt
pre-commit install --install-hooks
````
```

* If you have already committed files but are seeing errors during the continuous integration tests, you can run tests manually:
If you have already committed files but are seeing errors during the continuous integration tests, you can run tests manually:

````console
black .
isort . --check-only --skip docs
flake8 great_expectations/core
pyupgrade --py3-plus
````
```console
black <PATH/TO/YOUR/CHANGES>
isort <PATH/TO/YOUR/CHANGES> --check-only --skip docs
flake8 <PATH/TO/YOUR/CHANGES>
pyupgrade <PATH/TO/YOUR/CHANGES> --py3-plus
```

### Expectations

Expand Down
34 changes: 19 additions & 15 deletions docs/guides/expectations/creating_custom_expectations/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@ title: Overview
---
import Prerequisites from './components/prerequisites.jsx'

You can extend the functionality of Great Expectations by creating your own custom [**Expectations**](../../../reference/expectations/expectations.md). You can also enrich Great Expectations as a shared standard for data quality by contributing new Expectations to the open source project.
You can extend the functionality of Great Expectations by creating your own Custom [**Expectations**](../../../reference/expectations/expectations.md). You can also enrich Great Expectations as a shared standard for data quality by contributing new Expectations to the open source project.

These processes compliment each other and their steps are streamlined so that one flows into the other. Once you have created a custom Expectation, it is simple to contribute it to the open source project. This section will teach you how to do both.

<Prerequisites></Prerequisites>
<Prerequisites>
</Prerequisites>

## Steps to create a custom Expectation
## Creating Custom Expectation

A fully-developed, Production-ready Expectation needs to do a lot of things:
* Execute consistently across many types of data infrastructure
Expand All @@ -32,7 +33,7 @@ Grouped together, they constitute a Definition of Done for Expectations at each
<i class="fas fa-circle" style={{color: "#ffc107"}}></i> A Beta Expectation...

* Has basic input validation and type checking
* Has all three Statement Renderers: descriptive, prescriptive, diagnostic
* Has both Statement Renderers: prescriptive and diagnostic
<!-- * Has default `Parameter Builders` and Domain hooks to support Profiling -->
* Has core logic that passes tests for all applicable Execution Engines

Expand All @@ -43,9 +44,12 @@ Grouped together, they constitute a Definition of Done for Expectations at each
* Has a robust suite of tests, as determined by a code owner
* Has passed a manual review by a code owner for code standards and style guides

## How these docs are organized
### How these docs are organized

The docs in `Creating Custom Expectations` focus on completing the four steps for Experimental Expectations. Completing them will get to the point where your Expectation can be executed against one backend, with a couple tests to verify correctness, and a basic docstring and metadata to support diagonstics. Optionally, you can also publish experimental Expectations to the [Great Expectations open source gallery](https://greatexpectations.io/expectations) by following the steps [here](./overview.md#publishing-your-expectation-as-an-open-source-contribution).
The docs in `Creating Custom Expectations` focus on completing the four steps required for Experimental Expectations.
Completing them will leave your with a Custom Expectation that can be executed against one backend, with a couple tests to verify correctness, and a basic docstring and metadata to support diagonstics.

Optionally, you can also publish Custom Expectations to the [Great Expectations open source gallery](https://greatexpectations.io/expectations) by following the steps [here](../contributing/how_to_contribute_a_custom_expectation_to_great_expectations.md).

The code to achieve the first four steps looks somewhat different depending on the class of Expectation you're developing. Accordingly, there are separate how-to guides and templates for each class of Expectation.

Expand All @@ -59,19 +63,19 @@ Not all classes of Expectation currently have guides and templates. <br></br>
If you'd like to develop a different kind of Expectation, please reach out on [Slack](https://greatexpectations.io/slack).
:::

Beyond the first four steps, additional features are generally similar across all Expectation classes. Accordingly, each of the remaining steps has its own how-to guide in the `Adding Features to Custom Expectations` section of the table of contents.
Beyond the first four steps, additional features are generally similar across all Expectation classes. Accordingly, most of the remaining steps have their own how-to guide in the `Adding Features to Custom Expectations` section of the table of contents.

| Step | Guide |
|------|-------|
| Has basic input validation and type checking | [How to add configuration validation for an Expectation](../features_custom_expectations/how_to_add_input_validation_for_an_expectation.md) |
| Has all three Statement Renderers: descriptive, prescriptive, diagnostic | [How to add text renderers for custom Expectations](../features_custom_expectations/how_to_add_statement_renderers_for_an_expectation.md) |
| Has core logic that passes tests for all applicable Execution Engines | [How to add SQLAlchemy support for custom Metrics](../features_custom_expectations/how_to_add_sqlalchemy_support_for_an_expectation.md)<br/> [How to add Spark support for custom Metrics](../features_custom_expectations/how_to_add_spark_support_for_an_expectation.md)|
| Has basic input validation and type checking | [How to add input validation and type checking for a Custom Expectation](../features_custom_expectations/how_to_add_input_validation_for_an_expectation.md) |
| Has both Statement Renderers: prescriptive and diagnostic | [How to add Statement Renderers for Custom Expectations](../features_custom_expectations/how_to_add_statement_renderers_for_an_expectation.md) |
| Has core logic that passes tests for all applicable Execution Engines | [How to add SQLAlchemy support for Custom Expectations](../features_custom_expectations/how_to_add_sqlalchemy_support_for_an_expectation.md)<br/> [How to add Spark support for Custom Expectations](../features_custom_expectations/how_to_add_spark_support_for_an_expectation.md)|
| Passes all linting checks | |
| Has a robust suite of tests, as determined by a code owner | |
| Has passed a manual review by a code owner for code standards and style guides | |

## Publishing your Expectation as an open source contribution
The final two checks required for acceptance into the Great Expectations codebase at a Production level require manual review and guidance by a code owner.

If you believe your Custom Expectation is otherwise ready for contribution at a Production level, please submit a [Pull Request](https://github.com/great-expectations/great_expectations/pulls), and we will work with you to ensure your Custom Expectation meets these standards.

You can find more detailed instructions in the [Contributing](../../../contributing/contributing.md) section of the docs.
### Publishing your Expectation as an open source contribution

## Wrapping up
You can find more detailed instructions in our guide on [contributing Custom Expectations](../contributing/how_to_contribute_a_custom_expectation_to_great_expectations.md).

0 comments on commit bae6310

Please sign in to comment.