Overview
You can extend the functionality of Great Expectations by creating your own Custom Expectations. You can also enrich Great Expectations as a shared standard for data quality by contributing new Expectations to the open source project.
These processes compliment each other and their steps are streamlined so that one flows into the other. Once you have created a custom Expectation, it is simple to contribute it to the open source project. This section will teach you how to do both.
Prerequisites: This how-to guide assumes you have:
- Completed the Getting Started Tutorial
- Set up your dev environment
Creating Custom Expectation#
A fully-developed, Production-ready Expectation needs to do a lot of things:
- Execute consistently across many types of data infrastructure
- Render itself and its Validation Results into several formats
- Support Profiling against new data
- Be maintainable, with good tests, documentation, linting, type hints, etc.
In order to make development of Expectations as easy as possible, we've broken up the steps to create custom Expectations into a series of bite-sized steps. Each step can be completed in minutes. They can be completed (and contributed) incrementally, unlocking value at each step along the way.
Grouped together, they constitute a Definition of Done for Expectations at each Level of Maturity.
An Experimental Expectation...- Has a
library_metadataobject - Has a docstring, including a one-line short description
- Has at least one positive and negative example case, and all test cases pass
- Has core logic and passes tests on at least one Execution Engine
- Has basic input validation and type checking
- Has both Statement Renderers: prescriptive and diagnostic
- Has core logic that passes tests for all applicable Execution Engines
- Passes all linting checks
- Has a robust suite of tests, as determined by a code owner
- Has passed a manual review by a code owner for code standards and style guides
How these docs are organized#
The docs in Creating Custom Expectations focus on completing the four steps required for Experimental Expectations.
Completing them will leave your with a Custom Expectation that can be executed against one backend, with a couple tests to verify correctness, and a basic docstring and metadata to support diagonstics.
Optionally, you can also publish Custom Expectations to the Great Expectations open source gallery by following the steps here.
The code to achieve the first four steps looks somewhat different depending on the class of Expectation you're developing. Accordingly, there are separate how-to guides and templates for each class of Expectation.
| Guide: "How to create a custom..." | Template |
|---|---|
| Column Map Expectation | column_map_expectation_template |
| Column Aggregate Expectation | column_aggregate_expectation_template |
note
Not all classes of Expectation currently have guides and templates.
If you'd like to develop a different kind of Expectation, please reach out on Slack.
Beyond the first four steps, additional features are generally similar across all Expectation classes. Accordingly, most of the remaining steps have their own how-to guide in the Adding Features to Custom Expectations section of the table of contents.
| Step | Guide |
|---|---|
| Has basic input validation and type checking | How to add input validation and type checking for a Custom Expectation |
| Has both Statement Renderers: prescriptive and diagnostic | How to add Statement Renderers for Custom Expectations |
| Has core logic that passes tests for all applicable Execution Engines | How to add SQLAlchemy support for Custom Expectations How to add Spark support for Custom Expectations |
| Passes all linting checks | Great Expectations Code Style Guide: Linting |
The final two checks required for acceptance into the Great Expectations codebase at a Production level require manual review and guidance by a code owner.
If you believe your Custom Expectation is otherwise ready for contribution at a Production level, please submit a Pull Request, and we will work with you to ensure your Custom Expectation meets these standards.
Publishing your Expectation as an open source contribution#
You can find more detailed instructions in our guide on contributing Custom Expectations.