Data quality next plans POC

.circleci/config.yml

@@ -26,7 +26,7 @@ jobs:
           name: run tests
           command: |
             . venv/bin/activate
-            python -m pytest --cov=hamilton tests/
+            python -m pytest --cov=hamilton tests/ --ignore tests/integrations
 
   build-py37:
     docker:
@@ -50,7 +50,7 @@ jobs:
           name: run tests
           command: |
             . venv/bin/activate
-            python -m pytest --cov=hamilton tests/
+            python -m pytest --cov=hamilton tests/ --ignore tests/integrations
 
   build-py38:
     docker:
@@ -74,7 +74,7 @@ jobs:
           name: run tests
           command: |
             . venv/bin/activate
-            python -m pytest --cov=hamilton tests/
+            python -m pytest --cov=hamilton tests/ --ignore tests/integrations
 
   build-py39:
     docker:
@@ -98,7 +98,7 @@ jobs:
           name: run tests
           command: |
             . venv/bin/activate
-            python -m pytest --cov=hamilton tests/
+            python -m pytest --cov=hamilton tests/ --ignore tests/integrations
 
   pre-commit:
     docker:
@@ -202,6 +202,86 @@ jobs:
           command: |
             . venv/bin/activate
             python -m pytest graph_adapter_tests/h_spark
+  integrations-py36:
+    docker:
+      - image: circleci/python:3.6
+    steps:
+      - checkout
+      - run:
+          name: install all python dependencies for integrations
+          command: |
+            python -m venv venv || virtualenv venv
+            . venv/bin/activate
+            python --version
+            pip --version
+            pip install -e .[pandera] # TODO -- add more as we add more integrations
+            pip install -r requirements-test.txt
+      # run tests!
+      - run:
+          name: run tests
+          command: |
+            . venv/bin/activate
+            python -m pytest tests/integrations
+  integrations-py37:
+    docker:
+      - image: circleci/python:3.7
+    steps:
+      - checkout
+      - run:
+          name: install all python dependencies for integrations
+          command: |
+            python -m venv venv || virtualenv venv
+            . venv/bin/activate
+            python --version
+            pip --version
+            pip install -e .[pandera] # TODO -- add more as we add more integrations
+            pip install -r requirements-test.txt
+      # run tests!
+      - run:
+          name: run tests
+          command: |
+            . venv/bin/activate
+            python -m pytest tests/integrations
+  integrations-py38:
+    docker:
+      - image: circleci/python:3.8
+    steps:
+      - checkout
+      - run:
+          name: install all python dependencies for integrations
+          command: |
+            python -m venv venv || virtualenv venv
+            . venv/bin/activate
+            python --version
+            pip --version
+            pip install -e .[pandera] # TODO -- add more as we add more integrations
+            pip install -r requirements-test.txt
+      # run tests!
+      - run:
+          name: run tests
+          command: |
+            . venv/bin/activate
+            python -m pytest tests/integrations
+  integrations-py39:
+    docker:
+      - image: circleci/python:3.9
+    steps:
+      - checkout
+      - run:
+          name: install all python dependencies for integrations
+          command: |
+            python -m venv venv || virtualenv venv
+            . venv/bin/activate
+            python --version
+            pip --version
+            pip install -e .[pandera] # TODO -- add more as we add more integrations
+            pip install -r requirements-test.txt
+      # run tests!
+      - run:
+          name: run tests
+          command: |
+            . venv/bin/activate
+            python -m pytest tests/integrations
 workflows:
   version: 2
   unit-test-workflow:
@@ -215,3 +295,7 @@ workflows:
       - dask-py37
       - ray-py37
       - spark-py38
+      - integrations-py36
+      - integrations-py37
+      - integrations-py38
+      - integrations-py39

data_quality.md

@@ -0,0 +1,85 @@
+# Data Quality
+
+Hamilton has a simple but powerful data quality capability. This enables you to write functions
+that have assertion on the output. For example...
+
+```python
+import pandas as pd
+import numpy as np
+from hamilton.function_modifiers import check_output
+
+@check_output(
+    datatype=np.int64,
+    data_in_range=(0,100),
+    importance="warn",
+)
+def some_int_data_between_0_and_100() -> pd.Series:
+    pass
+```
+
+In the above, we run two assertions:
+
+1. That the series has an np.int64 datatype
+2. That every item in the series in between 0 and 100
+
+Furthermore, the workflow does not fail when this dies. Rather, it logs a warning
+More about configuring that later, but you can see its specified in the `importance` parameter above.
+
+## Design
+
+To add data quality validation, we run an additional computational step in your workflow after function calculation.
+See comments on the `BaseDataValidationDecorator` class for how it works.
+
+## Default Validators
+
+The available default validators are listed in the variable `AVAILABLE_DEFAULT_VALIDATORS`
+in `default_validators.py`. To add more, please implement the class in that file then add to the list.
+There is a test that ensures that everything is added to that list.
+
+## Custom Validators
+
+To add a custom validator, you need to implement the class `DataValidator`. You can then use the
+`@check_output_custom` decorator to run it on a function. For example:
+
+```python
+import pandas as pd
+import numpy as np
+
+@check_output_custom(AllPrimeValidator(...))
+def prime_number_generator(number_of_primes_to_generate: int) -> pd.Series:
+    pass
+```
+
+## Urgency Levels
+
+Currently there are two available urgency level:
+
+1. "warn"
+2. "fail"
+
+They do exactly as you'd expect. "warn" logs the failure to the terminal and continues on. "fail"
+raises an exception in the final node.
+
+Limitations/future work are as follows:
+
+1. Currently the actions are hardcoded. In the future, we will be considering adding
+special actions for each level that one can customize...
+2. One can only disable data quality checks by commenting out the decorator. We intend to allow node-specific overrides.
+3. Currently the data quality results apply to every output of that function. E.G. if it runs `extract_columns`
+it executes on every column that's extracted.
+
+## Handling the results
+
+We utilize tags to index nodes that represent data quality. All data-quality related tags start with the
+prefix `hamilton.data_quality`. Currently there are two:
+
+1. `hamilton.data_quality.contains_dq_results` -- this is a boolean that tells
+whether a node outputs a data quality results. These are nodes that get injected when
+a node is decorated, and can be queried by the end user.
+2. `hamilton.data_quality.source_node` -- this contains the name of the source_node
+the data to which the data quality points.
+
+Note that these tags will not be present if the node is not related to data quality --
+don't assume they're in every node.
+
+To query one can simply filter for all the nodes that contain these tags and access the results!

decorators.md

@@ -248,3 +248,30 @@ desired_outputs = [o.name for o in all_possible_outputs
                    if 'my_tag_value' == o.tags.get('my_tag_key')]
 output = dr.execute(desired_outputs)
 ```
+
+## @check_output
+
+The `@check_output` decorator enables you to add simple data quality checks to your code.
+
+For example:
+
+```python
+import pandas as pd
+import numpy as np
+from hamilton.function_modifiers import check_output
+
+@check_output(
+    datatype=np.int64,
+    data_in_range=(0,100),
+)
+def some_int_data_between_0_and_100() -> pd.Series:
+    pass
+```
+
+The check_output validator takes in arguments that each correspond to one of the default validators.
+These arguments tell it to add the default validator to the list. The above thus creates
+two validators, one that checks the datatype of the series, and one that checks whether the data is in a certain range.
+
+Note that you can also specify custom decorators using the `@check_output_custom` decorator.
+
+See [data_quality](data_quality.md) for more information on available validators and how to build custom ones.