I’m running into the following incompatibility with GE and Airflow:
Great Expectations requires a marshmallow >3.0, while airflow wants marshmallow < 3.0.
Specifically, pip install great_expectations installs marshmallow 3.5.2, and pip install apache-airflow uninstalls marshmallow 3.5.2 and installs marshmallow 2.21.0.
Then when I try to run great_expectations (Even just great_expectations --version), I get:
" ImportError: cannot import name ‘INCLUDE’ from ‘marshmallow’ ".
If, after installing apache-airflow, I uninstall and reinstall great_expectations, it uninstalls marshmallow 2.21.0 and installs marshmallow 3.5.2.
Then when I try to import great_expectations in a python script, it attempts to fill up the DagBag and write grammar tables, which fails. I can still run airflow, it just gives the same “writing failed” message every time.
Steps to reproduce the behavior:
- pip install great_expectations
- pip install apache-airflow
- great_expectations --version
- pip uninstall great_expectations
- pip install great_expectations
- in a python script: import great_expectations as ge
It sounds like the workaround right now is not running them in the same virtualenv, although maybe a ge package compatible with marshmallow 2.3 could be possible.