GE with Airflow -- Marshmallow incompatability

General Discussion
#1

I’m running into the following incompatibility with GE and Airflow:
Great Expectations requires a marshmallow >3.0, while airflow wants marshmallow < 3.0.
Specifically, pip install great_expectations installs marshmallow 3.5.2, and pip install apache-airflow uninstalls marshmallow 3.5.2 and installs marshmallow 2.21.0.
Then when I try to run great_expectations (Even just great_expectations --version), I get:
" ImportError: cannot import name ‘INCLUDE’ from ‘marshmallow’ ".
If, after installing apache-airflow, I uninstall and reinstall great_expectations, it uninstalls marshmallow 2.21.0 and installs marshmallow 3.5.2.
Then when I try to import great_expectations in a python script, it attempts to fill up the DagBag and write grammar tables, which fails. I can still run airflow, it just gives the same “writing failed” message every time.

Steps to reproduce the behavior:

  1. pip install great_expectations
  2. pip install apache-airflow
  3. great_expectations --version
  4. pip uninstall great_expectations
  5. pip install great_expectations
  6. in a python script: import great_expectations as ge

It sounds like the workaround right now is not running them in the same virtualenv, although maybe a ge package compatible with marshmallow 2.3 could be possible.

1 Like
#2

The best answer at the moment is not using airflow and Great Expectations in the same virtual environment.

This could take the shape as virtual environments you manage yourself, or perhaps the airflow.operators.python_operator.PythonVirtualenvOperator: https://airflow.apache.org/docs/stable/_api/airflow/operators/python_operator/index.html#airflow.operators.python_operator.PythonVirtualenvOperator

Internally Great Expectations uses features in recent versions of marshmallow, and we are not likely to reverse that decision in the near future.

1 Like