Can we use Great Expectations to quarantine all the bad data into a separate log file or table so it can be addressed in a cleaning process outside of the normal dag for all the good data?
Joining threads…
Here’s a previous thread that is a bit out of date on this topic:
There’s also this more recent post about seeing all the records with anomalies:
Also, here’s a toy example for doing this using pandas. It is not implemented for sql.