The logs provide little to no information on what's going on, the errors are so random that the last 200 people who had the same error all solved it differently. There are a gazillion parameters to configure and most of it is undocumented, tribal knowledge. /2

*org

I have a very specific scenario though I bet this has more general purpose use. Mine is about maintaining a lambda-architecture ingestion pipeline where the batch and the streaming tables can drift apart wrt to their schemas. We have a union-view on top of these tables which fails when this happens.

In Data, I often see this when an analyst is trying to fix that 0.0001% bad data in a dataset because it feels like a pebble in the shoe that you must get rid of.

Yeah. Np. We can and should surely have both. I just find it curious how different folks interpret data democracy differently.