We also added a new way to chain multiple flat-map calls into a single Timely operator. Coming from Rust's iterator API, it's sometimes not obvious that Timely's API has a different cost structure (each transform renders an operator). Here, a builder can mitigate some of the cost.

I think these change are a great example of taking existing APIs and moving from row-first to batch-of-data. Computers generally seem better at handling data by column rather than row, and these changes allow Timely programs to represent data in their own form.

A Distributor trait takes the concepts of Exchange, but moves away from row-by-row processing, opening up new kinds of containers that aren't row-first. For example, what is a row in a trie-structured container? The new trait allows to partition by whatever the user wants, not just by a "row".

Previously, the Container trait described behavior that is now separate: Accountable reveals a container's record count for progress tracking; nothing else. For higher-level APIs, IterContainer and DrainContainer reveal their contents (non-)destructively, for container-agnostic row-by-row operators.

github.com/release-plz/... for handling releases. GitHub CI for builds.

Jan '24: 1,338 kWh. That was a newly-built apartment in NYC. I don't miss this part!

Check out namespace.so

It'll certainly be an opt-in feature in its current form. Forcing a certain deployment pattern will be hard. Otoh, we could automate graceful replica restarts!

So, we didn't solve the underlying problem, but mitigated it for many of our workloads.

Now, in Materialize time advanced about every second, and the result is high CPU utilization 😿 But: we restart most things every week, so updates beyond a week will never be surfaced! The mitigation we implemented is to drop such updates, and add some safeguards to never produce incorrect results. 🎉

Future updates are stashed at the input to indexes, and they'll only act on them once the time has advanced far enough. Because we use Differential Dataflows, the updates are only partially ordered. To extract ready updates, DD scans the data and splits it in ready and pending, ~ in linear time.