Came to accept this is just natural but never not thinking about Hayao Miyazaki’s creative process.

(7/10) It seems in practise open DynamoDB Streams can spawn thousands of empty shards for no apparent reason, making what should be simple stream processing surprisingly complex. We run this on a prod table with a relatively fair load. We realised we ended up processing thousands of empty shards.

(6/10) There are multiple approaches to reading from streams. Preferred ones are via #lambdas or the #kinesis adaptor. For various reasons this wouldn't work for us. We are left with the manual approach of polling the streams ourselves. According to the docs this should have been pretty easy to do.

(5/10) Any #dynamodb table with dynamodb streams enabled can have one dynamodb stream active. At any point in time, we can only have one active stream. A #stream can have multiple shards. A shard can contain zero or more records. A record is the change that we want to be captured.

(2/10) So this is more on lessons learnt when pocs and balancing trade offs aren't enough. Our only requirement: almost real time audit logs on all dynamodb tables. Possible approach - record any db change via events and the transactional outbox pattern.

The two level alg achieves O(√n) for insert and search in worst complexity. But this can be improved.

In 1988, Randy Brown introduced Calendar Queues (CQ), which is organized like a calendar, divided into "buckets" (like days or time slots). Each bucket represents a fixed-width time interval.

The Two-Level (TL) algorithm proposed by Franta and Maly consists of 3 layers:
- the index list layer - divides the range of event times into equal-sized intervals.
- the secondary key layer - points to sublists of event notices
- the event notices layer - holds the event times

The original cron was designed by Ken Thompson in Unix V7 (~1979), later rewritten by Dennis Ritchie. It would read the crontab files, used an in memory table for scheduling and had one main process that woke up every min, check if any commands need to be run, and if so, run them as root. [1/*]

Part of the journey on attempting to better understand how the real world works - wrote a guide on how to decode #QRcodes. Next articles will dive into how the #encoding / #decoding part works from #Hamming to #Reed-Solomon codes.

thecraftydev.substack.com/p/from-pixel...