Chimney Sweepers Local 420
@bobbby.online
I make silly and frivolous stuff for fun, usually w/software. I like #ElixirLang. Live music, tech, weird markets, and urbanism chatter too. Often wrong
📍NYC
‘82 Jamaica Music Fest Docu person
@eastcoastemo.bsky.social's other half
📍NYC
‘82 Jamaica Music Fest Docu person
@eastcoastemo.bsky.social's other half
If it wouldn’t have been this actually interesting work thing I would have invented some shit to write for my website that did something dumb lol
November 21, 2025 at 5:31 AM
If it wouldn’t have been this actually interesting work thing I would have invented some shit to write for my website that did something dumb lol
Oh I know but I really didn’t wanna be alone with my thoughts tn lol
November 21, 2025 at 5:30 AM
Oh I know but I really didn’t wanna be alone with my thoughts tn lol
This is not healthy don't be like me. I needed to obsess over a problem so that I did not think about the various tire fires that surround me in my actual life.
November 21, 2025 at 4:49 AM
This is not healthy don't be like me. I needed to obsess over a problem so that I did not think about the various tire fires that surround me in my actual life.
We also have an entire system in ETS to take data from Postgres, then curry it up with data from Apache Druid, and then be able to create the ultimately-final record that represents the true "app state" that is meaningful and also has a structure of its "data state" it came from if needed
November 21, 2025 at 4:24 AM
We also have an entire system in ETS to take data from Postgres, then curry it up with data from Apache Druid, and then be able to create the ultimately-final record that represents the true "app state" that is meaningful and also has a structure of its "data state" it came from if needed
Overall I think its pretty suitable and the real concerns come when you are worrying about deploys on say, 3 k8s pods running the same thing vs doing native Erlang clustering etc
I've done it all the ways.
I've done it all the ways.
November 21, 2025 at 4:23 AM
Overall I think its pretty suitable and the real concerns come when you are worrying about deploys on say, 3 k8s pods running the same thing vs doing native Erlang clustering etc
I've done it all the ways.
I've done it all the ways.
Routinely, this is caching certain things from tables with billions of records keeping up to millions itself. Sure, RAM can be an issue, but like, you can also do "tricks" lol like compress large values and decompress when needed and keep storage in-transit and in-function low or whatever.
November 21, 2025 at 4:23 AM
Routinely, this is caching certain things from tables with billions of records keeping up to millions itself. Sure, RAM can be an issue, but like, you can also do "tricks" lol like compress large values and decompress when needed and keep storage in-transit and in-function low or whatever.
But yeah, I can shoot the shit about how we leverage ETS in production if you wanna shoot me a DM. It is quite a good setup for us and it is embedded in just about every part of the application in various ways. And we do it at a pretty big scale.
November 21, 2025 at 4:23 AM
But yeah, I can shoot the shit about how we leverage ETS in production if you wanna shoot me a DM. It is quite a good setup for us and it is embedded in just about every part of the application in various ways. And we do it at a pretty big scale.
a data warehouse that services like 50 engineering teams to a thing that then fits into our one suite of apps that then also translates that into a typed model representing a combination of Postgres, druid, and ETS, and then make all that live and reactive and not mutable, is like, hard.
November 21, 2025 at 4:21 AM
a data warehouse that services like 50 engineering teams to a thing that then fits into our one suite of apps that then also translates that into a typed model representing a combination of Postgres, druid, and ETS, and then make all that live and reactive and not mutable, is like, hard.
5. I am the first person to ever use this library in a real application and not a simple demo showing its props at a basic level etc
So in reality, ETS is the most sane and fast part of all of this, and has been 0 issue.
Now, the fact we have data models that have to represent things from
So in reality, ETS is the most sane and fast part of all of this, and has been 0 issue.
Now, the fact we have data models that have to represent things from
November 21, 2025 at 4:21 AM
5. I am the first person to ever use this library in a real application and not a simple demo showing its props at a basic level etc
So in reality, ETS is the most sane and fast part of all of this, and has been 0 issue.
Now, the fact we have data models that have to represent things from
So in reality, ETS is the most sane and fast part of all of this, and has been 0 issue.
Now, the fact we have data models that have to represent things from
2. there were problems with how the old contract handled the values from the APIs we have and their shape and the "new world" while much nicer requires a bit of archeology to get converted over
3. There was a subtle bug in the ETL process that cost me a day
4. The remaining pieces are undocumented
3. There was a subtle bug in the ETL process that cost me a day
4. The remaining pieces are undocumented
November 21, 2025 at 4:21 AM
2. there were problems with how the old contract handled the values from the APIs we have and their shape and the "new world" while much nicer requires a bit of archeology to get converted over
3. There was a subtle bug in the ETL process that cost me a day
4. The remaining pieces are undocumented
3. There was a subtle bug in the ETL process that cost me a day
4. The remaining pieces are undocumented
of some page state using a new UI library that has dumb stateless components and needs to work with a parameterized, typed model of things to structure what goes into the dumb components correctly.
The problem here has been a few things:
1. We were not getting the URL params totally right
The problem here has been a few things:
1. We were not getting the URL params totally right
November 21, 2025 at 4:21 AM
of some page state using a new UI library that has dumb stateless components and needs to work with a parameterized, typed model of things to structure what goes into the dumb components correctly.
The problem here has been a few things:
1. We were not getting the URL params totally right
The problem here has been a few things:
1. We were not getting the URL params totally right
the common things that break down as "dimensions" on a given data source and then update whenever those records underlying pieces do with some other magic.
So that is actually serving the client right now, like, a lot of the time.
My problem that is coming is in how we structure the representation
So that is actually serving the client right now, like, a lot of the time.
My problem that is coming is in how we structure the representation
November 21, 2025 at 4:21 AM
the common things that break down as "dimensions" on a given data source and then update whenever those records underlying pieces do with some other magic.
So that is actually serving the client right now, like, a lot of the time.
My problem that is coming is in how we structure the representation
So that is actually serving the client right now, like, a lot of the time.
My problem that is coming is in how we structure the representation
-> have all those files get posted to azure blob storage when done
-> write it all to Apache Druid
Druid serves the client.
Now, the results from druid queries that remain static, are actually cached by ETS
We have an entire engine built for this that gives us tons of super fast search over
-> write it all to Apache Druid
Druid serves the client.
Now, the results from druid queries that remain static, are actually cached by ETS
We have an entire engine built for this that gives us tons of super fast search over
November 21, 2025 at 4:21 AM
-> have all those files get posted to azure blob storage when done
-> write it all to Apache Druid
Druid serves the client.
Now, the results from druid queries that remain static, are actually cached by ETS
We have an entire engine built for this that gives us tons of super fast search over
-> write it all to Apache Druid
Druid serves the client.
Now, the results from druid queries that remain static, are actually cached by ETS
We have an entire engine built for this that gives us tons of super fast search over
This is not an ETS problem. ETS is actually part of this, and functions as a huge performance booster.
Basically, the path is:
warehouse from IRL receiving message feeds
-> airflow
-> structured warehouse
-> ETL done by our app
-> write files because upstream needs .gz files of data from ETL
Basically, the path is:
warehouse from IRL receiving message feeds
-> airflow
-> structured warehouse
-> ETL done by our app
-> write files because upstream needs .gz files of data from ETL
November 21, 2025 at 4:21 AM
This is not an ETS problem. ETS is actually part of this, and functions as a huge performance booster.
Basically, the path is:
warehouse from IRL receiving message feeds
-> airflow
-> structured warehouse
-> ETL done by our app
-> write files because upstream needs .gz files of data from ETL
Basically, the path is:
warehouse from IRL receiving message feeds
-> airflow
-> structured warehouse
-> ETL done by our app
-> write files because upstream needs .gz files of data from ETL
This isn't impressive I just found it amusing.
November 21, 2025 at 3:40 AM
This isn't impressive I just found it amusing.
It will download a new file. That is its job.
Press and hold the power button until it beeps twice to enter Safe Mode if you wanna boot it in a recovery sense and retry really clean
Press and hold the power button until it beeps twice to enter Safe Mode if you wanna boot it in a recovery sense and retry really clean
November 21, 2025 at 3:04 AM
It will download a new file. That is its job.
Press and hold the power button until it beeps twice to enter Safe Mode if you wanna boot it in a recovery sense and retry really clean
Press and hold the power button until it beeps twice to enter Safe Mode if you wanna boot it in a recovery sense and retry really clean
Tell him to chill tf out and remember that modern internet speeds are fast
November 21, 2025 at 3:03 AM
Tell him to chill tf out and remember that modern internet speeds are fast
sometimes I actually work hard but really this is just me being an obsessive and in a battle with myself that has no real winners
November 21, 2025 at 2:58 AM
sometimes I actually work hard but really this is just me being an obsessive and in a battle with myself that has no real winners
its like watchin' from the dugout instead of the stands lol
November 21, 2025 at 2:42 AM
its like watchin' from the dugout instead of the stands lol