BetaSciGuy
LightNews LightNews
betasci.bsky.social
BetaSciGuy
@betasci.bsky.social
20 followers 28 following 250 posts
Dad. Data enthusiast. Former academic, current data scientist/engineer. #dataengineering #datescience #dataviz
Posts Replies Media Videos
betasci.bsky.social
Amazon AWS experiences a major outage, impacting countless services!

...and yet...📈
Amazon stock price increases on October 20, 2025 dispute major AWS outage earlier in the day.
October 21, 2025 at 12:48 PM
betasci.bsky.social
Hey DOGE! I found evidence of government wasting time and tax payer dollars on something the people don't want!
www.congress.gov/bill/119th-c...
Screenshot of H.R. 1161 titled "To authorize the President to enter into negotiations to acquire Greenland and to rename Greenland as "Red, White, and Blueland".
February 12, 2025 at 3:21 AM
betasci.bsky.social
Fun fact: @npr.org and FCC headquarters are directly across the street from one another
Google maps image showing NPR and FCC headquarters being 2 min walk away from each other.
January 31, 2025 at 1:32 AM
betasci.bsky.social
This new government devolved into a meme in less than one day
Doge dot gov is just a picture of the doge dog
January 22, 2025 at 12:36 AM
betasci.bsky.social
GOD, I love Raygun (the mid-west clothing store, not the Australian break dancer, although I have nothing against her)
www.raygunsite.com/collections/...
Image from raygunsite.com selling a "shirt duct taped to floor" for 3.1 million dollars.
December 7, 2024 at 3:24 PM
betasci.bsky.social
The 'side of existential dread' got me 🤣
Blueskyroast.com result for betas I.bsky.social. Data and dad jokes apparently.
December 5, 2024 at 4:10 PM
betasci.bsky.social
Small detour experimenting with including lineage with charts and dashboards in Apache Superset. Took a lot of testing, but I figured out the specific necessary configuration details eventually.
#databs #DataHub #ApacheSuperset
December 4, 2024 at 5:42 AM
betasci.bsky.social
Returned to testing #DataHub and figuring out column-level lineage. Turns out, the issue was how #DBeaver sends the queries to Redshift and the query format being something that #DataHub can't automatically parse. I went to AWS's query editor (v2), ran the same queries and now it works!
#databs
DataHub column level lineage showing how columns in the 'sales per state' table depend on columns from the 'users', 'sales', and 'listing' tables.
December 3, 2024 at 5:04 PM
betasci.bsky.social
Duolingo got real aggressive with its end of year wrap up! Looks like Duo has some kind of secret dark side that I should be scared of...
"2024. Did you learn enough to be safe from Duo? Let's see!"
December 2, 2024 at 4:58 PM
betasci.bsky.social
Tonight I went on an adventure of trying to install OpenMetadata to test its data catalog functionality so I could compare it to DataHub. Most of the features seem to be the same, but OpenMetadata appears to struggle to automatically detect the table lineage in my Redshift database.
OpenMetadata deployment showing a lack of table lineage detected for one of the generated tables.
November 30, 2024 at 5:44 AM
betasci.bsky.social
OK, I can create derived tables in my redshift database and DataHub will detect the table-level lineage of those tables, but it's not giving me the column level lineage. I may need to dig a little deeper into this to figure out why this is... but not tonight! I've done enough today.
#databs
DataHub lineage visualization demo showing the monthly sales by user is derived from the sales table. Image shows how column level lineage is missing currently. DataHub lineage visualization demo showing the sales by state is derived from various upstream tables. Image shows how column level lineage is missing currently.
November 29, 2024 at 2:31 AM
betasci.bsky.social
I realized I'm not very good with map data, especially when it comes to plotting with generic shapefiles, so I'm trying to work on that. Here's a quick map I did using a DC ward shapefile that shows the change in population from 2010->2020.
Map showing percent change in population by D.C. ward between 2010-2020.
November 9, 2024 at 4:00 AM
betasci.bsky.social
My emotional state today in the form of a Ghibli movie poster:
Grave of the fireflies, Ghibli move poster.
November 6, 2024 at 9:28 PM
betasci.bsky.social
📊 Another interesting insight is to look at what day of the week people are joining. Looks like most people join near the end of the week (Thursday/Friday), but fewer join on the weekends. 6/
Histogram of the number of users in the sample by day of the week they joined.
September 18, 2024 at 3:26 AM
betasci.bsky.social
Noticing a lot of people posting their user numbers and when they joined, so I thought it would be interesting to see what this information reveals about the history of BlueSky.

Two events stand out ... 1/
Plot of BlueSky user numbers over time as derived from recent posts celebrating BlueSky reaching ten million users.
September 18, 2024 at 2:49 AM
betasci.bsky.social
Cool!
Bluesky celebration banner indicating BetaSciGuy was the 5,620,347th member of bluesky, joining on May 21, 2024.
September 17, 2024 at 12:23 PM
betasci.bsky.social
Average VantageScore credit score by state in 2024. 📊

Source: "What Is the Average Credit Score by State?", equifax.com (2024)
Visualization of the average credit score in 2024 for U.S. states.
July 30, 2024 at 2:41 AM
betasci.bsky.social
But if you make one of those columns a categorical variable, it will result in a table with every single possible combination of that categorical column and the other columns in the groupby:
Pandas.DataFrame.groupby() example with a categorical variable in the groupby list.
June 6, 2024 at 2:41 AM
betasci.bsky.social
To clarify, Pandas.DataFrame.groupby() does exactly what you'd expect it to do if you're dealing with columns that are strings or numbers:
Pandas.DataFrame.groupby example with integer column data types
June 6, 2024 at 2:38 AM