I see some discussion of folks diving into #ipeds data today. I noticed the Access Files are not available just yet. I have been cooking up an idea for a #dataeng project over winter break and have been waiting for the new files.
My question: Does anyone know if the Access files are close behind?
My question: Does anyone know if the Access files are close behind?
January 7, 2025 at 9:53 PM
AI/ML devs face the same orchestration headaches, just with more pressure to ship fast. Josh Smith breaks down why 90+ .ai companies use Temporal to move quicker & build more reliably. A must-read for anyone scaling AI systems.
📖 https://bit.ly/ai-ml-dataeng
📖 https://bit.ly/ai-ml-dataeng
AI, ML, and Data Engineering Workflows with Temporal
AI and ML developers bump into the same rough edges of system orchestration that nearly every developer encounters. Whether it's management of complex data pipelines, job coordination across GPU resources, failure handling, or deploying a model
bit.ly
June 5, 2025 at 12:59 PM
AI/ML devs face the same orchestration headaches, just with more pressure to ship fast. Josh Smith breaks down why 90+ .ai companies use Temporal to move quicker & build more reliably. A must-read for anyone scaling AI systems.
📖 https://bit.ly/ai-ml-dataeng
📖 https://bit.ly/ai-ml-dataeng
Tips and tricks for building resilient payment systems from a Staff Developer working on Shopify’s payment infrastructure.
shopify.engineering/building-res...
#dataeng #softeng
shopify.engineering/building-res...
#dataeng #softeng
10 Tips for Building Resilient Payment Systems - Shopify
Top ten tips and tricks for building resilient payment systems from a Staff Developer working on Shopify’s payment infrastructure.
shopify.engineering
September 20, 2024 at 8:45 PM
Tips and tricks for building resilient payment systems from a Staff Developer working on Shopify’s payment infrastructure.
shopify.engineering/building-res...
#dataeng #softeng
shopify.engineering/building-res...
#dataeng #softeng
🍻 it's official! inaugural data debug sf happy hour is live
july 29 • 5:30-7:30pm • sf (venue tba)
for everyone who's ever said "the data looks weird" and meant it
casual networking • data war stories • good drinks (alc & non-alc) • great people
rsvp: lu.ma/g92ckftj
#datadebug #dataeng #sf #irl
july 29 • 5:30-7:30pm • sf (venue tba)
for everyone who's ever said "the data looks weird" and meant it
casual networking • data war stories • good drinks (alc & non-alc) • great people
rsvp: lu.ma/g92ckftj
#datadebug #dataeng #sf #irl
Data Debug SF: Inaugural Happy Hour · Luma
🍻 Join us for the inaugural Data Debug SF happy hour!
We're launching San Francisco's community for data engineers, analytics engineers, and data…
lu.ma
July 10, 2025 at 9:33 PM
🍻 it's official! inaugural data debug sf happy hour is live
july 29 • 5:30-7:30pm • sf (venue tba)
for everyone who's ever said "the data looks weird" and meant it
casual networking • data war stories • good drinks (alc & non-alc) • great people
rsvp: lu.ma/g92ckftj
#datadebug #dataeng #sf #irl
july 29 • 5:30-7:30pm • sf (venue tba)
for everyone who's ever said "the data looks weird" and meant it
casual networking • data war stories • good drinks (alc & non-alc) • great people
rsvp: lu.ma/g92ckftj
#datadebug #dataeng #sf #irl
November 18, 2024 at 4:36 PM
Arkflow: A new Rust stream processor that combines performance with simplicity. Data pipelines just got better.
#rust #streaming #dataeng https://github.com/chenquan/arkflow
#rust #streaming #dataeng https://github.com/chenquan/arkflow
ArkFlow – High-performance Rust stream processing engine
Comments
github.com
March 14, 2025 at 10:22 AM
Arkflow: A new Rust stream processor that combines performance with simplicity. Data pipelines just got better.
#rust #streaming #dataeng https://github.com/chenquan/arkflow
#rust #streaming #dataeng https://github.com/chenquan/arkflow
Would you like to learn how to make clean, secure and reusable integration tests that cost 0$ to run?! #dotnet #devfeed #coding #bootstrappers #azure #dataeng
Check out my new article 👇
Check out my new article 👇
How to automate integration testing using docker -
In this article, we'll look at how to automate integration testing using Docker. By the end, you'll be able to set up a complete test environment
bit.ly
April 17, 2025 at 7:21 AM
There are 3 Types of Data Engineers
Then there were three. The final three.
www.linkedin.com
October 2, 2024 at 5:44 PM
Do you know of any #rstats teams who could use some help in 2025?
I'm a freelance consultant with 10yrs of experience in #datasci, #dataeng, #dataops, and providing #sysadmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around […]
I'm a freelance consultant with 10yrs of experience in #datasci, #dataeng, #dataops, and providing #sysadmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around […]
Original post on toot.bldrweb.org
toot.bldrweb.org
December 27, 2024 at 7:41 PM
i've been taking the chicago crime data and popping it through DBT as a kind of silly toy project. it's pretty neat. i have been doing manual downloads so i need to set up a daily cron thing but i like how it's shaping up.
i'll post the architecture / code on here when it's done.
#dataeng #dbt
i'll post the architecture / code on here when it's done.
#dataeng #dbt
December 14, 2024 at 7:54 PM
It's great to be back!
Sydney Data Engineering meetup is happening this Wednesday at the MSFT Reactor...
Join us:
www.meetup.com/sydney-data-...
Sydney Data Engineering meetup is happening this Wednesday at the MSFT Reactor...
Join us:
www.meetup.com/sydney-data-...
January 20, 2025 at 2:26 AM
It's great to be back!
Sydney Data Engineering meetup is happening this Wednesday at the MSFT Reactor...
Join us:
www.meetup.com/sydney-data-...
Sydney Data Engineering meetup is happening this Wednesday at the MSFT Reactor...
Join us:
www.meetup.com/sydney-data-...
This is very true for SWE estimates, but luckily it has an effective remedy.
Never flex on deadlines.
Flex on the delivery scope.
#Developers #Promosky #indiedev #dev #dataeng
Never flex on deadlines.
Flex on the delivery scope.
#Developers #Promosky #indiedev #dev #dataeng
April 4, 2025 at 1:28 PM
This is very true for SWE estimates, but luckily it has an effective remedy.
Never flex on deadlines.
Flex on the delivery scope.
#Developers #Promosky #indiedev #dev #dataeng
Never flex on deadlines.
Flex on the delivery scope.
#Developers #Promosky #indiedev #dev #dataeng
Sydney people, as you may have already seen, Melbourne is sold out for General Admission – DataEng Day Only tickets, and we have only 14 left in Sydney. If you only want to attend Data Engineering Day, this is your chance to book a single-day ticket.
dataengbytes.com/sydney
dataengbytes.com/sydney
July 11, 2025 at 12:00 AM
Sydney people, as you may have already seen, Melbourne is sold out for General Admission – DataEng Day Only tickets, and we have only 14 left in Sydney. If you only want to attend Data Engineering Day, this is your chance to book a single-day ticket.
dataengbytes.com/sydney
dataengbytes.com/sydney
Holy moly this looks amazing. I'm a big fan of SFN and excited to see where this service can go.
My scale has never been FAANG but it certainly addresses many data team requirements, at least in my view.
aws.amazon.com/blogs/comput...
#AWS #DataEng
My scale has never been FAANG but it certainly addresses many data team requirements, at least in my view.
aws.amazon.com/blogs/comput...
#AWS #DataEng
Simplifying developer experience with variables and JSONata in AWS Step Functions | Amazon Web Services
This post is written by Uma Ramadoss, Principal Specialist SA, Serverless and Dhiraj Mahapatro, Principal Specialist SA, Amazon Bedrock AWS Step Functions is introducing variables and JSONata data tra...
aws.amazon.com
November 27, 2024 at 1:54 AM
Holy moly this looks amazing. I'm a big fan of SFN and excited to see where this service can go.
My scale has never been FAANG but it certainly addresses many data team requirements, at least in my view.
aws.amazon.com/blogs/comput...
#AWS #DataEng
My scale has never been FAANG but it certainly addresses many data team requirements, at least in my view.
aws.amazon.com/blogs/comput...
#AWS #DataEng
Not too long ago, I wrote an article on how I leverage my GPU to run local LLM to boost my productivity. This way is free & safe.
I just updated this 💎 to show you how to achieve this using docker and thus avoiding cluttering up your system due to dependencies etc.
Enjoy!
#coding #dataeng
I just updated this 💎 to show you how to achieve this using docker and thus avoiding cluttering up your system due to dependencies etc.
Enjoy!
#coding #dataeng
Using cloud AI services to autocomplete code saves lets one focus on the creative parts of work.
However uncertainties regarding IP rights are still present as the rest of world is catching on.
Here's how to setup AI autocomplete completely safe and free.
👉 bit.ly/3RaiEFu #devfeed #developers
However uncertainties regarding IP rights are still present as the rest of world is catching on.
Here's how to setup AI autocomplete completely safe and free.
👉 bit.ly/3RaiEFu #devfeed #developers
May 12, 2025 at 10:45 AM
Melbourne, you’ve done it! We’re now officially sold out of the General Admission - DataEng Day Only tickets. There are only a few General Admission - 2 Day tickets left, and we’re confident they’ll be gone in no time.
Thank you for all your love and support.
Thank you for all your love and support.
July 10, 2025 at 4:00 AM
Melbourne, you’ve done it! We’re now officially sold out of the General Admission - DataEng Day Only tickets. There are only a few General Admission - 2 Day tickets left, and we’re confident they’ll be gone in no time.
Thank you for all your love and support.
Thank you for all your love and support.
"A wife sends her programmer husband to the grocery store for a loaf of 🍞.
On his way out, she says, "and if they have 🥚, get a dozen."
The programmer husband returns home with 12 loaves of 🍞."
Do you believe an AI would have perceived that differently?
#developers #devfeed #dataeng #programming
On his way out, she says, "and if they have 🥚, get a dozen."
The programmer husband returns home with 12 loaves of 🍞."
Do you believe an AI would have perceived that differently?
#developers #devfeed #dataeng #programming
April 24, 2025 at 7:29 AM
"A wife sends her programmer husband to the grocery store for a loaf of 🍞.
On his way out, she says, "and if they have 🥚, get a dozen."
The programmer husband returns home with 12 loaves of 🍞."
Do you believe an AI would have perceived that differently?
#developers #devfeed #dataeng #programming
On his way out, she says, "and if they have 🥚, get a dozen."
The programmer husband returns home with 12 loaves of 🍞."
Do you believe an AI would have perceived that differently?
#developers #devfeed #dataeng #programming
Vector databases, meanwhile, are all about unstructured data, built to support modern workloads like generative AI, machine learning inference, recommendations, and natural language processing.
#dataeng #vector #database
#dataeng #vector #database
Why vector databases aren’t just databases
Vector databases don’t just store your data. They find the most meaningful connections within it, driving insights and decisions at scale.
www.infoworld.com
September 30, 2024 at 11:49 PM
𝙃𝙚𝙧𝙚 𝙞𝙨 𝙝𝙤𝙬 𝙄 𝙗𝙤𝙤𝙨𝙩 𝙢𝙮 𝙘𝙤𝙙𝙞𝙣𝙜 𝙥𝙧𝙤𝙙𝙪𝙘𝙩𝙞𝙫𝙞𝙩𝙮 𝙨𝙖𝙛𝙚𝙡𝙮 👨💻
Just wrote an article on how I setup autocomplete, in VSCode, using local LLM with minimal hardware requirements.
Autocomplete boosts my productivity as I don’t have to spend energy on boiler plate code 🚀
#devfeed #developers #dataeng
👇
Just wrote an article on how I setup autocomplete, in VSCode, using local LLM with minimal hardware requirements.
Autocomplete boosts my productivity as I don’t have to spend energy on boiler plate code 🚀
#devfeed #developers #dataeng
👇
Autocomplete with your local LLM - Code your assets
This article will show how I setup autocomplete, in VSCode, using local LLM with minimal hardware requirements. Local llm hardware requirements.
bit.ly
April 9, 2025 at 9:54 AM
𝙃𝙚𝙧𝙚 𝙞𝙨 𝙝𝙤𝙬 𝙄 𝙗𝙤𝙤𝙨𝙩 𝙢𝙮 𝙘𝙤𝙙𝙞𝙣𝙜 𝙥𝙧𝙤𝙙𝙪𝙘𝙩𝙞𝙫𝙞𝙩𝙮 𝙨𝙖𝙛𝙚𝙡𝙮 👨💻
Just wrote an article on how I setup autocomplete, in VSCode, using local LLM with minimal hardware requirements.
Autocomplete boosts my productivity as I don’t have to spend energy on boiler plate code 🚀
#devfeed #developers #dataeng
👇
Just wrote an article on how I setup autocomplete, in VSCode, using local LLM with minimal hardware requirements.
Autocomplete boosts my productivity as I don’t have to spend energy on boiler plate code 🚀
#devfeed #developers #dataeng
👇
It would be a treat to work with @chris.wensel.net so if you are #DataEng with flare for #Java #opensource & #devops you may want to look into this
Hey all, I'm hiring for a #DataEng (and #DevOps) role in the SF Bay Area. Reach out directly to me for more info.
#Java background is a strong want plus AWS and a desire to work on #OpenSource stuff.
#DataBS
#Java background is a strong want plus AWS and a desire to work on #OpenSource stuff.
#DataBS
March 27, 2025 at 7:24 PM
It would be a treat to work with @chris.wensel.net so if you are #DataEng with flare for #Java #opensource & #devops you may want to look into this
Mis últimas semanas con Go me han llevado a una reflexión: ¿Son los **zero values** realmente una característica que simplifica o complican más la vida a la larga en sistemas de datos y APIs? Abro hilo para explicar mi perspectiva. 👇 #GoLang #SoftwareDesign #DataEng
May 26, 2025 at 7:59 PM
Mis últimas semanas con Go me han llevado a una reflexión: ¿Son los **zero values** realmente una característica que simplifica o complican más la vida a la larga en sistemas de datos y APIs? Abro hilo para explicar mi perspectiva. 👇 #GoLang #SoftwareDesign #DataEng
Hey all, I'm hiring for a #DataEng (and #DevOps) role in the SF Bay Area. Reach out directly to me for more info.
#Java background is a strong want plus AWS and a desire to work on #OpenSource stuff.
#DataBS
#Java background is a strong want plus AWS and a desire to work on #OpenSource stuff.
#DataBS
March 27, 2025 at 2:36 PM
Hey all, I'm hiring for a #DataEng (and #DevOps) role in the SF Bay Area. Reach out directly to me for more info.
#Java background is a strong want plus AWS and a desire to work on #OpenSource stuff.
#DataBS
#Java background is a strong want plus AWS and a desire to work on #OpenSource stuff.
#DataBS