TEGNicholas.bsky.social
@tegnicholas.bsky.social
Open-Source Software for science at Earthmover.io, built on Pangeo.io.
One of many xarray.dev core devs. https://tom-nicholas.com/
Previously dabbled in oceanography at [C]Worthy and Columbia Uni., originally did fusion plasma physics.
One of many xarray.dev core devs. https://tom-nicholas.com/
Previously dabbled in oceanography at [C]Worthy and Columbia Uni., originally did fusion plasma physics.
Pinned
Science needs a social network for sharing big data - HackMD
hackmd.io
I've been thinking a lot recently about how one thing science needs generally is a **social network for sharing big data**, so I wrote this.
hackmd.io/wKKm4cIDR6a9...
🧵
hackmd.io/wKKm4cIDR6a9...
🧵
Reposted by TEGNicholas.bsky.social
"[Matt Yglesias] can write whole essays claiming that fracking is good and we need fossil fuel friendly energy policies, dismissing progressives as childish, while never engaging with the scientific literature on the consequences of climate change"
Matt Yglesias Is Confidently Wrong About Everything
The Biden administration’s favorite centrist pundit produces smug pseudo-analysis that cannot be considered serious thought. He ought to be permanently disregarded.
www.currentaffairs.org
September 3, 2025 at 9:32 PM
"[Matt Yglesias] can write whole essays claiming that fracking is good and we need fossil fuel friendly energy policies, dismissing progressives as childish, while never engaging with the scientific literature on the consequences of climate change"
Reposted by TEGNicholas.bsky.social
New blog post on Obstore, fast, multi-provider cloud storage access for Python:
developmentseed.org/blog/2025-08...
developmentseed.org/blog/2025-08...
We Needed Better Cloud Storage for Python so We Built Obstore — Development Seed
Obstore solves the friction we kept hitting in cloud-native workflows.
developmentseed.org
August 4, 2025 at 4:11 PM
New blog post on Obstore, fast, multi-provider cloud storage access for Python:
developmentseed.org/blog/2025-08...
developmentseed.org/blog/2025-08...
Reposted by TEGNicholas.bsky.social
Open source, open science for earth, climate and geospatial science? Coming to #AGU25? Build tools in #Python @jupyter.org?
Submit an abstract for this session and come meet us and like minded scientists!
Submit an abstract for this session and come meet us and like minded scientists!
📣 Call for submissions: AGU25 IN029 "Open Source Geospatial Workflows in the Cloud" 🌍
Please submit your work using cutting-edge practices and tools that are shaping the future of geospatial science.
More info: events.geojupyter.org/conferences/...
#opensource #geospatial #jupyter #AGU25
Please submit your work using cutting-edge practices and tools that are shaping the future of geospatial science.
More info: events.geojupyter.org/conferences/...
#opensource #geospatial #jupyter #AGU25
July 24, 2025 at 9:54 PM
Open source, open science for earth, climate and geospatial science? Coming to #AGU25? Build tools in #Python @jupyter.org?
Submit an abstract for this session and come meet us and like minded scientists!
Submit an abstract for this session and come meet us and like minded scientists!
Reposted by TEGNicholas.bsky.social
Will Alpine co-wrote Microsoft's manifesto on how AI will be a powerful force for good for climate change
In a new interview, Alpine disavows the manifesto, saying he believes Microsoft used his work to distract from the much larger climate harms the company enables through contracts with Big Oil
In a new interview, Alpine disavows the manifesto, saying he believes Microsoft used his work to distract from the much larger climate harms the company enables through contracts with Big Oil
He helped Microsoft build AI to help the climate. Then Microsoft sold it to Big Oil.
A former Microsoft project manager reveals how the tech giant is using AI to help Big Oil drill—and how he and his partner are now pushing for change.
heated.world
July 21, 2025 at 5:24 PM
Will Alpine co-wrote Microsoft's manifesto on how AI will be a powerful force for good for climate change
In a new interview, Alpine disavows the manifesto, saying he believes Microsoft used his work to distract from the much larger climate harms the company enables through contracts with Big Oil
In a new interview, Alpine disavows the manifesto, saying he believes Microsoft used his work to distract from the much larger climate harms the company enables through contracts with Big Oil
Reposted by TEGNicholas.bsky.social
This is fucking insane. Closing these NOAA labs would obliterate our ability to observe, understand, and forecast the Earth System, from weather systems tomorrow to sea levels 50 years from now.
What a tragedy this is even being proposed on paper... 💔
NOAA FY2026 Congressional Justification: www.noaa.gov/sites/defaul...
NOAA FY2026 Congressional Justification: www.noaa.gov/sites/defaul...
June 30, 2025 at 4:59 PM
This is fucking insane. Closing these NOAA labs would obliterate our ability to observe, understand, and forecast the Earth System, from weather systems tomorrow to sea levels 50 years from now.
Reposted by TEGNicholas.bsky.social
NASA is being told to cancel 19 *active* missions to save $6B, which looks to be less than the ICE *hiring/retention* budget going forward.
I need people to let that sentence sink into their bones for a minute.
I need people to let that sentence sink into their bones for a minute.
If the GOP reconciliation bill passes, ICE gets through FY2029:
- $45 billion for detention, on top of the current annual budget of $3.4 billion
- $14.4 billion for transportation and removal, on top of the current annual budget of $750 million
- $8 billion for hiring/retention
- Billions more.
- $45 billion for detention, on top of the current annual budget of $3.4 billion
- $14.4 billion for transportation and removal, on top of the current annual budget of $750 million
- $8 billion for hiring/retention
- Billions more.
‼️‼️
June 29, 2025 at 4:17 PM
NASA is being told to cancel 19 *active* missions to save $6B, which looks to be less than the ICE *hiring/retention* budget going forward.
I need people to let that sentence sink into their bones for a minute.
I need people to let that sentence sink into their bones for a minute.
Reposted by TEGNicholas.bsky.social
I've been adding new accounts to the Open Source Geospatial starter pack. Who else wants on or off?
#gischat #geosky
go.bsky.app/PGYLmPG
#gischat #geosky
go.bsky.app/PGYLmPG
October 28, 2024 at 7:57 PM
I've been adding new accounts to the Open Source Geospatial starter pack. Who else wants on or off?
#gischat #geosky
go.bsky.app/PGYLmPG
#gischat #geosky
go.bsky.app/PGYLmPG
Reposted by TEGNicholas.bsky.social
It's outrageous that NASA GISS, one of the best earth & space science labs in the world, is being kicked out of its Columbia home. The outstanding scientists who work there can't say that publicly, but I can. And so can you --- call your reps, esp. (but not only) if you live in NYC or NY state.
Nice Guardian article about us losing our office in Manhattan
www.theguardian.com/environment/...
www.theguardian.com/environment/...
Godfather of climate science decries Trump plan to shut Nasa lab above Seinfeld diner: ‘It’s crazy’
Over breakfast at Tom’s Restaurant, right below the historic Giss lab, James Hansen calls Doge’s decision a ‘big mistake’
www.theguardian.com
May 21, 2025 at 4:45 PM
It's outrageous that NASA GISS, one of the best earth & space science labs in the world, is being kicked out of its Columbia home. The outstanding scientists who work there can't say that publicly, but I can. And so can you --- call your reps, esp. (but not only) if you live in NYC or NY state.
Reposted by TEGNicholas.bsky.social
𝐻𝑜𝑤 𝑑𝑜𝑒𝑠 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝑎𝑣𝑜𝑖𝑑 𝑟𝑒𝑑𝑢𝑛𝑑𝑎𝑛𝑡 𝑠𝑡𝑜𝑟𝑎𝑔𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑑𝑎𝑡𝑎 𝑣𝑒𝑟𝑠𝑖𝑜𝑛𝑠?
Icechunk stores only new or changed chunks for each version —no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead.
More: bit.ly/3F1XFST
Icechunk stores only new or changed chunks for each version —no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead.
More: bit.ly/3F1XFST
Icechunk: Efficient storage of versioned array data - Earthmover
We recently got an interesting question in Icechunk’s community Slack channel (thank you Iury Simoes-Sousa for motivating this post): I’m new to Icechunk. How is the storage managed for redundant info...
earthmover.io
May 14, 2025 at 4:09 PM
𝐻𝑜𝑤 𝑑𝑜𝑒𝑠 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝑎𝑣𝑜𝑖𝑑 𝑟𝑒𝑑𝑢𝑛𝑑𝑎𝑛𝑡 𝑠𝑡𝑜𝑟𝑎𝑔𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑑𝑎𝑡𝑎 𝑣𝑒𝑟𝑠𝑖𝑜𝑛𝑠?
Icechunk stores only new or changed chunks for each version —no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead.
More: bit.ly/3F1XFST
Icechunk stores only new or changed chunks for each version —no redundant copies or rewrites. You get instant time travel, branching, and efficient updates, all with negligible storage overhead.
More: bit.ly/3F1XFST
Excellent post by Brian Davis laying out why doing "Open Science" for data-driven workflows is almost impossible in practice, at least without much better data pipeline tools.
Our latest blog post dives into the chaos of the status quo - where every tweak means regenerating the 𝑤ℎ𝑜𝑙𝑒 𝑑𝑎𝑡𝑎𝑠𝑒𝑡 and collaboration and experimentation is often stifled by silos and secret knowledge. Check out the full post: earthmover.io/blog/tensoro...
TensorOps: Scientific Data Doesn't Have to Hurt - Earthmover
Curious how your team scores on the "Data Pain Survey"? Wondering why your teams are building Rube Goldberg machines just to put some data on a map? Or just want to see our plan to bring order to your...
earthmover.io
May 12, 2025 at 2:10 PM
Excellent post by Brian Davis laying out why doing "Open Science" for data-driven workflows is almost impossible in practice, at least without much better data pipeline tools.
Reposted by TEGNicholas.bsky.social
The proposed cuts to NOAA cold have profound consequences not just for climate change, but for our national security and the entire economy. Here's what I learned: www.propublica.org/article/trum...
White House Proposal Could Gut Climate Modeling the World Depends On
Potential funding cuts for NOAA and its research partners threaten irreparable harm not only to climate research but to American safety, competitiveness, and national security.
www.propublica.org
April 24, 2025 at 6:06 PM
The proposed cuts to NOAA cold have profound consequences not just for climate change, but for our national security and the entire economy. Here's what I learned: www.propublica.org/article/trum...
It's fun to work with real hardcore software engineers like @functionth.bsky.social who can teach you about database consistency and transactions and all that
Scientific data infrastructure should be built on solid foundations like this instead of on piles of janky code written by postdocs...
Scientific data infrastructure should be built on solid foundations like this instead of on piles of janky code written by postdocs...
1/ 🚨 New Blog Post Alert: "𝐿𝑒𝑎𝑟𝑛𝑖𝑛𝑔 𝐴𝑏𝑜𝑢𝑡 𝐼𝑐𝑒𝑐ℎ𝑢𝑛𝑘 𝐶𝑜𝑛𝑠𝑖𝑠𝑡𝑒𝑛𝑐𝑦 𝑤𝑖𝑡ℎ 𝑎 𝐶𝑙𝑖𝑐ℎ𝑒́𝑑 𝑏𝑢𝑡 𝐼𝑛𝑠𝑡𝑟𝑢𝑐𝑡𝑖𝑣𝑒 𝐸𝑥𝑎𝑚𝑝𝑙𝑒" 🏦🔁
👉 Read it here: earthmover.io/blog/learnin...
👉 Read it here: earthmover.io/blog/learnin...
Learning about Icechunk consistency with a clichéd but instructive example - Earthmover
In this post we’ll show what can happen when more than one process write to the same Icechunk repository concurrently, and how Icechunk uses transactions and conflict resolution to guarantee consisten...
earthmover.io
April 23, 2025 at 4:29 PM
It's fun to work with real hardcore software engineers like @functionth.bsky.social who can teach you about database consistency and transactions and all that
Scientific data infrastructure should be built on solid foundations like this instead of on piles of janky code written by postdocs...
Scientific data infrastructure should be built on solid foundations like this instead of on piles of janky code written by postdocs...
Reposted by TEGNicholas.bsky.social
the fact that I've never once thought about making a range request, and yet make them constantly for extremely targeted data pulls, is absolutely an invisible technical miracle
April 19, 2025 at 2:31 PM
the fact that I've never once thought about making a range request, and yet make them constantly for extremely targeted data pulls, is absolutely an invisible technical miracle
I wrote the article I wish I could have read back when I first heard of Zarr and cloud-native science back in 2018.
This explains how object storage and conventional filesystems are different, and the key properties that make @zarr.dev work so well in cloud object storage.
This explains how object storage and conventional filesystems are different, and the key properties that make @zarr.dev work so well in cloud object storage.
1/ 💡 Our latest blog post in the fundamentals series, written by @tegnicholas.bsky.social, demystifies cloud-optimized scientific data formats!
Read more: earthmover.io/blog/fundame...
Read more: earthmover.io/blog/fundame...
Fundamentals: What is Cloud-Optimized Scientific Data?
What cloud-optimized data really means, and how Zarr and Icechunk enable fast access to massive scientific datasets in cloud object storage.
earthmover.io
April 17, 2025 at 5:50 PM
I wrote the article I wish I could have read back when I first heard of Zarr and cloud-native science back in 2018.
This explains how object storage and conventional filesystems are different, and the key properties that make @zarr.dev work so well in cloud object storage.
This explains how object storage and conventional filesystems are different, and the key properties that make @zarr.dev work so well in cloud object storage.
1/ Flux solves the impedance mismatch between geospatial data providers and consumers.
Providers want to manage data lakes stored in cloud-optimized formats like Zarr, but consumers want their applications to keep being fed data in ways they already understand.
Providers want to manage data lakes stored in cloud-optimized formats like Zarr, but consumers want their applications to keep being fed data in ways they already understand.
1/ Today we are launching Flux, a powerful new addition to the Earthmover platform. Flux is a high-performance gateway for exploring, querying, and visualizing geospatial data via standards-compliant APIs (EDR, WMS, and OPeNDAP).
April 16, 2025 at 1:41 PM
1/ Flux solves the impedance mismatch between geospatial data providers and consumers.
Providers want to manage data lakes stored in cloud-optimized formats like Zarr, but consumers want their applications to keep being fed data in ways they already understand.
Providers want to manage data lakes stored in cloud-optimized formats like Zarr, but consumers want their applications to keep being fed data in ways they already understand.
Reposted by TEGNicholas.bsky.social
Hard to overstate this plan's reach, which touches nearly every aspect of NOAA's work - dissolving its research arm, gutting climate science, diminishing sat observations, boosting fossil fuels. With amazing colleagues Daniel Cusick and @scottpwaldman.bsky.social :
www.politico.com/news/2025/04...
www.politico.com/news/2025/04...
White House outlines plan to gut NOAA, smother climate research
The agency’s Office of Oceanic and Atmospheric Research would be “eliminated as a line office,” according to a memo from the Office of Management and Budget.
www.politico.com
April 11, 2025 at 6:38 PM
Hard to overstate this plan's reach, which touches nearly every aspect of NOAA's work - dissolving its research arm, gutting climate science, diminishing sat observations, boosting fossil fuels. With amazing colleagues Daniel Cusick and @scottpwaldman.bsky.social :
www.politico.com/news/2025/04...
www.politico.com/news/2025/04...
You could also do this for arbitrarily large scientific array datasets using Xarray + Icechunk + R2/Tigris
juhache.substack.com/p/0-data-dis...
juhache.substack.com/p/0-data-dis...
0$ Data Distribution
Ju Data Engineering Weekly - Ep 78
juhache.substack.com
April 10, 2025 at 8:14 PM
You could also do this for arbitrarily large scientific array datasets using Xarray + Icechunk + R2/Tigris
juhache.substack.com/p/0-data-dis...
juhache.substack.com/p/0-data-dis...
Reposted by TEGNicholas.bsky.social
📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...
Exploring Icechunk scalability: untangling S3's prefix story | Earthmover
We show Icechunk can scale to extremely high concurrency levels, and explain how it achieves this in modern object stores.
earthmover.io
April 9, 2025 at 3:27 PM
📣 Blog post alert! 𝐄𝐱𝐩𝐥𝐨𝐫𝐢𝐧𝐠 𝐈𝐜𝐞𝐜𝐡𝐮𝐧𝐤 𝐬𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: 𝐮𝐧𝐭𝐚𝐧𝐠𝐥𝐢𝐧𝐠 𝐒𝟑'𝐬 𝐩𝐫𝐞𝐟𝐢𝐱 𝐬𝐭𝐨𝐫𝐲. This technical post by @functionth.bsky.social dives deep into the internals of how S3 shards data, showing that distributed Icechunk can easily perform 230,000 object reads/sec and beyond. earthmover.io/blog/explori...
Several times some database comp sci nerd has suggested to me that you could just do everything in array land using tabular database tools. Whilst they are technically correct that you _could_, this article convincingly shows why you _should not_ - that would be horribly inefficient.
⭐ We just released the first post in our Fundamentals series. This one is called 𝐓𝐞𝐧𝐬𝐨𝐫𝐬 𝐯𝐬. 𝐓𝐚𝐛𝐥𝐞𝐬 - 𝐖𝐡𝐲 𝐭𝐚𝐛𝐮𝐥𝐚𝐫 𝐭𝐨𝐨𝐥𝐬 𝐭𝐫𝐢𝐩 𝐨𝐯𝐞𝐫 𝐠𝐫𝐢𝐝𝐝𝐞𝐝 𝐝𝐚𝐭𝐚. earthmover.io/blog/tensors...
Fundamentals: Tensors vs. Tables | Earthmover
Why tabular tools trip over gridded data.
earthmover.io
April 3, 2025 at 5:48 PM
Several times some database comp sci nerd has suggested to me that you could just do everything in array land using tabular database tools. Whilst they are technically correct that you _could_, this article convincingly shows why you _should not_ - that would be horribly inefficient.
Reposted by TEGNicholas.bsky.social
I am so excited to share this new project with NASA and @developmentseed.org . By applying Icechunk and the "virtualization" approach, we can make Petabytes of archival data accessible to high performance analytics. Unlocking NASA's cloud archives will have immense downstream benefits for the world.
1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 2:27 PM
I am so excited to share this new project with NASA and @developmentseed.org . By applying Icechunk and the "virtualization" approach, we can make Petabytes of archival data accessible to high performance analytics. Unlocking NASA's cloud archives will have immense downstream benefits for the world.
Very proud to be working on enabling this - the potential improvements for NASA (and eventually all fields of science) are enormous, benefiting users and internal staff.
1/ 🚀 Solving #NASA ’s cloud data dilemma: Icechunk unlocks 100x faster access to archival data formats
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
We're thrilled to publish results from our pilot project with NASA and @developmentseed.org to enable high-performance cloud-native access for NASA’s 100s of petabytes of Earth observation data.
March 28, 2025 at 2:14 PM
Very proud to be working on enabling this - the potential improvements for NASA (and eventually all fields of science) are enormous, benefiting users and internal staff.
Who else is going to @cloudnativegeo.org conference this year?
Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis.
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
March 24, 2025 at 8:55 PM
Who else is going to @cloudnativegeo.org conference this year?
Reposted by TEGNicholas.bsky.social
Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis.
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
March 24, 2025 at 6:31 PM
Register to attend our team's workshop at @cloudnativegeo.org! The workshop will teach attendees how to use the @zarr.dev data format model for cloud-native geospatial datacube analysis.
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
🗓️ Apr 30-May 2
📍Snowbird, Utah
🔗 Register here: 2025-ut.cloudnativegeo.org
With Flux all the data engineers serving climate and weather data for maps, timeseries, and scientific analysis will no longer need to run their own servers
Join our upcoming webinar to learn about our new product, Flux. Flux hosts standards-compliant WMS, EDR, and OPeNDAP endpoints, allowing seamless integration between Arraylake data and other tools, including web maps, GIS applications, and even Excel spreadsheets. Register here: bit.ly/4kEGeaV
March 20, 2025 at 4:18 PM
With Flux all the data engineers serving climate and weather data for maps, timeseries, and scientific analysis will no longer need to run their own servers
Reposted by TEGNicholas.bsky.social
You nailed it a while back.
March 19, 2025 at 2:34 PM
You nailed it a while back.