Rohan Desai
@rohanpd.bsky.social
Co-Founder of @responsive.dev. Views are my own.
Reposted by Rohan Desai
This will be the foundation for being able to branch @kafkastreams.bsky.social apps to support seamless blue/green deploys. It also allows you to branch a previous version of the state to debug an issue from the past.
Powerful stuff coming to the world of stream processing!
Powerful stuff coming to the world of stream processing!
SlateDB now has clones 🤯
Users can clone an existing DB's data to a new location. It's nearly instantaneous since it references the data from the old bucket rather than copying. Writes to the clone update the new location. Compaction lazily merges old data into the new directory./ht @responsive.dev
Users can clone an existing DB's data to a new location. It's nearly instantaneous since it references the data from the old bucket rather than copying. Writes to the clone update the new location. Compaction lazily merges old data into the new directory./ht @responsive.dev
Initial clone implementation by hachikuji · Pull Request #430 · slatedb/slatedb
Fixes #315. This patch contains the logic to create and initialize clones as outlined in RFC-0004.
github.com
February 3, 2025 at 7:06 PM
This will be the foundation for being able to branch @kafkastreams.bsky.social apps to support seamless blue/green deploys. It also allows you to branch a previous version of the state to debug an issue from the past.
Powerful stuff coming to the world of stream processing!
Powerful stuff coming to the world of stream processing!
Reposted by Rohan Desai
For any Kafka Streams fans out there who want to know what it's like scaling an EOS application up to multiple TB of state at ~10k+ records/s with minimal downtime -- make sure to catch the Q&A we're hosting tomorrow with Metronome.
Join us Dec 12th at 9:30am PST!
www.linkedin.com/events/72647...
Join us Dec 12th at 9:30am PST!
www.linkedin.com/events/72647...
Q&A: Scaling Event Driven Architectures with Metronome | LinkedIn
We're hosting a live Q&A session with two engineers who have seen it all:
- Casey Crites, a founding engineer at Metronome, implemented and scaled the Kafka pipeline at Metronome from zero to where i...
www.linkedin.com
December 11, 2024 at 11:26 PM
For any Kafka Streams fans out there who want to know what it's like scaling an EOS application up to multiple TB of state at ~10k+ records/s with minimal downtime -- make sure to catch the Q&A we're hosting tomorrow with Metronome.
Join us Dec 12th at 9:30am PST!
www.linkedin.com/events/72647...
Join us Dec 12th at 9:30am PST!
www.linkedin.com/events/72647...
Reposted by Rohan Desai
This was a fun conversation with @thegeeknarrator.bsky.social and @yingjunwu.bsky.social about the world of SQL stream processing on one side and event driven apps on the other. Thanks for having me on the show, Kaivalya!
Excited to announce the next episode on The GeekNarrator podcast on Stream Processing.
Really great insights from two different worlds of stream processing, @responsive.dev and @risingwave.bsky.social
Really great insights from two different worlds of stream processing, @responsive.dev and @risingwave.bsky.social
World of Stream Processing with Apurva Mehta and Yingjun Wu
YouTube video by The Geek Narrator
youtu.be
December 4, 2024 at 7:33 PM
This was a fun conversation with @thegeeknarrator.bsky.social and @yingjunwu.bsky.social about the world of SQL stream processing on one side and event driven apps on the other. Thanks for having me on the show, Kaivalya!
Reposted by Rohan Desai
It's said that Silicon Valley is special because the density of smart and motivated people leads to chance encounters that don't happen elsewhere.
I can attest to that.
Here's how a coffee resulted in @responsive.dev building a database optimized for stream processing in 8 months. (1/n)
I can attest to that.
Here's how a coffee resulted in @responsive.dev building a database optimized for stream processing in 8 months. (1/n)
November 26, 2024 at 5:35 PM
It's said that Silicon Valley is special because the density of smart and motivated people leads to chance encounters that don't happen elsewhere.
I can attest to that.
Here's how a coffee resulted in @responsive.dev building a database optimized for stream processing in 8 months. (1/n)
I can attest to that.
Here's how a coffee resulted in @responsive.dev building a database optimized for stream processing in 8 months. (1/n)
Reposted by Rohan Desai
From one of @responsive.dev 's customers:
"will report back .. when we start to change ttl lengths, which we're currently set up to do through launchdarkly based on the client and environment...pretty slick."
They are changing the TTL of rows in their Kafka Streams state stores dynamically 🤩
"will report back .. when we start to change ttl lengths, which we're currently set up to do through launchdarkly based on the client and environment...pretty slick."
They are changing the TTL of rows in their Kafka Streams state stores dynamically 🤩
November 16, 2024 at 11:12 PM
From one of @responsive.dev 's customers:
"will report back .. when we start to change ttl lengths, which we're currently set up to do through launchdarkly based on the client and environment...pretty slick."
They are changing the TTL of rows in their Kafka Streams state stores dynamically 🤩
"will report back .. when we start to change ttl lengths, which we're currently set up to do through launchdarkly based on the client and environment...pretty slick."
They are changing the TTL of rows in their Kafka Streams state stores dynamically 🤩
Reposted by Rohan Desai
Welcome @ableegoldman.bsky.social to BlueSky! Kafka Streams extraordinaire!
November 20, 2024 at 7:04 AM
Welcome @ableegoldman.bsky.social to BlueSky! Kafka Streams extraordinaire!
Reposted by Rohan Desai
Some problems are impossible to solve without stream processing: for instance, did you know that metronome.com leverages Kafka and Kafka Streams to deliver real time billing features like spend limits at scale? (1/3)
November 13, 2024 at 4:33 PM
Some problems are impossible to solve without stream processing: for instance, did you know that metronome.com leverages Kafka and Kafka Streams to deliver real time billing features like spend limits at scale? (1/3)
Reposted by Rohan Desai
Coming from Kafka land, I’m always surprised at how “time” is an afterthought in many other databases.
Hoping SlateDB gets it right: 1) have only one clock, 2) let users specify the clock, 3) enforce monotonic clocks, 4) use seq numbers (not time) for txns.
Are we missing anything?
Hoping SlateDB gets it right: 1) have only one clock, 2) let users specify the clock, 3) enforce monotonic clocks, 4) use seq numbers (not time) for txns.
Are we missing anything?
November 10, 2024 at 5:14 PM
Coming from Kafka land, I’m always surprised at how “time” is an afterthought in many other databases.
Hoping SlateDB gets it right: 1) have only one clock, 2) let users specify the clock, 3) enforce monotonic clocks, 4) use seq numbers (not time) for txns.
Are we missing anything?
Hoping SlateDB gets it right: 1) have only one clock, 2) let users specify the clock, 3) enforce monotonic clocks, 4) use seq numbers (not time) for txns.
Are we missing anything?
Reposted by Rohan Desai
Looking forward to this! We just started using SlateDB in @tensorlake.bsky.social’s compute engine, looking forward to not needing dynamodb for CAS. One less moving part!
@chris.blue when is the new release coming? :)
@chris.blue when is the new release coming? :)
And it's done. Thank you so much @benesch.bsky.social for doing this. Now SlateDB will be able to have CAS for all major object stores. 🔥 It just works.
object_store: Conditional put and rename_if_not_exist on S3 · Issue #6285 · apache/arrow-rs
Is your feature request related to a problem or challenge? Please describe what you are trying to do. This is now supported on S3: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-condi...
github.com
November 9, 2024 at 4:07 PM
Looking forward to this! We just started using SlateDB in @tensorlake.bsky.social’s compute engine, looking forward to not needing dynamodb for CAS. One less moving part!
@chris.blue when is the new release coming? :)
@chris.blue when is the new release coming? :)
Reposted by Rohan Desai
And it's done. Thank you so much @benesch.bsky.social for doing this. Now SlateDB will be able to have CAS for all major object stores. 🔥 It just works.
object_store: Conditional put and rename_if_not_exist on S3 · Issue #6285 · apache/arrow-rs
Is your feature request related to a problem or challenge? Please describe what you are trying to do. This is now supported on S3: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-condi...
github.com
November 8, 2024 at 8:02 PM
And it's done. Thank you so much @benesch.bsky.social for doing this. Now SlateDB will be able to have CAS for all major object stores. 🔥 It just works.
Reposted by Rohan Desai
The new stack just put up a post on SlateDB!
SlateDB: 'Bottomless' Databases Built on Cloud Object Stores
SlateDB can dramatically cut costs of running a key-value store in the cloud, as long as users don't mind a bit of latency.
thenewstack.io
November 7, 2024 at 5:20 PM
The new stack just put up a post on SlateDB!
Reposted by Rohan Desai
👋 @vigneshc.bsky.social! All, Vinesh is a maintainer and committer for slatedb.io. He implemented the manifest and has been working on fizzbee.io proofs with JP. In his spare time, he’s a manager for Azure’s streaming team. 😁
November 3, 2024 at 5:39 AM
👋 @vigneshc.bsky.social! All, Vinesh is a maintainer and committer for slatedb.io. He implemented the manifest and has been working on fizzbee.io proofs with JP. In his spare time, he’s a manager for Azure’s streaming team. 😁
Reposted by Rohan Desai
I am soon interviewing @apurvamehta.com from @responsive.dev and Yingjun Wu from @risingwave.bsky.social on all things stream processing.
This is going to be an amazing episode. Feel free to ask any questions related to Stream processing and I will add the most interesting ones. Shoot!
This is going to be an amazing episode. Feel free to ask any questions related to Stream processing and I will add the most interesting ones. Shoot!
November 3, 2024 at 5:57 PM
I am soon interviewing @apurvamehta.com from @responsive.dev and Yingjun Wu from @risingwave.bsky.social on all things stream processing.
This is going to be an amazing episode. Feel free to ask any questions related to Stream processing and I will add the most interesting ones. Shoot!
This is going to be an amazing episode. Feel free to ask any questions related to Stream processing and I will add the most interesting ones. Shoot!
Reposted by Rohan Desai
Is it end of the road for RocksDB in stream processing?
Disaggregated state is the clearly superior architecture, with @responsive.dev investing heavily in SlateDB.io while Flink 2.0 has forked RocksDB.
Here's why we've bet on SlateDB for Kafka Streams: www.responsive.dev/blog/why-sla...
Disaggregated state is the clearly superior architecture, with @responsive.dev investing heavily in SlateDB.io while Flink 2.0 has forked RocksDB.
Here's why we've bet on SlateDB for Kafka Streams: www.responsive.dev/blog/why-sla...
If not RocksDB, then what? Why SlateDB is the right choice for Stream Processing.
Why we think SlateDB is the right foundation for a state storage service for Kafka Streams.
www.responsive.dev
October 29, 2024 at 4:51 PM
Is it end of the road for RocksDB in stream processing?
Disaggregated state is the clearly superior architecture, with @responsive.dev investing heavily in SlateDB.io while Flink 2.0 has forked RocksDB.
Here's why we've bet on SlateDB for Kafka Streams: www.responsive.dev/blog/why-sla...
Disaggregated state is the clearly superior architecture, with @responsive.dev investing heavily in SlateDB.io while Flink 2.0 has forked RocksDB.
Here's why we've bet on SlateDB for Kafka Streams: www.responsive.dev/blog/why-sla...
Reposted by Rohan Desai
This is gonna be cool...
"The writer will also support initializing a new database from an existing checkpoint. This allows users to 'fork' an instance of SlateDb, allowing it to access all the original db’s data from the checkpoint, but isolate writes to a new db instance."
"The writer will also support initializing a new database from an existing checkpoint. This allows users to 'fork' an instance of SlateDb, allowing it to access all the original db’s data from the checkpoint, but isolate writes to a new db instance."
RFC for supporting checkpoints/clones by rodesai · Pull Request #278 · slatedb/slatedb
A cloud native embedded storage engine built on object storage. - RFC for supporting checkpoints/clones by rodesai · Pull Request #278 · slatedb/slatedb
github.com
October 28, 2024 at 7:31 PM
This is gonna be cool...
"The writer will also support initializing a new database from an existing checkpoint. This allows users to 'fork' an instance of SlateDb, allowing it to access all the original db’s data from the checkpoint, but isolate writes to a new db instance."
"The writer will also support initializing a new database from an existing checkpoint. This allows users to 'fork' an instance of SlateDb, allowing it to access all the original db’s data from the checkpoint, but isolate writes to a new db instance."