This is approximately what we do today, yea. "Traditional" just has this weird connotation of "legacy." It's also not specific or descriptive. "Normal" also makes BYOC sound abnormal which it is increasingly not.
October 25, 2024 at 8:19 PM
This is approximately what we do today, yea. "Traditional" just has this weird connotation of "legacy." It's also not specific or descriptive. "Normal" also makes BYOC sound abnormal which it is increasingly not.
Ok, fair question! I think it's because of the way it's used. Unmounting and remounting it in a failure case is weird and slow and subject to bad corner cases (local i/o abstraction over a network service). Same issues as NFS, plus or minus.
October 25, 2024 at 4:22 PM
Ok, fair question! I think it's because of the way it's used. Unmounting and remounting it in a failure case is weird and slow and subject to bad corner cases (local i/o abstraction over a network service). Same issues as NFS, plus or minus.
I was lucky enough to see a very large, very high throughput, very margin-sensitive system, with enterprise SLAs in the cloud that informed my most recent opinions on this. Those include never using EBS. Even there, and to the earlier point, a storage service w/ ephemeral disk backed by s3 = better.
October 25, 2024 at 4:20 PM
I was lucky enough to see a very large, very high throughput, very margin-sensitive system, with enterprise SLAs in the cloud that informed my most recent opinions on this. Those include never using EBS. Even there, and to the earlier point, a storage service w/ ephemeral disk backed by s3 = better.
I'm not shilling for rocksdb or anything. Flink's state backends and checkpointing system are due for an update. The Flink 2.x work might be The Way (too early to tell). Just suggesting we don't throw the baby out with the bath water on this one.
October 25, 2024 at 4:14 PM
I'm not shilling for rocksdb or anything. Flink's state backends and checkpointing system are due for an update. The Flink 2.x work might be The Way (too early to tell). Just suggesting we don't throw the baby out with the bath water on this one.
There are valid questions about how things like rescaling and restoration work, but I think there are good patterns for dealing with this that get into the design of the storage layout and primitives. TLDR, it should be possible to page in/flush chunks rather than everything, but that's obvious.
October 25, 2024 at 4:12 PM
There are valid questions about how things like rescaling and restoration work, but I think there are good patterns for dealing with this that get into the design of the storage layout and primitives. TLDR, it should be possible to page in/flush chunks rather than everything, but that's obvious.