@dev.trystops.xyz
(finally) testing a theory about the database resetting -- I think the app might have been set to use too many database connections, overriding the limits in the db app and causing a restart.
April 28, 2025 at 3:06 AM
(finally) testing a theory about the database resetting -- I think the app might have been set to use too many database connections, overriding the limits in the db app and causing a restart.
scaled up the database from a tiny server to a less tiny server which I think should address the feed getting periodically wiped
April 23, 2025 at 1:08 AM
scaled up the database from a tiny server to a less tiny server which I think should address the feed getting periodically wiped
Reposted
Hi @jay.bsky.team @bsky.app @support.bsky.team sounds like you need a tech writer.
I’d heavily discount my contracting fee so I can finally have a working feed with my friends 😔😔
I’d heavily discount my contracting fee so I can finally have a working feed with my friends 😔😔
April 21, 2025 at 5:45 AM
Hi @jay.bsky.team @bsky.app @support.bsky.team sounds like you need a tech writer.
I’d heavily discount my contracting fee so I can finally have a working feed with my friends 😔😔
I’d heavily discount my contracting fee so I can finally have a working feed with my friends 😔😔
trying out a new thing (bluesky api that only sends you the posts you want instead of all the posts). it might have broken replies showing in the feed though
April 21, 2025 at 5:18 AM
trying out a new thing (bluesky api that only sends you the posts you want instead of all the posts). it might have broken replies showing in the feed though
testing on aws results:
- graviton performance is good, better than I'm currently getting on fly.io
- but not good enough to be able to run the feed on a burstable instance
- m5g.medium is somewhat cheaper than a fly dedicated cpu instance but aws data egress is much more expensive
- graviton performance is good, better than I'm currently getting on fly.io
- but not good enough to be able to run the feed on a burstable instance
- m5g.medium is somewhat cheaper than a fly dedicated cpu instance but aws data egress is much more expensive
April 21, 2025 at 1:51 AM
testing on aws results:
- graviton performance is good, better than I'm currently getting on fly.io
- but not good enough to be able to run the feed on a burstable instance
- m5g.medium is somewhat cheaper than a fly dedicated cpu instance but aws data egress is much more expensive
- graviton performance is good, better than I'm currently getting on fly.io
- but not good enough to be able to run the feed on a burstable instance
- m5g.medium is somewhat cheaper than a fly dedicated cpu instance but aws data egress is much more expensive
learning forbidden knowledge about AWS billing right now
slightly concerned I am going to have to port this thing to an ec2 graviton instance however
April 20, 2025 at 8:21 PM
learning forbidden knowledge about AWS billing right now
think I found the culprit for at least one of these problems. was seeing a lot of graphs that looked like the one attached -- spike in throughput at startup, then dropping down to a low baseline. not a pattern I was seeing on my laptop either
April 19, 2025 at 11:34 PM
think I found the culprit for at least one of these problems. was seeing a lot of graphs that looked like the one attached -- spike in throughput at startup, then dropping down to a low baseline. not a pattern I was seeing on my laptop either
interesting knot to untangle:
- the code as written is ingesting events slower than the firehose can produce them
- instrumentation and logging might be contributing to the problem, but if I remove instrumentation and logging I can't tell what's going on as easily
- the code as written is ingesting events slower than the firehose can produce them
- instrumentation and logging might be contributing to the problem, but if I remove instrumentation and logging I can't tell what's going on as easily
performance problem identified: the feed is not keeping up with the firehose, indexing is lagging post time by almost an hour (and I think growing). some logging I added earlier is possibly a contributor
April 19, 2025 at 10:34 PM
interesting knot to untangle:
- the code as written is ingesting events slower than the firehose can produce them
- instrumentation and logging might be contributing to the problem, but if I remove instrumentation and logging I can't tell what's going on as easily
- the code as written is ingesting events slower than the firehose can produce them
- instrumentation and logging might be contributing to the problem, but if I remove instrumentation and logging I can't tell what's going on as easily
performance problem identified: the feed is not keeping up with the firehose, indexing is lagging post time by almost an hour (and I think growing). some logging I added earlier is possibly a contributor
April 19, 2025 at 9:27 PM
performance problem identified: the feed is not keeping up with the firehose, indexing is lagging post time by almost an hour (and I think growing). some logging I added earlier is possibly a contributor
feed still seems to be misbehaving but check out that latency improvement yesterday
April 19, 2025 at 8:23 PM
feed still seems to be misbehaving but check out that latency improvement yesterday
instead of the smart solution (atomic cursor updates) decided to go for the dumb solution (only run one app instance). if this post goes through I'm going to feel very vindicated
developed a theory about this after work today, then had to go “hang out” with my “friends”. will report back on if this is a postgres race condition tomorrow
trystops feed update: sometimes posts are just not getting sent through the firehose, if my logging is to be believed
April 19, 2025 at 4:13 PM
instead of the smart solution (atomic cursor updates) decided to go for the dumb solution (only run one app instance). if this post goes through I'm going to feel very vindicated
developed a theory about this after work today, then had to go “hang out” with my “friends”. will report back on if this is a postgres race condition tomorrow
trystops feed update: sometimes posts are just not getting sent through the firehose, if my logging is to be believed
April 19, 2025 at 5:26 AM
developed a theory about this after work today, then had to go “hang out” with my “friends”. will report back on if this is a postgres race condition tomorrow