Zach Levonian
LightNews LightNews
banner
zwlevonian.bsky.social
Zach Levonian
@zwlevonian.bsky.social
520 followers 1.3K following 120 posts
Human–computer interaction researcher. PhD from University of Minnesota. Tacoma, WA. Mastodon: zwlevonian@hci.social
Posts Replies Media Videos
zwlevonian.bsky.social
A quick love letter to the Internet Archive's free online book loans: it makes it so easy to verify extremely obscure facts! I'm constantly using it to verify info on Wikipedia.
Info cards for 10 books checked out from the Internet Archive. The collection of books is eclectic, and consists of mostly 20th century reference works.
September 13, 2025 at 6:32 PM
zwlevonian.bsky.social
Thinking again about AI as tarmac. (Here's Fred Turner, writing in 2017.) levon003.github.io/2024/05/19/m...
Fred Turner: Any whole-system approach doesn’t work. What I would recommend is not that we abandon technology, but that we deal with it as an integrated part of our world, and that we engage it the same way that we engage the highway system, the architecture that supports our buildings, or the way we organize hospitals. The technologies that we’ve developed are infrastructures. We don’t have a language yet for infrastructure as politics. And enough magic still clings to the devices that people are very reluctant to start thinking about them as ordinary as tarmac. But we need to start thinking about them as ordinary as tarmac. And we need to develop institutional settings for thinking about how we want to make our traffic laws. To the extent that technologies enable new collaborations and new communities, more power to them. But let’s be thoughtful about how they function.
September 8, 2025 at 6:00 PM
zwlevonian.bsky.social
My team got a nice write-up in a blog post from @thelearningagency.bsky.social. I can't complain about effusive comments from my boss!
the-learning-agency.com/the-cutting-...
Translating Technical Talent into Education Impact What does this support look like in practice? Zach Levonian, a Senior Machine Learning Engineer with Digital Harbor and the LEVI Engineering Hub, has worked closely with the team at Rising Academies to build a data infrastructure and alerting system for their AI-powered math tutor, Rori. This system ensures that Rori can effectively moderate student input and generate safe LLM outputs. “This kind of work requires a broad range of skills,” notes Christie. “It requires back-end development, automated testing, data pipelines, user experience research, and knowledge of the educational domain. It’s difficult to find a single person with this broad of a skillset, but Zach has it.” What makes experts like Levonian even rarer finds is their ability to speak to both technological and educational audiences. Someone may have a strong grasp of natural language processing, but may not have a sense of how to design feedback loops that actually align with student learning needs and progressions. Designed and staffed to support LEVI’s target learning outcome, the Engineering Hub brings that dual perspective.
April 27, 2025 at 6:45 PM
zwlevonian.bsky.social
Big turnout in Olympia, WA #handsoff
Bad photo of the Olympia, WA Capitol Dome. The stairs are covered in protesters, and protest signs are visible in the foreground with anti-Trump and anti-DOGE messages. A transparent protest sign reads "Hands Off Our Democracy" in front of the Olympia Capitol Dome.
April 6, 2025 at 12:45 AM
zwlevonian.bsky.social
There's so much here, and I'm struggling to understand the details! What does "we filtered the principles whose according rewards are aligned with the ground truth" mean in section 2.2? Use some subset of principles that maximize Reward Bench score when included in the reward model, but how?
Text in section 2.2: Generalist RM requires to generate high-quality rewards beyond specific domains (Hendrycks et al., 2021; Jimenez et al., 2024), where the criteria for rewards are more diverse and complex, and there are often no explicit reference or ground truth. To this end, for general domains, we adopt principles to guide reward generation in place of artificial rules. Principles for LLMs are first introduced in Constitutional AI (Bai et al., 2022b; Sharma et al., 2025), which are handicraft criteria that guide the LLMs or curated classifiers to construct safe data pipelines. With principles, the reward generation of GRMs changes to R = C ∼ rθ (x, {y_i}^n_{i=1}, {p_i}^m_{i=1}) where {p_i}^m_{i=1} denotes the principles. We conduct a preliminary experiment to examine the influence of proper principles on reward quality, with the Chat Hard subset of Reward Bench (Lambert et al., 2024) and the IFEval subset of the PPE benchmark (Frick et al., 2025). We used GPT-4o-2024-08-06 to generate the principles and then pointwise rewards four times for each sample. And we filtered the principles whose according rewards are aligned with the ground truth. We tested different LLMs with principles generated by themselves and the filtered principles, and compared them with the default setting with no principle guidance. The results are shown in Table 1. We found that the self-generated principles barely improve performance, but the filtered principles could significantly boost the reward quality. This indicates that proper principles better guide reward generation under correctly summoned criteria. Details are depicted in Appendix D
April 4, 2025 at 10:34 PM
zwlevonian.bsky.social
We cited 155 (!) papers in health, HCI, recsys, and IR for this work. One darling I refused to kill was this table I put together of the many characteristics that researchers have said might make for a good peer recommendation. Not a lot of agreement!
A research table categorizing dozens of papers into four broad peer characteristic categories: Health (e.g. specific diagnosis), Demographics (e.g. age, gender), Life (e.g. geography, social role), and Other (e.g. Communication style, interests).
January 29, 2025 at 8:05 PM
zwlevonian.bsky.social
I highly recommend following @listifications.app just to see the totally wild set of lists you get added to. I would really love to know the inclusion criteria for some of these!
6 screenshots show listifications DMs: @sniffer.on.computer has added you to the "a bit stinky" moderation list @jacksmith.is-extremely.gay has added you to the "Smith Family Reunion Apr 16 Be There!" moderation list @stepheny.gaysky.social has added you to the "NSFW / Pro-Ship" moderation list @hogan.swifties.social has added you to the "Porn Bots" moderation list @handle.invalid has added you to the "Block Furry Posts" moderation list @august.musicsky.social has added you to the "Zionist Allies" custom feed list
January 27, 2025 at 7:21 PM
zwlevonian.bsky.social
Agree with proprietary and online-only, but I don’t think Google uses docs data for training? My understanding may be out of date, please correct me. cloud.google.com/document-ai/...
Does Google use customer data to improve the model(s)? No. Google does not use any of your content (such as documents and predictions) for any purpose except to provide you with the Document AI service. See the Document AI data usage policy. At Google Cloud, we never use customer data to train our Document AI models.
December 26, 2024 at 7:34 PM
zwlevonian.bsky.social
This is a great conceptualization of how LLMs can be useful for search tasks.
I think of the LLM use here a bit in terms of what my colleague Sam Wineburg calls “taking bearing” — when we land in the middle of a cognitive task like this the impulse is to run straight off towards the first issue we think of. Taking bearing means surveying the landscape before running off into your research session. Part of that is understanding what you’re looking at, whether a source, claim, or set of Google results. But I think part of it is this too, getting something to generate a list of relevant questions, the stuff you might not be thinking about, the things that aren’t part of your initial reaction. You can chose the path yourself, but here’s what’s available. There are usually more directions you can go than you think.
December 8, 2024 at 12:15 AM
zwlevonian.bsky.social
Interesting example! I had to look this up, and it looks like this might be an example of a 10pp difference *not* affecting decision-making. www.vanityfair.com/news/2016/11...
Danny and Lanir set out to give Israeli foreign minister Yigal Allon precise numerical estimates of the likelihood of some very specific bad things happening. They assembled a list of possible “critical events or concerns”: regime change in Jordan, U.S. recognition of the Palestinian Liberation Organization, another full-scale war with Syria, and so on. They then surveyed experts and well-informed observers to establish the likelihood of each event. Among these people, they found a remarkable consensus: there wasn’t a lot of disagreement about the odds. When Danny asked the experts what the effect might be of the failure of Kissinger’s negotiations on the probability of war with Syria, for instance, their answers clustered around “raises the chance of war by 10 percent.” Danny and Lanir then presented their probabilities to Israel’s Foreign Ministry. (“The National Gamble,” they called their report.) Foreign Minister Allon looked at the numbers and said, “Ten percent increase? That is a small difference.” Danny was stunned: if a 10 percent increase in the chances of full-scale war with Syria wasn’t enough to interest Allon in Kissinger’s peace process, how much would it take to turn his head? That number represented the best estimate of the odds. Apparently, the foreign minister didn’t want to rely on the best estimates. He preferred his own internal probability calculator: his gut. “That was the moment I gave up on decision analysis,” said Danny. “No one ever made a decision because of a number. They need a story.”
December 3, 2024 at 9:19 PM
zwlevonian.bsky.social
😍😍😍😍
Diagram from "Burning Diagrams in Anthropology: An Inverse Museum" (Tristan Partridge, 2024) Diagram from "Burning Diagrams in Anthropology: An Inverse Museum" (Tristan Partridge, 2024) Diagram from "Burning Diagrams in Anthropology: An Inverse Museum" (Tristan Partridge, 2024) Diagram from "Burning Diagrams in Anthropology: An Inverse Museum" (Tristan Partridge, 2024)
November 21, 2024 at 11:55 PM