Jon Mellon
@jonmellon.bsky.social
Co-director British Election Study. Political Scientist and Data Scientist. Political science methods/political behavior/causal inference. Posts do not represent employer.
Come see our work on using LLMs to track political science research methods at APSA at 4pm today!
We find the latest LLMs beat graduate student RAs at extracting information from academic papers
We find the latest LLMs beat graduate student RAs at extracting information from academic papers
September 13, 2025 at 8:02 PM
Come see our work on using LLMs to track political science research methods at APSA at 4pm today!
We find the latest LLMs beat graduate student RAs at extracting information from academic papers
We find the latest LLMs beat graduate student RAs at extracting information from academic papers
can anyone see a downside to the plan of ordering this desktop specifically for the narrow set of workflows where I have really high RAM requirements and using a modern Macbook pro for every thing else?
August 14, 2025 at 12:51 AM
can anyone see a downside to the plan of ordering this desktop specifically for the narrow set of workflows where I have really high RAM requirements and using a modern Macbook pro for every thing else?
To give a concrete example, we tested LLMs on coding open text survey responses into categories. The best LLMs had human-level accuracy (94%) so we've adopted this. If the accuracy had been 80% we would have tried again in a year to see if the performance had improved
August 9, 2025 at 4:25 PM
To give a concrete example, we tested LLMs on coding open text survey responses into categories. The best LLMs had human-level accuracy (94%) so we've adopted this. If the accuracy had been 80% we would have tried again in a year to see if the performance had improved
I suspect it's easier to just train it to know that it sucks at character manipulation and to fall back on code
August 8, 2025 at 2:00 PM
I suspect it's easier to just train it to know that it sucks at character manipulation and to fall back on code
The underlying cause of this is tokenization that means the LLM doesn't have direct access to characters. Your solution might work, but it also might be brittle unless the LLM reliably encodes how the tokens translate into single characters somewhere.
August 8, 2025 at 1:44 PM
The underlying cause of this is tokenization that means the LLM doesn't have direct access to characters. Your solution might work, but it also might be brittle unless the LLM reliably encodes how the tokens translate into single characters somewhere.
Paid research assistant opportunity with the British Election Study for UK-based graduate students. Help us validate turnout in our surveys against Electoral Commission records
July 17, 2025 at 4:55 PM
Paid research assistant opportunity with the British Election Study for UK-based graduate students. Help us validate turnout in our surveys against Electoral Commission records
The latter one could still be misused but it is work to get it to cheat whereas it is work to get chatGPT not to cheat. Here's the system prompt.
July 8, 2025 at 12:52 PM
The latter one could still be misused but it is work to get it to cheat whereas it is work to get chatGPT not to cheat. Here's the system prompt.
One thing that is underdiscussed around AI and education is how much AI chat interfaces are setup to default to helping students cheat. It's also really easy to prompt them not to do this. Compare standard ChatGPT's answer to a custom prompt that tells the AI to make the student think critically
July 8, 2025 at 12:49 PM
One thing that is underdiscussed around AI and education is how much AI chat interfaces are setup to default to helping students cheat. It's also really easy to prompt them not to do this. Compare standard ChatGPT's answer to a custom prompt that tells the AI to make the student think critically
Here's the twitter poll I did on this a while back
May 27, 2025 at 5:19 PM
Here's the twitter poll I did on this a while back
While there’s a lot of concern about parts of education that AI is breaking, it also enables types of student experiences that were previously infeasible. My student capstone group built AI bots that act as enlisted troops for them to practice counseling skills on
May 15, 2025 at 11:44 PM
While there’s a lot of concern about parts of education that AI is breaking, it also enables types of student experiences that were previously infeasible. My student capstone group built AI bots that act as enlisted troops for them to practice counseling skills on
How do you feel about plotting an estimate's posterior density?
April 24, 2025 at 8:15 PM
How do you feel about plotting an estimate's posterior density?
Tikz for mathy ones and gephi for pretty ones (with as much of the work as possible done in rgephi)
April 13, 2025 at 2:42 PM
Tikz for mathy ones and gephi for pretty ones (with as much of the work as possible done in rgephi)
I talk a fair bit about how impressive LLMs can be for data extraction. But sometimes they also just stink. This is a very poor contextual reading of what the variable "2010 Labour" means in this regression table
April 8, 2025 at 1:01 PM
I talk a fair bit about how impressive LLMs can be for data extraction. But sometimes they also just stink. This is a very poor contextual reading of what the variable "2010 Labour" means in this regression table
Here's the ideal versus more plausible DAG
March 24, 2025 at 7:08 PM
Here's the ideal versus more plausible DAG
Here's the LLM-labeling system we're using for our paper on extracting information from polisci papers (osf.io/v7fe8/). Can you follow the logic based just on the diagram? Is there anything that could be clearer?
March 11, 2025 at 12:14 PM
Here's the LLM-labeling system we're using for our paper on extracting information from polisci papers (osf.io/v7fe8/). Can you follow the logic based just on the diagram? Is there anything that could be clearer?
Thinking through the pre-registration for an LLM validation study and realized I hadn't got a good set of benchmarks for sensitivity and specificity in my head.
March 3, 2025 at 8:32 PM
Thinking through the pre-registration for an LLM validation study and realized I hadn't got a good set of benchmarks for sensitivity and specificity in my head.
It knows it's being lazy too.
February 26, 2025 at 8:33 PM
It knows it's being lazy too.
Criminology has its own separate list. But AJS not cracking the top 20 is still wild.
February 26, 2025 at 8:26 PM
Criminology has its own separate list. But AJS not cracking the top 20 is still wild.
"Programming" in 2025
February 26, 2025 at 8:03 PM
"Programming" in 2025
also apparently AJS doesn't make the top 20 sociology journals of google scholar?
February 26, 2025 at 5:50 PM
also apparently AJS doesn't make the top 20 sociology journals of google scholar?
This is our first stab at defining interpretivist methods/epistemology. Interested in your thoughts
February 20, 2025 at 2:05 PM
This is our first stab at defining interpretivist methods/epistemology. Interested in your thoughts
We're thinking through how to represent the average of the political science literature.
Does it make sense to take the article as the core unit or the journal? e.g. for saying how common survey experiments are over time.
Journals vary dramatically in how much they publish
Does it make sense to take the article as the core unit or the journal? e.g. for saying how common survey experiments are over time.
Journals vary dramatically in how much they publish
February 18, 2025 at 3:01 PM
We're thinking through how to represent the average of the political science literature.
Does it make sense to take the article as the core unit or the journal? e.g. for saying how common survey experiments are over time.
Journals vary dramatically in how much they publish
Does it make sense to take the article as the core unit or the journal? e.g. for saying how common survey experiments are over time.
Journals vary dramatically in how much they publish