See the paper for more details and experiments: arxiv.org/pdf/2410.13502
Or check out the codebase to generate your own problems: github.com/eth-lre/math...

All models are sensitive to a simple change in sentence ordering, where we take one sentence and move it to the beginning. We also find that the problem is easiest for LLMs if the sentence is moved from near the beginning or end, rather than from the middle!

OpenAI’s o1 and DeepSeek-R1 are certainly impressive. However, when we permuted the ordering of the sentences their performance went down to 5% and 11% respectively (with the token limit set to 25,000 as recommended by OpenAI).

Here are the results for what we call “nonlinear” problems. Solving them requires keeping intermediate results in memory for subsequent steps before they can be used for further deduction. The most complex problems are pretty hard for all models, but they are still able to solve some of them!

We apply MathGAP to perform a systematic analysis on whether LLMs can use simple examples in context to solve more complex ones at inference. Generalization to proof width turns out to be harder than to proof depth, but we see a steady decrease in performance as proofs get both deeper and wider 💡

With our proof system we can generate new MWPs that adhere to the structure of proof trees, as well as ground-truth CoT traces! From the proof trees we then characterize the complexity of reasoning in several ways, e.g., depth, width, shape, and ordering of nodes (i.e., sentences).

Our work builds on a simple observation: Math word problems (MWPs) are deductive reasoning problems, so solving them can be thought of as applying inference rules. We can thus view solution/reasoning traces as proof trees, the structure of which tells us how hard/complex the problem is to solve.