Project Benchmarks

click on snapshot for details

    model data provided by Surge AI
  1. Sep 1 September 1, 2025
  2. Nov 1 November 1, 2025
  3. Nov 26 November 26, 2025
  4. Mar 23 March 23, 2026
  5. Apr 11 April 11, 2026
  6. Apr 30 April 30, 2026

Published on November 1, 2025

209
research-level problems
56
contributing researchers
Algebra, Combinatorics & Geometry
main areas
Model Name Model Type Correct Answer
GPT-5 Active Model 43%
Grok-4 Active Model 32%
o3 Active Model 32%
DeepSeek-V3.1 Active Model 31%
Gemini 2.5 Pro Active Model 26%
DeepSeek R1 Legacy Model 23%
o3-mini Legacy Model 23%
Gemini 2.5 Flash Legacy Model 14%
Claude Opus 4.1 Active Model 12%
Claude Sonnet 4 Legacy Model 12%
Based on 200 submissions that stump at least 1 active model. All models were queried via the API, using the strongest available version.
Model Name Model Type Correct Answer
GPT-5 Active Model 36%
Grok-4 Active Model 27%
o3 Active Model 26%
DeepSeek-V3.1 Active Model 25%
DeepSeek R1 Legacy Model 20%
Gemini 2.5 Pro Active Model 20%
o3-mini Legacy Model 19%
Claude Sonnet 4 Legacy Model 9%
Gemini 2.5 Flash Legacy Model 9%
Claude Opus 4.1 Active Model 8%
Based on 180 submissions that stump at least 2 active models. All models were queried via the API, using the strongest available version.

Sample Problems

Matroid Theory Topology
Let $M = (E, \mathcal{B})$ be a matroid. Let $\mathbb{P}L_{M}$ be projectivization of the space of Lorentzian polynomials whose support is precisely equal to the collection $\mathcal{B}$. When $M = F_{7}$, the Fano matroid, find the dimension of $\mathbb{P}L_{M}$.
Homological Algebra
Let $Q = 1 \longrightarrow 2 \longrightarrow 3 \longleftarrow 4$. Consider $M = X_{[1,2]}^2 \oplus X_{[1,3]}^3 \oplus X_{[1,4]}^2 \oplus X_{[2]} \oplus X_{[2,3]}^2 \oplus X_{[2,4]}$. Calculate the generic Jordan form data of $M$.
Combinatorics Discrete Geometry
In a card game each card has three attributes: color, number and shape. Each of these attributes has 4 possible variants. Color: red, blue, green, yellow. Number: 1, 2, 3, 4. Shape: circle, square, triangle, cross. All possible cards appear once in the deck. Define a "quartet" to be a set of four cards from the deck above such that for each of the three attributes, one of the following three possibilities holds for the four values: all four values are the same, all four values are different, or there are exactly two pairs of values. What is the minimum number of cards that must be randomly drawn from the deck so that one can guarantee that there is always at least one quartet among them?