Project Benchmarks

click on snapshot for details

    model data provided by Surge AI
  1. Sep 1 September 1, 2025
  2. Nov 1 November 1, 2025
  3. Nov 26 November 26, 2025
  4. Mar 23 March 23, 2026
  5. Apr 11 April 11, 2026
  6. Apr 30 April 30, 2026

Published on November 26, 2025

140
research-level problems
56
contributing researchers
Algebra, Combinatorics & Geometry
main areas
Now including Gemini 3 Pro, GPT 5.1, and Claude Opus 4.5, each performing significantly better.
Model Name Model Type Correct Answer
Gemini 3 Pro Active Model 46%
GPT-5.1 Active Model 41%
GPT-5 Legacy Model 35%
Grok-4 Active Model 23%
o3 Active Model 23%
Claude Opus 4.5 Active Model 20%
DeepSeek-V3.1 Active Model 20%
Gemini 2.5 Pro Legacy Model 20%
o3-mini Legacy Model 19%
DeepSeek R1 Legacy Model 18%
Claude Opus 4.1 Legacy Model 11%
Gemini 2.5 Flash Legacy Model 10%
Claude Sonnet 4 Legacy Model 8%
Based on 140 submissions that stump at least 1 active model. All models were queried via the API, using the strongest available version.
Model Name Model Type Correct Answer
Gemini 3 Pro Active Model 40%
GPT-5.1 Active Model 33%
GPT-5 Legacy Model 29%
o3 Active Model 17%
Grok-4 Active Model 16%
DeepSeek R1 Legacy Model 15%
Gemini 2.5 Pro Legacy Model 15%
o3-mini Legacy Model 15%
DeepSeek-V3.1 Active Model 14%
Claude Opus 4.5 Active Model 13%
Claude Opus 4.1 Legacy Model 7%
Claude Sonnet 4 Legacy Model 7%
Gemini 2.5 Flash Legacy Model 6%
Based on 130 submissions that stump at least 2 active models. All models were queried via the API, using the strongest available version.

Sample Problems

Commutative Algebra Algebraic Combinatorics
Let $R=\mathbb{C}[x_1,\ldots,x_6,y_1,\ldots,y_6]$ with bigrading $\deg(x_i)=(1,0)$, $\deg(y_i)=(0,1).$ Let $I\subset R$ be the bihomogeneous ideal generated by the $2$-minors of the generic matrix \[\begin{pmatrix} x_1 & x_2 & \cdots & x_6 \\ y_1 & y_2 & \cdots & y_6\end{pmatrix}.\] What is the dimension of the $\mathbb{K}$-vector space $(I^2/I^3)_{(5,2)}$?
Metric Geometry
Let $B$ be the unit ball in $\mathbb{R}^{8193}$ with respect to the standard Euclidean norm. What is the smallest natural number $r$ such that there exist hermitian $r\times r$ matrices $A_0,\ldots,A_{8193}$ with $B=\{p\in\mathbb{R}^{1025}\mid A_0+p_1\cdot A_1+\cdots+p_{8193}\cdot A_{8193}\textrm{ is positive semidefinite}\}$?
Algebraic Combinatorics Commutative Algebra
Let $W$ be the Weyl algebra $\mathbb{C}\left<D,X \mid DX - XD = 1\right>$; this is the $\mathbb{C}$-algebra with two generators $D$ and $X$ and a single relation $DX - XD = 1$. Given an integer $n \ge 0$, we define an *$n$-monomial element* to be an element of $W$ that can be written as a product of $n$ generators from the set $\left\{D,X\right\}$, i.e., as $G_1G_2\cdots G_n$ where each $G_i \in \left\{D,X\right\}$. Note that some $n$-monomial elements can be written in several ways in such a form (for instance, $DXXD = XDDX$). Let $a_n$ denote the number of $n$-monomial elements. Find $a_{11} + a_{12}$.