GPQA: Benchmark Detailed Guide

knowledge CC-BY-4.0 Updated 2024-01-01

Detailed guide to GPQA: category, metrics, sources and applicable models.

Overview

由生物学、物理学、化学领域的博士专家撰写的难题集，确保问题难以通过Google搜索直接找到答案

Metrics

Metric Unit Direction
accuracy % ↑ Higher is better

Sources

Model Score Ranking

# Model Vendor Score
1 GPT-4o (2024-08-06) openai 62
2 Gemini 2.0 Flash Thinking google 61
3 GPT-4o (2024-05-13) openai 58.8
4 o1 Preview openai 57.7
5 o1 openai 57.5
6 Gemini 1.5 Pro 002 google 56.5
7 Gemini 2.0 Flash google 55
8 Qwen2 72B alibaba 53.6
9 Llama 3.3 70B meta 52.8
10 Claude 3.5 Sonnet (2024-10-22) anthropic 50.9
11 Jamba 1.5 Large other 50.8
12 Yi Vision other 50.3
13 Sonar Reasoning other 50.1
14 GPT-4 Turbo openai 49.2
15 Mistral Medium mistral 48.7
16 Gemini 1.0 Pro google 47.7
17 Gemini 1.0 Ultra google 46.9
18 GPT-4 0125 Preview openai 46.8
19 Sonar Huge other 46.7
20 Claude 3 Opus anthropic 46
21 Qwen1.5 110B alibaba 45.9
22 Command Nightly cohere 45.5
23 Command R (08-2024) cohere 44.6
24 Qwen2.5 14B alibaba 44.5
25 Claude 3 Haiku anthropic 44.4
26 Command R cohere 44
27 StableLM 2 12B other 44
28 Grok-2 Vision xai 43.8
29 Mistral Small mistral 43.7
30 Falcon 180B other 43.2
31 Mixtral 8x7B mistral 43
32 GPT-4o mini openai 42.6
33 Llama 3.1 Nemotron 70B meta 42.6
34 Orca 2 13B other 42.5
35 Llama 3.2 90B Vision meta 42.2
36 GPT-4 Vision openai 41.8
37 Claude 3 Sonnet (2024-02-29) anthropic 41.7
38 Sonar Large other 41.6
39 WizardLM Team WizardLM 2 8x22B other 41.4
40 GPT-4 32K openai 40.9
41 o1 mini openai 40.8
42 DeepSeek Math 7B deepseek 40.5
43 Hermes 3 Llama 3.1 405B other 40.5
44 Phi-3.5 MoE other 40.2
45 Llama 3.1 70B meta 40
46 Gemini 1.5 Flash-8B 002 google 39.8
47 Qwen1.5 72B alibaba 39.8
48 Claude 3 Haiku (2024-03-07) anthropic 39.6
49 Qwen2.5 32B alibaba 39.4
50 Qwen2 57B alibaba 39.4
51 DBRX Base other 39.3
52 Microsoft WizardLM 2 8x22B other 39.3
53 GPT-4 openai 39.2
54 Nous Hermes 2 Yi 34B other 39
55 Command R+ (08-2024) cohere 38.7
56 Claude 3 Opus (2024-02-29) anthropic 38.6
57 Gemini 1.5 Flash google 38.5
58 DeepSeek V2 Chat deepseek 38.3
59 Llama 3 70B meta 38.2
60 Yi Large Turbo other 38.2
61 Mistral Large mistral 38.1
62 Yi 1.5 9B other 38
63 Gemini 1.0 Flash google 37.7
64 NVIDIA Llama 3.1 Nemotron 70B other 37.6
65 Mathstral 7B mistral 37.5
66 Jamba Instruct other 37.4
67 Llemma 7B other 37.4
68 OLMo 7B other 37.4
69 DeepSeek LLM 67B deepseek 37.3
70 GLM-4 9B Chat other 37.2
71 Gemma 7B google 37.1
72 Zephyr ORPO 141B Alpha other 36.8
73 Microsoft WizardCoder Python 34B other 36.6
74 Code Llama 13B meta 36.4
75 Codestral Mamba mistral 36.3
76 GPT-4 Vision Preview openai 36.3
77 GPT-4 1106 Preview openai 36.2
78 StarCoder2 3B other 36.2
79 Mistral Nemo mistral 36.1
80 Claude 3 Sonnet anthropic 35.9
81 Nous Hermes 2 Solar 10.7B other 35.9
82 Llama 3.2 11B Vision meta 35.5
83 Gemma 2 27B google 35.4
84 GLM-4 Plus other 35.4
85 StarCoder2 15B other 35.3
86 Yi Large other 35.2
87 Ministral 8B mistral 35.1
88 Nous Hermes 2 Mixtral 8x7B other 35.1
89 Gemini 1.5 Flash-8B google 35
90 Open-Platypus other 34.9
91 Llama 3.1 8B meta 34.8
92 Baichuan2 13B Chat other 34.7
93 DBRX Instruct other 34.7
94 Yi 1.5 34B other 34.7
95 Sonar Small other 34.5
96 Zephyr 7B Alpha other 34.3
97 Jamba 1.5 Mini other 34.2
98 MPT 7B other 34.2
99 Mistral 7B v0.2 mistral 33.9
100 Qwen1.5 14B alibaba 33.9
101 StarCoder2 7B other 33.6
102 Baichuan2 7B Chat other 33.5
103 Argilla Notus 7B v1 other 32.7
104 Codestral mistral 32.4
105 Grok Vision Beta xai 32.4
106 Gemini 1.5 Flash 002 google 32
107 OLMo 7B SFT other 31.8
108 Grok-2 Mini xai 31.7
109 Qwen2 7B alibaba 31.6
110 DeepSeek V2 deepseek 31.5
111 Hermes 3 Llama 3.1 8B other 31.5
112 Claude 2 anthropic 31.4
113 GLM-4 Air other 31.3
114 GLM-4 Flash other 31.3
115 Mistral 7B v0.3 mistral 31.2
116 TigerBot 70B Chat other 31.2
117 Claude 3.5 Haiku anthropic 31.1
118 Nous Capybara 7B other 31
119 Command R7B cohere 30.9
120 Phi-3 Small other 30.9
121 Qwen1.5 32B alibaba 30.6
122 OLMo 7B Instruct other 30.5
123 Gemma 2 9B google 30.3
124 OpenChat 3.6 8B other 30.3
125 Jamba 1.5 other 30.2
126 Microsoft WizardMath 7B v1 other 30.2
127 Command Light cohere 30.1
128 Hermes 3 Llama 3.1 70B other 30.1
129 Mistral Small 3 mistral 30.1
130 Phi-4 other 30.1
131 Nous Capybara 34B other 29.7
132 Code Llama 7B meta 29.5
133 Yi 1.5 6B other 29.2
134 Code Bison google 29.1
135 Jurassic-2 Ultra other 28.7
136 Mistral Tiny mistral 28.7
137 Flan-T5 XXL other 28.6
138 Zephyr 7B Beta other 28.6
139 Qwen2.5 7B alibaba 28.5
140 GLM-4V 9B other 28.3
141 Llama 3 8B meta 27.9
142 Llama Guard 3 8B meta 27.9
143 Text Bison google 27.7
144 Flan-UL2 other 27.2
145 Llama 2 70B meta 27.2
146 Code Llama 34B meta 27
147 GPT-3.5 openai 26.9
148 OLMo 1.7 7B other 26.9
149 DeepSeek Coder 33B deepseek 26.8
150 StableLM 3 4B other 26.8
151 Claude 2.1 anthropic 26.4
152 OLMo 2 1124 7B other 26.3
153 Yi 6B other 26.3
154 Microsoft WizardLM 2 7B other 26.1
155 StarChat2 15B v0.1 other 25.9
156 Jurassic-2 Mid other 25.7
157 Phi-3 Medium other 25.6
158 DeepSeek Coder V2 deepseek 25.4
159 Code Llama 70B meta 25.3
160 Mistral 7B v0.1 mistral 25.3
161 Chat Bison google 25
162 StableCode 3B other 24.5
163 Phi-3.5 Mini other 24.4
164 MPT 30B other 24.2
165 DeepSeek Coder 7B deepseek 23.9
166 Qwen2.5 0.5B alibaba 23.2
167 Llama 2 7B meta 23.1
168 Flan-T5 XL other 22.6
169 Llama 3.2 1B meta 22.6
170 GPT-3.5 Turbo openai 22.4
171 Llama 2 13B meta 22.3
172 Phi-1 other 22.1
173 Qwen2 1.5B alibaba 22.1
174 Claude Instant 1 anthropic 21.8
175 Phi-2 other 21.8
176 GPT-3.5 Turbo 16K openai 21.6
177 Falcon 40B other 21.1
178 StableLM Zephyr 3B other 21
179 OpenChat 3.5 1210 other 20.9
180 Phi-3 Mini other 20.7
181 Grok Beta xai 20.6
182 Yi 34B other 20.6
183 PaLM 2 google 20.3
184 Qwen2.5 1.5B alibaba 20.2
185 ChatGLM3 6B other 20.1
186 Ministral 3B mistral 19
187 Llama 3.2 3B meta 18.8
188 Capybara 1.5B other 18.1
189 Gemma 2B google 18
190 Llama Guard 2 8B meta 18
191 Qwen2.5 3B alibaba 18
192 Phi-3 Vision other 17.4
193 Phi-1.5 other 16.9
194 StableLM 2 1.6B other 15
195 Embed English v3 cohere 0

GPQA

Description

由生物学、物理学、化学领域的博士专家撰写的难题集，确保问题难以通过Google搜索直接找到答案

Core Specifications

Category License Last Updated
knowledge CC-BY-4.0 2024-01-01

Benchmark

Unit
%

Sources

Official URL