ARC: Benchmark Detailed Guide

reasoning CC-BY-SA-4.0 Updated 2018-02-01

Detailed guide to ARC: category, metrics, sources and applicable models.

Overview

小学级别科学考试问题集，分为Easy和Challenge两个子集，挑战子集需要推理能力

Metrics

Metric Unit Direction
challenge_acc % ↑ Higher is better
easy_acc % ↑ Higher is better

Sources

Model Score Ranking

# Model Vendor Score
1 Claude 3.5 Sonnet (2024-10-22) anthropic 96.8
2 GPT-4o (2024-05-13) openai 96.8
3 o1 openai 96.8
4 Sonar Reasoning other 96.2
5 o1 Preview openai 96
6 Command Nightly cohere 95.9
7 Gemini 1.5 Pro 002 google 95.9
8 GPT-4o (2024-08-06) openai 95.9
9 Gemini 1.0 Ultra google 95.7
10 Gemini 2.0 Flash google 95.7
11 Qwen1.5 110B alibaba 95.7
12 Yi Large Turbo other 95.6
13 GPT-4 0125 Preview openai 95.5
14 Yi Vision other 95.1
15 Gemini 2.0 Flash Thinking google 95
16 GPT-4 32K openai 95
17 StableLM 2 12B other 94.9
18 Claude 3 Sonnet anthropic 94.8
19 Claude 3 Sonnet (2024-02-29) anthropic 94.8
20 Sonar Huge other 94.8
21 Yi Large other 94.8
22 DeepSeek V2 Chat deepseek 94.6
23 GPT-4 Vision Preview openai 94.6
24 Llama 3.2 90B Vision meta 94.6
25 Mistral Medium mistral 94.6
26 Qwen2 57B alibaba 94.6
27 Claude 3 Opus anthropic 94.5
28 Gemini 1.5 Flash-8B google 94.5
29 GLM-4 Plus other 94.5
30 Hermes 3 Llama 3.1 405B other 94.5
31 Nous Hermes 2 Mixtral 8x7B other 94.5
32 Command R (08-2024) cohere 94.4
33 Qwen2 72B alibaba 94.4
34 Claude 3 Haiku anthropic 94.3
35 GPT-4 1106 Preview openai 94.3
36 GLM-4 Flash other 94.2
37 GPT-4 Turbo openai 94.2
38 Jamba 1.5 other 94.2
39 Qwen2.5 14B alibaba 94.2
40 Gemini 1.0 Pro google 94.1
41 GPT-4 openai 94.1
42 Sonar Large other 94.1
43 DeepSeek LLM 67B deepseek 94
44 Command R cohere 93.9
45 DBRX Instruct other 93.9
46 GPT-4o mini openai 93.9
47 Llama 3.3 70B meta 93.9
48 Qwen2.5 32B alibaba 93.9
49 Jamba 1.5 Large other 93.8
50 Llama 3.1 Nemotron 70B meta 93.7
51 o1 mini openai 93.7
52 Yi 1.5 34B other 93.7
53 Claude 3 Opus (2024-02-29) anthropic 93.4
54 Command R+ (08-2024) cohere 93.4
55 Gemini 1.5 Flash google 93.4
56 Llama 3 70B meta 93.4
57 Nous Hermes 2 Yi 34B other 93.4
58 DBRX Base other 93.3
59 GPT-4 Vision openai 93.3
60 Grok-2 Vision xai 93.3
61 Mistral Large mistral 93.3
62 Gemini 1.5 Flash-8B 002 google 93.2
63 NVIDIA Llama 3.1 Nemotron 70B other 93.2
64 Gemma 2 27B google 93.1
65 Qwen1.5 14B alibaba 92.7
66 Gemini 1.0 Flash google 92.6
67 Qwen1.5 72B alibaba 92.6
68 GLM-4 Air other 92.3
69 Llama 3.1 70B meta 92.3
70 WizardLM Team WizardLM 2 8x22B other 92.3
71 DeepSeek Coder 33B deepseek 92.2
72 Grok-2 Mini xai 92.2
73 DeepSeek V2 deepseek 92.1
74 Microsoft WizardLM 2 8x22B other 92.1
75 Phi-3.5 MoE other 91.9
76 Claude 3.5 Haiku anthropic 91.8
77 Gemini 1.5 Flash 002 google 91.8
78 Jamba 1.5 Mini other 91.8
79 Mistral Small 3 mistral 91.8
80 OLMo 1.7 7B other 91.8
81 Sonar Small other 91.7
82 StarChat2 15B v0.1 other 91.7
83 Claude 3 Haiku (2024-03-07) anthropic 91.6
84 Falcon 180B other 91.6
85 Mixtral 8x7B mistral 91.6
86 Mathstral 7B mistral 91.5
87 Hermes 3 Llama 3.1 70B other 91.4
88 Microsoft WizardMath 7B v1 other 91.4
89 Orca 2 13B other 91.4
90 Qwen1.5 32B alibaba 91.4
91 Zephyr ORPO 141B Alpha other 91.4
92 Jamba Instruct other 91.2
93 Llama 3 8B meta 91.2
94 Microsoft WizardCoder Python 34B other 91.2
95 Mistral Small mistral 91.2
96 Command R7B cohere 90.7
97 DeepSeek Math 7B deepseek 90.5
98 Mistral Nemo mistral 90.5
99 StarCoder2 7B other 90.5
100 Gemma 2 9B google 90.3
101 GLM-4V 9B other 90.2
102 Code Llama 7B meta 90.1
103 Phi-3 Medium other 90.1
104 Argilla Notus 7B v1 other 89.8
105 Code Llama 70B meta 89.8
106 Phi-3 Small other 89.6
107 Llama 2 13B meta 89.5
108 OLMo 7B SFT other 89.3
109 OpenChat 3.6 8B other 89.2
110 PaLM 2 google 89.2
111 ChatGLM3 6B other 88.9
112 Mistral 7B v0.3 mistral 88.9
113 Flan-UL2 other 88.8
114 Code Llama 34B meta 88.7
115 Hermes 3 Llama 3.1 8B other 88.7
116 Mistral 7B v0.1 mistral 88.7
117 GLM-4 9B Chat other 88.6
118 DeepSeek Coder V2 deepseek 88.5
119 Gemma 7B google 88.4
120 Phi-1 other 88.4
121 Qwen2.5 7B alibaba 88.4
122 Yi 6B other 88.4
123 Grok Vision Beta xai 88.2
124 OpenChat 3.5 1210 other 88.1
125 Flan-T5 XL other 88
126 OLMo 7B Instruct other 88
127 TigerBot 70B Chat other 87.9
128 Code Llama 13B meta 87.8
129 Yi 1.5 9B other 87.8
130 StarCoder2 15B other 87.6
131 MPT 7B other 87.5
132 StarCoder2 3B other 87.5
133 Codestral mistral 87.4
134 Codestral Mamba mistral 87.4
135 Llama 3.1 8B meta 87.4
136 Llemma 7B other 87.4
137 Yi 34B other 87.2
138 Nous Capybara 7B other 87
139 GPT-3.5 Turbo openai 86.8
140 Qwen2 7B alibaba 86.8
141 StableLM 3 4B other 86.8
142 Text Bison google 86.7
143 Llama 2 7B meta 86.5
144 OLMo 7B other 86.4
145 StableCode 3B other 86.4
146 DeepSeek Coder 7B deepseek 86.3
147 OLMo 2 1124 7B other 86.2
148 Yi 1.5 6B other 86.2
149 Code Bison google 86.1
150 Baichuan2 7B Chat other 86
151 Jurassic-2 Mid other 86
152 Llama 3.2 11B Vision meta 86
153 Command Light cohere 85.9
154 Nous Hermes 2 Solar 10.7B other 85.9
155 Microsoft WizardLM 2 7B other 85.6
156 Ministral 8B mistral 85.5
157 Mistral 7B v0.2 mistral 85.5
158 Qwen2 1.5B alibaba 85.4
159 Qwen2.5 0.5B alibaba 85.4
160 Phi-4 other 85.1
161 Claude 2.1 anthropic 84.8
162 Zephyr 7B Alpha other 84.7
163 Phi-3 Mini other 84.4
164 Baichuan2 13B Chat other 83.2
165 GPT-3.5 Turbo 16K openai 83.1
166 Falcon 40B other 82.9
167 Chat Bison google 82.6
168 Mistral Tiny mistral 82.4
169 Jurassic-2 Ultra other 82.1
170 StableLM Zephyr 3B other 82.1
171 GPT-3.5 openai 81.8
172 Zephyr 7B Beta other 81.8
173 Flan-T5 XXL other 81.7
174 Grok Beta xai 81.7
175 Ministral 3B mistral 81.5
176 Phi-3.5 Mini other 81.5
177 Claude Instant 1 anthropic 80.9
178 MPT 30B other 80.7
179 Nous Capybara 34B other 80.7
180 Claude 2 anthropic 80.4
181 Llama 2 70B meta 80.3
182 Capybara 1.5B other 80.1
183 Open-Platypus other 80.1
184 Gemma 2B google 77.7
185 StableLM 2 1.6B other 77.7
186 Phi-3 Vision other 77.4
187 Qwen2.5 1.5B alibaba 76.7
188 Qwen2.5 3B alibaba 76.5
189 Llama 3.2 3B meta 76.3
190 Llama Guard 2 8B meta 76.3
191 Llama Guard 3 8B meta 76
192 Llama 3.2 1B meta 75.9
193 Phi-1.5 other 75.9
194 Phi-2 other 75.1
195 Embed English v3 cohere 0

ARC

Description

小学级别科学考试问题集，分为Easy和Challenge两个子集，挑战子集需要推理能力

Core Specifications

Category License Last Updated
reasoning CC-BY-SA-4.0 2018-02-01

Benchmark

Unit
%
%

Sources

Official URL