WinoGrande: Benchmark Detailed Guide

reasoning CC-BY-4.0 Updated 2024-02-01

Detailed guide to WinoGrande: category, metrics, sources and applicable models.

Overview

大规模Winograd Schema挑战集，需要常识推理来消除代词指代歧义

Metrics

Metric Unit Direction
accuracy % ↑ Higher is better

Sources

Model Score Ranking

# Model Vendor Score
1 Gemini 2.0 Flash google 89.5
2 GPT-4o (2024-05-13) openai 89.3
3 Gemini 2.0 Flash Thinking google 88.1
4 Claude 3 Opus (2024-02-29) anthropic 87.8
5 GPT-4o (2024-08-06) openai 87.7
6 Sonar Huge other 87.7
7 Grok-2 Vision xai 87.5
8 Sonar Reasoning other 87.2
9 Claude 3.5 Sonnet (2024-10-22) anthropic 86.9
10 GPT-4 Vision openai 86.9
11 Llama 3.3 70B meta 86.8
12 o1 openai 86.7
13 GPT-4 0125 Preview openai 86.6
14 Hermes 3 Llama 3.1 405B other 86.5
15 Gemini 1.5 Pro 002 google 85.8
16 GLM-4 Plus other 85.8
17 o1 Preview openai 85.2
18 Claude 3 Haiku anthropic 84.8
19 Command R cohere 84.8
20 DeepSeek V2 deepseek 84.7
21 Grok-2 Mini xai 84.7
22 Llama 3.1 Nemotron 70B meta 84.7
23 NVIDIA Llama 3.1 Nemotron 70B other 84.7
24 Qwen2.5 14B alibaba 84.7
25 Gemini 1.5 Flash-8B 002 google 84.6
26 Gemini 1.0 Pro google 84.5
27 Jamba 1.5 other 84.5
28 Jamba 1.5 Large other 84.3
29 StableLM 2 12B other 84.1
30 Yi Large other 84.1
31 Qwen1.5 110B alibaba 84
32 Hermes 3 Llama 3.1 70B other 83.9
33 Claude 3.5 Haiku anthropic 83.8
34 Claude 3 Sonnet anthropic 83.8
35 WizardLM Team WizardLM 2 8x22B other 83.5
36 Command R+ (08-2024) cohere 83.4
37 Nous Hermes 2 Yi 34B other 83.4
38 Mistral Large mistral 83.3
39 Qwen1.5 72B alibaba 83.1
40 Qwen2 72B alibaba 82.5
41 Mistral Medium mistral 82.4
42 Yi Large Turbo other 82.3
43 Yi Vision other 82.3
44 GPT-4 32K openai 82.2
45 Nous Hermes 2 Mixtral 8x7B other 82.1
46 GPT-4 1106 Preview openai 82
47 Mixtral 8x7B mistral 82
48 Claude 3 Opus anthropic 81.9
49 Gemini 1.5 Flash 002 google 81.9
50 Qwen2.5 32B alibaba 81.9
51 DBRX Instruct other 81.8
52 Microsoft WizardLM 2 8x22B other 81.6
53 Sonar Small other 81.4
54 GLM-4 Flash other 81.1
55 Sonar Large other 81.1
56 Jamba 1.5 Mini other 81
57 Llama 3.1 70B meta 81
58 Command Nightly cohere 80.8
59 GPT-4 Turbo openai 80.8
60 GPT-4 Vision Preview openai 80.6
61 Gemma 2 27B google 80.5
62 Gemini 1.0 Ultra google 80.4
63 Claude 3 Sonnet (2024-02-29) anthropic 80.3
64 GPT-4 openai 80.3
65 DeepSeek V2 Chat deepseek 80.2
66 Orca 2 13B other 80.2
67 Gemini 1.0 Flash google 80.1
68 Code Llama 34B meta 80
69 Falcon 180B other 80
70 DBRX Base other 79.8
71 DeepSeek Coder 33B deepseek 79.8
72 DeepSeek Coder 7B deepseek 79.8
73 Gemini 1.5 Flash-8B google 79.7
74 Zephyr ORPO 141B Alpha other 79.7
75 GPT-4o mini openai 79.6
76 Mistral 7B v0.1 mistral 79.6
77 Qwen1.5 32B alibaba 79.6
78 Claude 3 Haiku (2024-03-07) anthropic 79.5
79 GLM-4 9B Chat other 79.4
80 Command R (08-2024) cohere 79.3
81 Llama 3.2 90B Vision meta 79.3
82 DeepSeek Math 7B deepseek 79.2
83 Llama 3 70B meta 79.2
84 Yi 1.5 34B other 79.2
85 Gemini 1.5 Flash google 78.9
86 OLMo 7B Instruct other 78.9
87 DeepSeek Coder V2 deepseek 78.8
88 Mistral Small 3 mistral 78.8
89 OLMo 7B SFT other 78.8
90 OLMo 7B other 78.7
91 Qwen1.5 14B alibaba 78.6
92 Jamba Instruct other 78.3
93 o1 mini openai 78.3
94 Phi-3.5 MoE other 78.3
95 Phi-4 other 78.3
96 Mistral Small mistral 78.2
97 DeepSeek LLM 67B deepseek 78.1
98 GLM-4 Air other 78.1
99 Hermes 3 Llama 3.1 8B other 78.1
100 Qwen2 57B alibaba 78.1
101 Code Bison google 77.9
102 MPT 30B other 77.9
103 Llama 3.1 8B meta 77.6
104 Qwen2 7B alibaba 77.3
105 Claude Instant 1 anthropic 77.2
106 Llama 2 7B meta 77.2
107 Nous Capybara 7B other 77.1
108 StarCoder2 3B other 76.9
109 Microsoft WizardLM 2 7B other 76.8
110 GPT-3.5 Turbo openai 76.7
111 OLMo 1.7 7B other 76.6
112 Yi 1.5 9B other 76.6
113 Llama 2 13B meta 76.4
114 Code Llama 7B meta 76.3
115 Flan-UL2 other 76.2
116 Codestral mistral 75.8
117 Llama 3 8B meta 75.6
118 Code Llama 70B meta 75.5
119 Flan-T5 XL other 75.5
120 Ministral 8B mistral 75.3
121 OLMo 2 1124 7B other 75.3
122 StarChat2 15B v0.1 other 75
123 Zephyr 7B Alpha other 75
124 Phi-3 Small other 74.9
125 Command Light cohere 74.7
126 Phi-3 Vision other 74.7
127 OpenChat 3.6 8B other 74.5
128 Phi-1 other 74.5
129 Yi 1.5 6B other 74.5
130 Jurassic-2 Ultra other 74.4
131 Nous Hermes 2 Solar 10.7B other 74.4
132 Flan-T5 XXL other 74.2
133 Codestral Mamba mistral 74.1
134 Command R7B cohere 74.1
135 Gemma 7B google 74.1
136 Phi-1.5 other 74.1
137 StableLM 3 4B other 74.1
138 Text Bison google 74.1
139 GPT-3.5 Turbo 16K openai 74
140 Zephyr 7B Beta other 74
141 Llama 2 70B meta 73.4
142 Mistral Nemo mistral 73.2
143 GLM-4V 9B other 73.1
144 StarCoder2 7B other 73.1
145 Phi-2 other 72.8
146 GPT-3.5 openai 72.7
147 Llama 3.2 11B Vision meta 72.7
148 Mistral 7B v0.3 mistral 72.7
149 Gemma 2 9B google 72.5
150 Yi 6B other 72.2
151 Phi-3 Medium other 72.1
152 Gemma 2B google 72
153 Mistral 7B v0.2 mistral 71.7
154 StableLM Zephyr 3B other 71.6
155 Yi 34B other 71.5
156 Qwen2.5 7B alibaba 71.4
157 Mathstral 7B mistral 71
158 StableCode 3B other 70.9
159 Llemma 7B other 70.8
160 Microsoft WizardMath 7B v1 other 70.5
161 Code Llama 13B meta 70.4
162 Microsoft WizardCoder Python 34B other 70.4
163 StableLM 2 1.6B other 70.3
164 StarCoder2 15B other 70.2
165 Argilla Notus 7B v1 other 69.8
166 Baichuan2 13B Chat other 69.5
167 OpenChat 3.5 1210 other 69.3
168 Grok Beta xai 69.2
169 Capybara 1.5B other 68.9
170 PaLM 2 google 68.6
171 Phi-3.5 Mini other 68.6
172 Baichuan2 7B Chat other 68.3
173 Llama Guard 3 8B meta 68
174 Llama 3.2 3B meta 67.8
175 ChatGLM3 6B other 67.7
176 Grok Vision Beta xai 67.5
177 Claude 2.1 anthropic 67.4
178 Nous Capybara 34B other 67.3
179 Jurassic-2 Mid other 67.1
180 Phi-3 Mini other 67.1
181 Open-Platypus other 66.7
182 Mistral Tiny mistral 66.6
183 Llama Guard 2 8B meta 66.5
184 Claude 2 anthropic 66.4
185 TigerBot 70B Chat other 66.3
186 Chat Bison google 66.2
187 Falcon 40B other 65.6
188 MPT 7B other 65.5
189 Ministral 3B mistral 64.8
190 Qwen2.5 1.5B alibaba 64.4
191 Qwen2.5 0.5B alibaba 64.3
192 Llama 3.2 1B meta 64
193 Qwen2 1.5B alibaba 62.9
194 Qwen2.5 3B alibaba 61.4
195 Embed English v3 cohere 0

WinoGrande

Description

大规模Winograd Schema挑战集，需要常识推理来消除代词指代歧义

Core Specifications

Category License Last Updated
reasoning CC-BY-4.0 2024-02-01

Benchmark

Unit
%

Sources

Official URL