Gemini 2.0 Flash: Complete Benchmark Performance Guide

google 2.0-flash 2024-12-11 Proprietary

In-depth analysis of Gemini 2.0 Flash performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorgoogle
Version2.0-flash
Release Date2024-12-11
Context Window1.048576e+06 tokens
Input Modalitiestext, image, audio, video
Output Modalitiestext
LicenseProprietary
Documentationhttps://ai.google.dev/gemini-api/docs

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 86.3 % 2024-12-11 5-shot view
HUMANEVAL 90.7 pass@1 2024-12-11 — view
GSM8K 92.7 % 2024-12-11 0-shot CoT view
MATH 71.3 % 2024-12-11 0-shot CoT view
BBH 83.4 % 2024-12-11 3-shot CoT view
GPQA 55 % 2024-12-11 0-shot view
IFEVAL 85.9 % 2024-12-11 prompt_strict view
ARC 95.7 % 2024-12-11 challenge view
MUSR 69.3 % 2024-12-11 0-shot view
WINOGRANDE 89.5 % 2024-12-11 0-shot view

Pricing

Tier Price Currency
Input$0.1 / MtokUSD
Output$0.4 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.google.dev/pricing · as of 2024-12-11

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Gemini 2.0 Flash

Model Overview

Google Gemini 2.0 Flash 新一代模型, 1M 上下文, 原生工具调用与多模态输出, 性能超越 1.5 Pro。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Google 2.0-flash 2024-12-11 1048K text, image, audio, video text Proprietary

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 86.3 % 5-shot
HumanEval 90.7 pass@1 —
GSM8K (Grade School Math 8K) 92.7 % 0-shot CoT
MATH 71.3 % 0-shot CoT
BBH (BIG-Bench Hard) 83.4 % 3-shot CoT
GPQA 55.0 % 0-shot
IFEval 85.9 % prompt_strict
ARC 95.7 % challenge
MUSR 69.3 % 0-shot
WinoGrande 89.5 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • MMLU score 86.3, strong knowledge reasoning.
  • HumanEval 90.7, excellent code generation.
  • GSM8K 92.7, robust math reasoning.
  • Input Modalities: text, image, audio.

Weaknesses

  • Proprietary, not self-hostable.

Use Cases

  • Code generation and debugging
  • Vision and image understanding
  • Agent workflows and tool use

References