DeepSeek Coder V2: Complete Benchmark Performance Guide

deepseek coder-v2 2024-06-21 DeepSeek License

In-depth analysis of DeepSeek Coder V2 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendordeepseek
Versioncoder-v2
Release Date2024-06-21
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseDeepSeek License
Documentationhttps://api-docs.deepseek.com/

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 72.3 % 2024-06-21 5-shot view
HUMANEVAL 88.9 pass@1 2024-06-21 — view
GSM8K 62.9 % 2024-06-21 0-shot CoT view
MATH 38.2 % 2024-06-21 0-shot CoT view
BBH 72.3 % 2024-06-21 3-shot CoT view
GPQA 25.4 % 2024-06-21 0-shot view
IFEVAL 56.7 % 2024-06-21 prompt_strict view
ARC 88.5 % 2024-06-21 challenge view
MUSR 40.3 % 2024-06-21 0-shot view
WINOGRANDE 78.8 % 2024-06-21 0-shot view

Pricing

Tier Price Currency
Input$0.14 / MtokUSD
Output$0.28 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://api-docs.deepseek.com/quick_start/pricing · as of 2024-06-21

Compliance

  • Data Residency: CN
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

DeepSeek Coder V2

Model Overview

DeepSeek Coder V2 代码专用 MoE 模型, 128K 上下文, 支持 338 种编程语言, 代码生成能力突出。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Deepseek coder-v2 2024-06-21 128K text text DeepSeek License

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 72.3 % 5-shot
HumanEval 88.9 pass@1 —
GSM8K (Grade School Math 8K) 62.9 % 0-shot CoT
MATH 38.2 % 0-shot CoT
BBH (BIG-Bench Hard) 72.3 % 3-shot CoT
GPQA 25.4 % 0-shot
IFEval 56.7 % prompt_strict
ARC 88.5 % challenge
MUSR 40.3 % 0-shot
WinoGrande 78.8 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • HumanEval 88.9, excellent code generation.
  • Mixture-of-Experts architecture.

Weaknesses

  • Proprietary, not self-hostable.

Use Cases

  • Code generation and debugging

References