Ministral 8B: Complete Benchmark Performance Guide

mistral ministral-8b 2024-10-16 Mistral Research License

In-depth analysis of Ministral 8B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versionministral-8b
Release Date2024-10-16
Context Window128000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseMistral Research License
Documentationhttps://docs.mistral.ai/

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 64.9 % 2024-10-16 5-shot view
HUMANEVAL 63.3 pass@1 2024-10-16 — view
GSM8K 51.7 % 2024-10-16 0-shot CoT view
MATH 39.9 % 2024-10-16 0-shot CoT view
BBH 60 % 2024-10-16 3-shot CoT view
GPQA 35.1 % 2024-10-16 0-shot view
IFEVAL 56.1 % 2024-10-16 prompt_strict view
ARC 85.5 % 2024-10-16 challenge view
MUSR 37 % 2024-10-16 0-shot view
WINOGRANDE 75.3 % 2024-10-16 0-shot view

Pricing

Tier Price Currency
Input$0.1 / MtokUSD
Output$0.1 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-10-16

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Ministral 8B

Model Overview

Mistral Ministral 8B 边缘模型, 128K 上下文, 8B 参数, 为本地与边缘计算优化, 性能超越 Llama 3 8B。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Mistral ministral-8b 2024-10-16 128K text text Mistral Research License

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 64.9 % 5-shot
HumanEval 63.3 pass@1 —
GSM8K (Grade School Math 8K) 51.7 % 0-shot CoT
MATH 39.9 % 0-shot CoT
BBH (BIG-Bench Hard) 60.0 % 3-shot CoT
GPQA 35.1 % 0-shot
IFEval 56.1 % prompt_strict
ARC 85.5 % challenge
MUSR 37.0 % 0-shot
WinoGrande 75.3 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Context window 128K.

Weaknesses

  • Proprietary, not self-hostable.

Use Cases

  • Long document summarization

References