Mistral 7B v0.1: Complete Benchmark Performance Guide

mistral 7b-v0.1 2023-09-27 Apache 2.0

In-depth analysis of Mistral 7B v0.1 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Version7b-v0.1
Release Date2023-09-27
Context Window8192 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://docs.mistral.ai/

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 68.4 % 2023-09-27 5-shot view
HUMANEVAL 56.4 pass@1 2023-09-27 — view
GSM8K 63.1 % 2023-09-27 0-shot CoT view
MATH 26.5 % 2023-09-27 0-shot CoT view
BBH 65.4 % 2023-09-27 3-shot CoT view
GPQA 25.3 % 2023-09-27 0-shot view
IFEVAL 58.9 % 2023-09-27 prompt_strict view
ARC 88.7 % 2023-09-27 challenge view
MUSR 37.4 % 2023-09-27 0-shot view
WINOGRANDE 79.6 % 2023-09-27 0-shot view

Pricing

Tier Price Currency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2023-09-27

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Mistral 7B v0.1

Model Overview

Mistral 7B v0.1 首版开源模型, 8K 上下文, 7B 参数, 在所有 7B 模型中表现领先, Apache 2.0 可商用。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Mistral 7b-v0.1 2023-09-27 8K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 68.4 % 5-shot
HumanEval 56.4 pass@1 —
GSM8K (Grade School Math 8K) 63.1 % 0-shot CoT
MATH 26.5 % 0-shot CoT
BBH (BIG-Bench Hard) 65.4 % 3-shot CoT
GPQA 25.3 % 0-shot
IFEval 58.9 % prompt_strict
ARC 88.7 % challenge
MUSR 37.4 % 0-shot
WinoGrande 79.6 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • Proprietary, not self-hostable.
  • Context window 8K is limited.

Use Cases

  • General chat and Q&A

References