Mistral Tiny: Complete Benchmark Performance Guide

mistral tiny 2024-02-26 Apache 2.0

In-depth analysis of Mistral Tiny performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendormistral
Versiontiny
Release Date2024-02-26
Context Window32000 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://docs.mistral.ai/

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 54.2 % 2024-02-26 5-shot view
HUMANEVAL 45.8 pass@1 2024-02-26 — view
GSM8K 57.2 % 2024-02-26 0-shot CoT view
MATH 16 % 2024-02-26 0-shot CoT view
BBH 51.3 % 2024-02-26 3-shot CoT view
GPQA 28.7 % 2024-02-26 0-shot view
IFEVAL 51.7 % 2024-02-26 prompt_strict view
ARC 82.4 % 2024-02-26 challenge view
MUSR 41.4 % 2024-02-26 0-shot view
WINOGRANDE 66.6 % 2024-02-26 0-shot view

Pricing

Tier Price Currency
Input$0.15 / MtokUSD
Output$0.15 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://mistral.ai/technology/ · as of 2024-02-26

Compliance

  • Data Residency: EU
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Mistral Tiny

Model Overview

Mistral AI Tiny 经济型模型, 32K 上下文, 基于 Mistral 7B, 价格最低, 适合简单任务。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Mistral tiny 2024-02-26 32K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 54.2 % 5-shot
HumanEval 45.8 pass@1 —
GSM8K (Grade School Math 8K) 57.2 % 0-shot CoT
MATH 16.0 % 0-shot CoT
BBH (BIG-Bench Hard) 51.3 % 3-shot CoT
GPQA 28.7 % 0-shot
IFEval 51.7 % prompt_strict
ARC 82.4 % challenge
MUSR 41.4 % 0-shot
WinoGrande 66.6 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • MMLU 54.2, weak knowledge reasoning.
  • HumanEval 45.8, coding weak.
  • Proprietary, not self-hostable.

Use Cases

  • General chat and Q&A

References