MPT 7B: Complete Benchmark Performance Guide

other mpt-7b 2023-05-04 Apache 2.0

In-depth analysis of MPT 7B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionmpt-7b
Release Date2023-05-04
Context Window2048 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://huggingface.co/models

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 51.7 % 2023-05-04 5-shot view
HUMANEVAL 38.1 pass@1 2023-05-04 — view
GSM8K 44.6 % 2023-05-04 0-shot CoT view
MATH 18.4 % 2023-05-04 0-shot CoT view
BBH 59.9 % 2023-05-04 3-shot CoT view
GPQA 34.2 % 2023-05-04 0-shot view
IFEVAL 59.9 % 2023-05-04 prompt_strict view
ARC 87.5 % 2023-05-04 challenge view
MUSR 42.4 % 2023-05-04 0-shot view
WINOGRANDE 65.5 % 2023-05-04 0-shot view

Pricing

Tier Price Currency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2023-05-04

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

MPT 7B

Model Overview

MosaicML MPT 7B 经济型开源模型, 2K 上下文, 7B 参数, Apache 2.0 可商用, 适合本地部署。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Other mpt-7b 2023-05-04 2K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 51.7 % 5-shot
HumanEval 38.1 pass@1 —
GSM8K (Grade School Math 8K) 44.6 % 0-shot CoT
MATH 18.4 % 0-shot CoT
BBH (BIG-Bench Hard) 59.9 % 3-shot CoT
GPQA 34.2 % 0-shot
IFEval 59.9 % prompt_strict
ARC 87.5 % challenge
MUSR 42.4 % 0-shot
WinoGrande 65.5 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • MMLU 51.7, weak knowledge reasoning.
  • HumanEval 38.1, coding weak.
  • Proprietary, not self-hostable.
  • Context window 2K is limited.

Use Cases

  • General chat and Q&A

References