MPT 30B: Complete Benchmark Performance Guide

other mpt-30b 2023-06-22 Apache 2.0

In-depth analysis of MPT 30B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionmpt-30b
Release Date2023-06-22
Context Window8192 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://huggingface.co/models

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 55.2 % 2023-06-22 5-shot view
HUMANEVAL 49.7 pass@1 2023-06-22 — view
GSM8K 59.6 % 2023-06-22 0-shot CoT view
MATH 21.2 % 2023-06-22 0-shot CoT view
BBH 49.2 % 2023-06-22 3-shot CoT view
GPQA 24.2 % 2023-06-22 0-shot view
IFEVAL 51.2 % 2023-06-22 prompt_strict view
ARC 80.7 % 2023-06-22 challenge view
MUSR 28.9 % 2023-06-22 0-shot view
WINOGRANDE 77.9 % 2023-06-22 0-shot view

Pricing

Tier Price Currency
Input$0.4 / MtokUSD
Output$0.4 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2023-06-22

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

MPT 30B

Model Overview

MosaicML MPT 30B 开源模型, 8K 上下文, 300 亿参数, 改进长上下文处理, Apache 2.0 可商用。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Other mpt-30b 2023-06-22 8K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 55.2 % 5-shot
HumanEval 49.7 pass@1 —
GSM8K (Grade School Math 8K) 59.6 % 0-shot CoT
MATH 21.2 % 0-shot CoT
BBH (BIG-Bench Hard) 49.2 % 3-shot CoT
GPQA 24.2 % 0-shot
IFEval 51.2 % prompt_strict
ARC 80.7 % challenge
MUSR 28.9 % 0-shot
WinoGrande 77.9 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • MMLU 55.2, weak knowledge reasoning.
  • HumanEval 49.7, coding weak.
  • Proprietary, not self-hostable.
  • Context window 8K is limited.

Use Cases

  • General chat and Q&A

References