OLMo 7B SFT: Complete Benchmark Performance Guide

other olmo-7b-sft 2024-02-01 Apache 2.0

In-depth analysis of OLMo 7B SFT performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionolmo-7b-sft
Release Date2024-02-01
Context Window2048 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://huggingface.co/models

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 73.4 % 2024-02-01 5-shot view
HUMANEVAL 53.9 pass@1 2024-02-01 — view
GSM8K 62.8 % 2024-02-01 0-shot CoT view
MATH 43.1 % 2024-02-01 0-shot CoT view
BBH 64 % 2024-02-01 3-shot CoT view
GPQA 31.8 % 2024-02-01 0-shot view
IFEVAL 56.2 % 2024-02-01 prompt_strict view
ARC 89.3 % 2024-02-01 challenge view
MUSR 44.3 % 2024-02-01 0-shot view
WINOGRANDE 78.8 % 2024-02-01 0-shot view

Pricing

Tier Price Currency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2024-02-01

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

OLMo 7B SFT

Model Overview

AllenAI OLMo 7B SFT 监督微调版本, 2K 上下文, 基于 Tulu 数据集微调, 适合通用对话任务。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Other olmo-7b-sft 2024-02-01 2K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 73.4 % 5-shot
HumanEval 53.9 pass@1 —
GSM8K (Grade School Math 8K) 62.8 % 0-shot CoT
MATH 43.1 % 0-shot CoT
BBH (BIG-Bench Hard) 64.0 % 3-shot CoT
GPQA 31.8 % 0-shot
IFEval 56.2 % prompt_strict
ARC 89.3 % challenge
MUSR 44.3 % 0-shot
WinoGrande 78.8 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • Proprietary, not self-hostable.
  • Context window 2K is limited.

Use Cases

  • General chat and Q&A

References