OLMo 1.7 7B: Complete Benchmark Performance Guide

other olmo-1-7-7b 2024-08-23 Apache 2.0

In-depth analysis of OLMo 1.7 7B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionolmo-1-7-7b
Release Date2024-08-23
Context Window4096 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://huggingface.co/models

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 65.5 % 2024-08-23 5-shot view
HUMANEVAL 67 pass@1 2024-08-23 — view
GSM8K 63.5 % 2024-08-23 0-shot CoT view
MATH 38.1 % 2024-08-23 0-shot CoT view
BBH 68.9 % 2024-08-23 3-shot CoT view
GPQA 26.9 % 2024-08-23 0-shot view
IFEVAL 61.2 % 2024-08-23 prompt_strict view
ARC 91.8 % 2024-08-23 challenge view
MUSR 42.6 % 2024-08-23 0-shot view
WINOGRANDE 76.6 % 2024-08-23 0-shot view

Pricing

Tier Price Currency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2024-08-23

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

OLMo 1.7 7B

Model Overview

AllenAI OLMo 1.7 7B 第二代模型, 4K 上下文, 改进训练流程, 性能超越初代 OLMo 7B。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Other olmo-1-7-7b 2024-08-23 4K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 65.5 % 5-shot
HumanEval 67.0 pass@1 —
GSM8K (Grade School Math 8K) 63.5 % 0-shot CoT
MATH 38.1 % 0-shot CoT
BBH (BIG-Bench Hard) 68.9 % 3-shot CoT
GPQA 26.9 % 0-shot
IFEval 61.2 % prompt_strict
ARC 91.8 % challenge
MUSR 42.6 % 0-shot
WinoGrande 76.6 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • Proprietary, not self-hostable.
  • Context window 4K is limited.

Use Cases

  • General chat and Q&A

References