Argilla Notus 7B v1: Complete Benchmark Performance Guide

other notus-7b-v1 2023-12-15 Apache 2.0

In-depth analysis of Argilla Notus 7B v1 performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionnotus-7b-v1
Release Date2023-12-15
Context Window4096 tokens
Input Modalitiestext
Output Modalitiestext
LicenseApache 2.0
Documentationhttps://huggingface.co/models

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 71.4 % 2023-12-15 5-shot view
HUMANEVAL 36.8 pass@1 2023-12-15 — view
GSM8K 58.8 % 2023-12-15 0-shot CoT view
MATH 31 % 2023-12-15 0-shot CoT view
BBH 62.4 % 2023-12-15 3-shot CoT view
GPQA 32.7 % 2023-12-15 0-shot view
IFEVAL 42.8 % 2023-12-15 prompt_strict view
ARC 89.8 % 2023-12-15 challenge view
MUSR 30.1 % 2023-12-15 0-shot view
WINOGRANDE 69.8 % 2023-12-15 0-shot view

Pricing

Tier Price Currency
Input$0.18 / MtokUSD
Output$0.18 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2023-12-15

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Argilla Notus 7B v1

Model Overview

Argilla Notus 7B v1 对话微调模型, 4K 上下文, 基于 Zephyr 改进, 通过高质量反馈数据提升性能。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Other notus-7b-v1 2023-12-15 4K text text Apache 2.0

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 71.4 % 5-shot
HumanEval 36.8 pass@1 —
GSM8K (Grade School Math 8K) 58.8 % 0-shot CoT
MATH 31.0 % 0-shot CoT
BBH (BIG-Bench Hard) 62.4 % 3-shot CoT
GPQA 32.7 % 0-shot
IFEval 42.8 % prompt_strict
ARC 89.8 % challenge
MUSR 30.1 % 0-shot
WinoGrande 69.8 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • HumanEval 36.8, coding weak.
  • Proprietary, not self-hostable.
  • Context window 4K is limited.

Use Cases

  • General chat and Q&A

References