Microsoft WizardCoder Python 34B: Complete Benchmark Performance Guide

other wizardcoder-python-34b 2023-08-17 Llama 2 Community License

In-depth analysis of Microsoft WizardCoder Python 34B performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorother
Versionwizardcoder-python-34b
Release Date2023-08-17
Context Window16384 tokens
Input Modalitiestext
Output Modalitiestext
LicenseLlama 2 Community License
Documentationhttps://huggingface.co/models

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 60 % 2023-08-17 5-shot view
HUMANEVAL 79.8 pass@1 2023-08-17 — view
GSM8K 46.7 % 2023-08-17 0-shot CoT view
MATH 40.6 % 2023-08-17 0-shot CoT view
BBH 67.2 % 2023-08-17 3-shot CoT view
GPQA 36.6 % 2023-08-17 0-shot view
IFEVAL 53.8 % 2023-08-17 prompt_strict view
ARC 91.2 % 2023-08-17 challenge view
MUSR 53.1 % 2023-08-17 0-shot view
WINOGRANDE 70.4 % 2023-08-17 0-shot view

Pricing

Tier Price Currency
Input$0.5 / MtokUSD
Output$0.5 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://huggingface.co/models · as of 2023-08-17

Compliance

  • Data Residency: self-host
  • SOC2: ✗
  • HIPAA: ✗
  • GDPR: ✗
  • ISO 27001: ✗

Microsoft WizardCoder Python 34B

Model Overview

Microsoft WizardCoder Python 34B 代码模型, 16K 上下文, 基于 Code Llama 34B, Python 代码生成突出。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Other wizardcoder-python-34b 2023-08-17 16K text text Llama 2 Community License

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 60.0 % 5-shot
HumanEval 79.8 pass@1 —
GSM8K (Grade School Math 8K) 46.7 % 0-shot CoT
MATH 40.6 % 0-shot CoT
BBH (BIG-Bench Hard) 67.2 % 3-shot CoT
GPQA 36.6 % 0-shot
IFEval 53.8 % prompt_strict
ARC 91.2 % challenge
MUSR 53.1 % 0-shot
WinoGrande 70.4 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • Proprietary, not self-hostable.
  • Context window 16K is limited.

Use Cases

  • Code generation and debugging

References