Chat Bison: Complete Benchmark Performance Guide

google chat-bison 2023-05-10 Proprietary

In-depth analysis of Chat Bison performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorgoogle
Versionchat-bison
Release Date2023-05-10
Context Window8192 tokens
Input Modalitiestext
Output Modalitiestext
LicenseProprietary
Documentationhttps://ai.google.dev/gemini-api/docs

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 63.1 % 2023-05-10 5-shot view
HUMANEVAL 36 pass@1 2023-05-10 — view
GSM8K 52.4 % 2023-05-10 0-shot CoT view
MATH 19.8 % 2023-05-10 0-shot CoT view
BBH 54.6 % 2023-05-10 3-shot CoT view
GPQA 25 % 2023-05-10 0-shot view
IFEVAL 48.2 % 2023-05-10 prompt_strict view
ARC 82.6 % 2023-05-10 challenge view
MUSR 32.8 % 2023-05-10 0-shot view
WINOGRANDE 66.2 % 2023-05-10 0-shot view

Pricing

Tier Price Currency
Input$0.5 / MtokUSD
Output$1 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.google.dev/pricing · as of 2023-05-10

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Chat Bison

Model Overview

Google Vertex AI Chat Bison 对话模型, 基于 PaLM 2, 8K 上下文, 适合企业级对话应用。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Google chat-bison 2023-05-10 8K text text Proprietary

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 63.1 % 5-shot
HumanEval 36.0 pass@1 —
GSM8K (Grade School Math 8K) 52.4 % 0-shot CoT
MATH 19.8 % 0-shot CoT
BBH (BIG-Bench Hard) 54.6 % 3-shot CoT
GPQA 25.0 % 0-shot
IFEval 48.2 % prompt_strict
ARC 82.6 % challenge
MUSR 32.8 % 0-shot
WinoGrande 66.2 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • HumanEval 36.0, coding weak.
  • Proprietary, not self-hostable.
  • Context window 8K is limited.

Use Cases

  • General chat and Q&A

References