Text Bison: Complete Benchmark Performance Guide

google text-bison 2023-05-10 Proprietary

In-depth analysis of Text Bison performance on MMLU, HumanEval, GSM8K, MATH, BBH benchmarks, including pricing and use cases.

Key Specifications

Vendorgoogle
Versiontext-bison
Release Date2023-05-10
Context Window8192 tokens
Input Modalitiestext
Output Modalitiestext
LicenseProprietary
Documentationhttps://ai.google.dev/gemini-api/docs

Benchmark Performance

Benchmark Score Unit Evaluated At Notes Source
MMLU 53.9 % 2023-05-10 5-shot view
HUMANEVAL 45.1 pass@1 2023-05-10 — view
GSM8K 42.8 % 2023-05-10 0-shot CoT view
MATH 12.1 % 2023-05-10 0-shot CoT view
BBH 48.2 % 2023-05-10 3-shot CoT view
GPQA 27.7 % 2023-05-10 0-shot view
IFEVAL 59.5 % 2023-05-10 prompt_strict view
ARC 86.7 % 2023-05-10 challenge view
MUSR 40.8 % 2023-05-10 0-shot view
WINOGRANDE 74.1 % 2023-05-10 0-shot view

Pricing

Tier Price Currency
Input$0.5 / MtokUSD
Output$1 / MtokUSD
Cache Read$0 / MtokUSD
Cache Write$0 / MtokUSD

Source: https://ai.google.dev/pricing · as of 2023-05-10

Compliance

  • Data Residency: US
  • SOC2: ✓
  • HIPAA: ✗
  • GDPR: ✓
  • ISO 27001: ✓

Text Bison

Model Overview

Google Vertex AI Text Bison 文本生成模型, 基于 PaLM 2, 8K 上下文, 适合指令跟随与生成任务。

Core Specifications

Vendor Version Release Date Context Window Input Modalities Output Modalities License
Google text-bison 2023-05-10 8K text text Proprietary

Benchmark Performance

Benchmark Score Unit Notes
MMLU (Massive Multitask Language Understanding) 53.9 % 5-shot
HumanEval 45.1 pass@1 —
GSM8K (Grade School Math 8K) 42.8 % 0-shot CoT
MATH 12.1 % 0-shot CoT
BBH (BIG-Bench Hard) 48.2 % 3-shot CoT
GPQA 27.7 % 0-shot
IFEval 59.5 % prompt_strict
ARC 86.7 % challenge
MUSR 40.8 % 0-shot
WinoGrande 74.1 % 0-shot

Pricing

Input Output Cache Read Cache Write
— — — —

per million tokens

Strengths

  • Reliable general-purpose model.

Weaknesses

  • MMLU 53.9, weak knowledge reasoning.
  • HumanEval 45.1, coding weak.
  • Proprietary, not self-hostable.
  • Context window 8K is limited.

Use Cases

  • General chat and Q&A

References