Automatic Discovery of Model-Specific Agent Harness Profiles

An early experimental framework for measuring how a shared agent harness affects model behavior. It provides execution and measurement infrastructure; the central harness-fit hypothesis remains to be tested.

GitHub → Quickstart

Features

What HarnessFit provides for agent harness research.

Hidden-Test Evaluation

Six split benchmark tasks are evaluated with hidden acceptance tests, regression checks, and deterministic scoring.

Multi-Provider Runtime

One normalized runtime executes OpenAI, Anthropic, and Gemini models under the same generic harness.

Parameterized Harness

A machine-readable harness has 37 typed settings spanning prompts, tools, context, validation, retries, and completion.

Held-Out Evaluation

Evaluate a supplied harness JSON on the configured test split with persisted success, score, cost, and latency data.

Persisted Run Evidence

SQLite stores run metadata, events, patches, deterministic score components, costs, and durations; harness configs are hashed.

Research Roadmap

Optimization, transfer matrices, stability analysis, and statistical acceptance remain planned work—not current results.

Architecture

Provider-agnostic pipeline from experiment definition to results.

Architecture pipeline diagram

By the Numbers

Current project state.

101 passing tests
7 live-provider checks are opt-in
0 type errors
Full strict TypeScript with branded types
6 benchmark tasks
3 train · 2 dev · 1 test
3 providers
OpenAI · Anthropic · Google Gemini adapters

Quickstart

Get started in under a minute.

# Clone and install
git clone https://github.com/rmax-ai/harness-fit.git
cd harness-fit

# Install dependencies (requires Bun)
bun install

# Run tests
bun test packages/ apps/ benchmarks/repositories/

# Type check
bun run typecheck

# Initialize and validate provider credentials
bun harnessfit init
bun harnessfit providers check

# Run the generic-harness baseline (requires provider keys)
bun harnessfit baseline --experiment experiments/definitions/default.yaml

API Keys

Set OPENAI_API_KEY, ANTHROPIC_API_KEY, and GOOGLE_API_KEY in your environment. Copy .env.example to get started. Provider model IDs can be overridden with HARNESSFIT_*_MODEL variables when an account does not expose the defaults.