Model Call Parameters

Use LiteLLM for the provider call. Add one small helper only to choose the model prefix, API key, token-budget field, and sampling params that are safe to send for the configured model.

Do not add provider branches inside every call site.

1. Add One Helper

Put this near the real model-call functions.

PROVIDER_PREFIXES = {
    "openai": "openai",
    "anthropic": "anthropic",
    "google": "gemini",
}


def resolve_litellm_model(model_config):
    provider = model_config["provider"]
    prefix = PROVIDER_PREFIXES.get(provider)
    if prefix is None:
        raise ValueError(f"Unknown provider: {provider}")

    model_id = os.getenv(model_config.get("model_env", "")) or model_config["model"]
    return f"{prefix}/{model_id}"


def build_completion_kwargs(model_config, messages, generation_config=None):
    generation_config = generation_config or {}
    provider = model_config["provider"]
    model_name = resolve_litellm_model(model_config)

    api_key_env = model_config["api_key_env"]
    api_key = os.getenv(api_key_env)
    if not api_key:
        raise ValueError(f"Missing API key: {api_key_env}")

    kwargs = {
        "model": model_name,
        "messages": messages,
        "api_key": api_key,
    }

    if provider == "openai":
        kwargs["max_completion_tokens"] = model_config.get(
            "max_completion_tokens",
            generation_config.get("max_completion_tokens", 4096),
        )
        if not model_name.startswith("openai/gpt-5"):
            kwargs["temperature"] = model_config.get(
                "temperature",
                generation_config.get("temperature", 0),
            )
        return kwargs

    kwargs["max_tokens"] = model_config.get(
        "max_tokens",
        generation_config.get("max_tokens", 1200),
    )

    if provider == "anthropic":
        if not model_name.endswith("claude-opus-4-7"):
            kwargs["temperature"] = model_config.get(
                "temperature",
                generation_config.get("temperature", 0),
            )
        return kwargs

    if provider == "google":
        if "temperature" in model_config or "temperature" in generation_config:
            kwargs["temperature"] = model_config.get(
                "temperature",
                generation_config.get("temperature", 0),
            )
        return kwargs

    raise ValueError(f"Unknown provider: {provider}")

Why this is enough:

  • LiteLLM still handles the provider request.
  • OpenAI GPT-5.x gets max_completion_tokens and no forced temperature.
  • Anthropic and Gemini keep max_tokens.
  • The Opus judge does not receive unsupported sampling params.

2. Use It In The Tutor Call

kwargs = build_completion_kwargs(model_config, messages, models["generation"])
response = litellm.completion(**kwargs)

assistant_text = response.choices[0].message.content
if not assistant_text:
    raise ValueError("Model returned no assistant text.")

3. Use It In The Judge Call

messages = [{"role": "user", "content": judge_prompt}]
kwargs = build_completion_kwargs(judge_config, messages, models["generation"])
response = litellm.completion(**kwargs)

judge_text = response.choices[0].message.content
if not judge_text:
    raise ValueError("Judge returned no text.")

4. Keep Config Simple

The shared generation config can stay small:

{
  "generation": {
    "temperature": 0,
    "max_tokens": 1200,
    "max_completion_tokens": 4096
  }
}

If OpenAI cuts off outputs, raise only the OpenAI completion budget:

{
  "name": "openai-frontier",
  "provider": "openai",
  "model": "gpt-5.5",
  "api_key_env": "OPENAI_API_KEY",
  "max_completion_tokens": 6000,
  "enabled": true
}

5. Verify Before Full Evals

uv run python -m py_compile run_conversations.py run_judge.py analyze.py
uv run python run_conversations.py --dry-run --task proof01 --model openai-frontier
uv run python run_conversations.py --stub --task proof01 --output-dir /tmp/proof-assistance-evals-smoke

Then run one real call per provider family:

uv run python run_conversations.py --task proof01 --model openai-frontier --output-dir /tmp/proof-assistance-evals-smoke
uv run python run_conversations.py --task proof01 --model anthropic-balanced --output-dir /tmp/proof-assistance-evals-smoke
uv run python run_conversations.py --task proof01 --model google-economy-preview --output-dir /tmp/proof-assistance-evals-smoke

For the final study, do not assume exact determinism. Fix prompts, model versions, request kwargs, and token budgets; log all metadata; and use repeated runs when comparing models seriously.