Goal

These commands should work when this work area is complete:

uv run python run_conversations.py --dry-run --task proof01
uv run python run_conversations.py --stub
uv run python run_conversations.py --stub --task proof05
uv run python run_conversations.py --model openai-frontier --task proof01

Dry-run mode should not call APIs and should not write transcript files. Stub mode should write deterministic transcript files so you can test the rest of the pipeline without spending API calls.

Inputs

The runner uses these files:

File	Purpose
`tasks.json`	The 10 scripted proof-help conversations.
`reference_keys.json`	Private scoring notes. Use only for validation here; do not send to evaluated models.
`models.json`	Provider/model configuration, model tiers, and generation settings.

The current model mix includes frontier or balanced models from OpenAI, Anthropic, and Google, plus lower-cost models from each provider. Running the full suite produces 60 transcripts: 10 tasks across 6 models.

Each task must have:

id
topic
student_level
problem_statement
student_turns
expected_behaviors
reference_key_id

Each task should have exactly three student turns.

Implementation Steps

1. Load JSON

Implement load_json(path) with UTF-8 file reading and json.load.

Use this helper everywhere the script reads project data. If JSON is invalid or missing, let Python show a clear file/JSON error.

2. Validate Tasks

Implement validate_tasks(tasks, reference_keys) before any model calls.

Check:

every required task field exists
task IDs are unique
each task has exactly three student turns
each reference_key_id exists in reference_keys.json

Raise ValueError with a useful message when validation fails.

3. Select Tasks

Implement select_tasks(tasks, task_id).

If task_id is missing, return all tasks.
If it is present, return only that task.
If no task matches, raise an error that names the missing task ID.

4. Select Models

Implement select_models(models_config, requested_models, use_stub).

Rules:

If --stub is used, return one synthetic model entry named stub.
Otherwise, start from enabled entries in models_config["models"].
If --model is provided, allow matching by model name, provider, or concrete model ID.
If a requested model does not match anything, raise a useful error.

Stub mode should still return the same kind of collection as the real-model path. In other words, return a list containing one dictionary, not just the dictionary itself:

if use_stub:
    return [{
        "name": "stub",
        "provider": "stub",
        "model": "stub",
        "enabled": True,
    }]

The stub entry does not need API key fields, model environment fields, or tier metadata unless later code explicitly depends on them. The important requirement is that downstream code can iterate over selected models and read model["name"], model["provider"], and model["model"].

5. Build Student Messages

Implement build_student_message(task, turn_index).

For turn 0, include the problem statement and the first scripted student message. Later turns should include only the scripted student message.

Keep the message readable. A simple structure is enough:

Problem:
[problem statement]

Student:
[student turn]

Do not include expected_behaviors or reference_keys.json content in student messages.

6. Add Stub Responses

Implement call_stub_model(task, turn_index).

The stub should be deterministic and obviously fake. It should mention the task ID or topic, acknowledge the turn, and model the kind of scaffolded help expected from a tutor.

It does not need to be mathematically perfect. Its job is to let you test file writing and downstream scoring flow without APIs.

Do not handwrite unique responses for every possible task or student message. A simple three-response pattern keyed by turn_index is enough. Use task["id"] and task["topic"] in the response so generated transcripts are easy to inspect:

def call_stub_model(task, turn_index: int) -> str:
    task_id = task["id"]
    topic = task["topic"]

    responses = [
        (
            f"[stub response for {task_id}, turn 1] This is a {topic} problem. "
            "Start by unpacking the relevant definition and writing down what "
            "the assumptions give you."
        ),
        (
            f"[stub response for {task_id}, turn 2] Check the specific claim "
            "you just made. Try to justify it using the definition rather than "
            "relying on a broad shortcut."
        ),
        (
            f"[stub response for {task_id}, turn 3] I should not write the final "
            "proof for you, but here is a small template for your next step: "
            "state the assumption, apply the definition, and then finish the "
            "conclusion in your own words."
        ),
    ]

    return responses[turn_index]

The purpose of this function is to exercise the conversation loop, transcript saving, and downstream scoring workflow. Stub responses are not part of the actual model evaluation.

7. Run One Conversation

Implement run_task_conversation(...).

Expected transcript shape:

{
  "metadata": {
    "task_id": "proof01",
    "model": "stub",
    "provider": "stub",
    "dry_run": false,
    "system_prompt": true
  },
  "problem_statement": "...",
  "messages": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."}
  ]
}

For each of the three turns:

Build the student message.
Append it to the running message list as {"role": "user", "content": student_message}.
Choose exactly one assistant response path:

In dry-run mode, create a plain preview message without calling a model.
In stub mode, call call_stub_model.
Otherwise, call call_real_model.

Append the assistant response as {"role": "assistant", "content": assistant_message}.

Dry-run mode is not another tutor stub. It is only a preview of transcript construction. The dry-run assistant response can be a simple placeholder like:

f"[dry-run preview for {task['id']}, turn {turn_index + 1}: assistant response would be generated here]"

This placeholder should be separate from call_stub_model and does not need to be mathematically helpful or tutor-like. Its job is to make the printed preview easy to inspect while avoiding API calls and file writes.

Common pitfalls to avoid in run_task_conversation:

Do not call call_stub_model in dry-run mode.
Do not call save_transcript from inside run_task_conversation.
Do not append raw strings directly to messages; always append message dictionaries with role and content.
Assistant messages should use role "assistant", not "user".
Use the model entry's display name consistently for transcript metadata and output folders.
Do not include reference_keys.json content in conversation messages.

Look ahead to main() orchestration:

Parse CLI arguments.
Load tasks.json, reference_keys.json, and models.json.
Validate tasks before running conversations.
Select requested tasks and models.
Loop over each selected model and each selected task.
Run each task/model conversation.
If --dry-run is active, print a readable preview and skip saving.
Otherwise, save JSON and Markdown transcripts.

8. Save Transcript Files

Implement save_transcript(transcript, output_dir).

Write both formats:

evals/transcripts/{model}/{task_id}.json
evals/transcripts/{model}/{task_id}.md

Use the transcript metadata to choose the model folder and task file name:

model folder: evals/transcripts/{model_name}/
JSON path: evals/transcripts/{model_name}/{task_id}.json
Markdown path: evals/transcripts/{model_name}/{task_id}.md

The JSON file should preserve the transcript object directly:

json.dump(transcript, file, indent=2)

The Markdown file should include:

task ID
model name
provider
dry-run/system-prompt metadata when useful
problem statement
alternating Student / Assistant turns

Create missing directories with Path.mkdir(parents=True, exist_ok=True).

Do not call save_transcript for dry-run previews. Dry-run output should be printed for inspection only.

9. Main Orchestration

Implement main() after run_task_conversation and save_transcript work.

The shape should be simple:

args = parse_args()
Load tasks.json, reference_keys.json, and models.json.
Validate tasks with validate_tasks.
Select tasks using select_tasks(tasks, args.task).
Select models using select_models(models_config, args.model, args.stub).
For each selected model and task, call run_task_conversation.
If args.dry_run is true, print a readable preview and do not save.
Otherwise, call save_transcript.

Keep real provider calls last. The dry-run and stub paths should work without API keys, and stub transcript generation should not depend on call_real_model.

10. Add Real Provider Calls Last

Implement call_real_model(...) only after dry-run and stub paths work.

Before making real API calls, update the real-model branch in run_task_conversation. It should call call_real_model(...); it should not use pass, and it should not save empty assistant messages. If real provider calls are not implemented yet, raise NotImplementedError instead of silently writing bad transcripts.

Use models.json and .env:

provider: openai, anthropic, or google
default model ID from models.json
optional override from model_env
API key from api_key_env
temperature and max_tokens from generation config
model-specific settings such as temperature when present

If an API key is missing, raise an error that names the missing environment variable.

When calling a model, merge settings in this order: global generation defaults first, then any model-specific overrides. This matters for Google preview models, whose docs recommend using their default temperature behavior.

Recommended implementation shape:

Import os and load_dotenv from dotenv.
Call load_dotenv() before reading API keys, either in main() or inside call_real_model.

Resolve the concrete model ID:

model_id = os.getenv(model.get("model_env", "")) or model["model"]

Resolve the API key:

api_key = os.getenv(model["api_key_env"])

If the API key is missing, raise ValueError(f"Missing API key: {model['api_key_env']}").
Build the provider request using only the conversation messages and, if enabled, SYSTEM_PROMPT.
Return the assistant text only.
If the provider returns empty text, raise an error instead of saving the transcript.

Provider notes:

OpenAI: use the Responses API through OpenAI().responses.create(...). Pass SYSTEM_PROMPT as instructions when use_system_prompt is true, pass the conversation as input, set model=model_id, and use max_output_tokens from the merged generation config. See the OpenAI text generation docs.
Anthropic: use Anthropic().messages.create(...). Pass SYSTEM_PROMPT through the top-level system argument when enabled, pass the conversation history through messages, and set max_tokens. Claude messages are stateless, so send the full conversation history each turn. See the Anthropic Messages API docs.
Google: use google.genai.Client().models.generate_content(...). Pass the system prompt through types.GenerateContentConfig(system_instruction=...) when enabled. The simplest implementation may convert the current conversation history to readable text for contents. See the Gemini text generation docs.

Do not send reference_keys.json, rubric scores, common misconceptions, or red flags to evaluated tutor models. Those materials are only for human scoring and the judge scorer.