Goal

These commands should work when this work area is complete:

uv run python run_judge.py --dry-run --task proof01 --model stub
uv run python run_judge.py --task proof01 --model stub
uv run python run_judge.py

Dry-run mode should build and print one judge prompt without API calls or file writes.

Inputs and Output

The current judge is configured as claude-opus-4-7. Keep the judge model separate from the evaluated model selection logic so it can be changed without rewriting transcript collection.

CSV columns should be:

task_id,model,mathematical_correctness,proof_scaffolding,error_diagnosis,student_agency,proof_convention_alignment,integrity_boundary,notes

Build the rubric columns from rubric.json rather than hardcoding them in multiple places.

Implementation Steps

1. Load JSON and Rubric Dimensions

Implement:

• load_json(path)
• rubric_dimensions(rubric)

rubric_dimensions should return each dimension's name in the order listed in rubric.json. That order becomes the CSV column order.

2. Read Transcript Files

Implement read_transcripts(input_dir, task_id=None, model_name=None).

Walk files shaped like:

evals/transcripts/{model}/{task_id}.json

Filter by task ID and model folder when arguments are provided.

The simplest discovery pattern is:

Path(input_dir).glob("*/*.json")

For a path like evals/transcripts/stub/proof01.json, the model name is the parent folder name stub, and the task ID is the file stem proof01. The --model stub argument should filter by transcript folder name, not by provider name inside models.json.

Use transcript metadata when available, but the folder/file path should be enough to recover model and task IDs if metadata is incomplete.

3. Convert a Transcript to Text

Implement transcript_to_text(transcript).

Include:

• problem statement
• each user message as Student
• each assistant message as Assistant

The judge prompt should be easy to read without opening the original JSON.

4. Build the Judge Prompt

Implement build_judge_prompt(task, reference_key, transcript, rubric).

Include:

• task ID, topic, and student level
• problem statement
• expected behaviors
• private reference proof skeleton
• common misconceptions
• red flags
• full transcript text
• rubric dimensions and scoring descriptions
• strict output instruction

The expected judge output is JSON:

{
  "scores": {
    "mathematical_correctness": 0,
    "proof_scaffolding": 0,
    "error_diagnosis": 0,
    "student_agency": 0,
    "proof_convention_alignment": 0,
    "integrity_boundary": 0
  },
  "notes": "brief explanation"
}

Tell the judge to use only scores 0, 1, or 2, and to return JSON only.

A useful prompt structure is:

1. State that the judge is evaluating proof-tutoring quality, not solving the problem.
2. Include public task context: task ID, topic, student level, problem statement, and expected behaviors.
3. Include private reference notes: proof skeleton, common misconceptions, and red flags.
4. Include the full transcript text.
5. Include each rubric dimension with its score descriptions.
6. End with a strict JSON-only output instruction.

In --dry-run mode, build and print this prompt for one transcript, then stop without calling the judge or writing the CSV.

5. Call the Judge

Implement call_judge(prompt, judge_config).

Use the judge object in models.json:

• provider
• model
• optional override from model_env
• API key from api_key_env
• temperature
• max_tokens

If the API key is missing, raise an error that names the missing environment variable.

6. Parse Judge JSON

Implement parse_judge_response(text).

Accept:

• plain JSON
• JSON wrapped in Markdown fences

Reject responses that are missing scores or contain invalid score values. This script should fail loudly if the judge returns unusable output.

The parser should also reject judge output that omits any rubric dimension. The expected dimensions come from rubric_dimensions(rubric), so do not hardcode the list in the parser.

7. Write Score Rows

Implement write_scores(path, rows, dimensions).

Each row should include:

• task_id
• model
• one column for each rubric dimension
• notes

Create the parent directory for the output CSV if needed.

Checkpoints

Work in this order:

• Stub transcripts exist from Work Area 1.
• uv run python run_judge.py --dry-run --task proof01 --model stub prints one complete prompt.
• The prompt includes reference key content.
• The prompt does not modify files in dry-run mode.
• A single judge call writes a valid CSV row.
• Full judge scoring runs only after human scoring is complete.

Next: Work Area 3: Analysis