Goal

These commands should work when this work area is complete:

uv run python analyze.py --human examples/fixture_human_scores.csv --judge examples/fixture_judge_scores.csv
uv run python analyze.py --human examples/fixture_human_scores.csv --judge examples/fixture_judge_scores.csv --output analysis.md
uv run python analyze.py

Start with the fixture files because they are small and deterministic.

Inputs and Output

By default, the script should print a Markdown report. If --output is provided, write that same report to the given file.

Implementation Steps

1. Load Rubric Dimensions

Implement:

• load_json(path)
• rubric_dimensions(rubric_path)

Return dimension names from rubric.json in order. The fixture and real CSVs use these names as columns.

2. Read Score CSVs

Implement read_scores(path, dimensions).

Return rows shaped like:

{
    "task_id": "proof01",
    "model": "stub",
    "notes": "Good targeted scaffolding.",
    "scores": {
        "mathematical_correctness": 2,
        "proof_scaffolding": 2,
        "error_diagnosis": 2,
        "student_agency": 2,
        "proof_convention_alignment": 2,
        "integrity_boundary": 2,
    },
}

Parse scores as integers. If a score cell is blank, store None so partial files can still be analyzed.

Use csv.DictReader so the column names come from the CSV header. Required non-score fields are task_id, model, and notes; the score fields should match the dimension names from rubric.json. Strip whitespace from cells before parsing. A blank score cell becomes None; a nonblank score should become an integer.

3. Average by Model and Task

Implement average_by(rows, key_name, dimensions).

Use it for:

• model averages with key_name="model"
• task averages with key_name="task_id"

Average all available dimension scores for each group. Ignore None values.

For each group, collect every non-None score across every row and every rubric dimension, then return one overall average for that group. Round only when rendering the report, not while computing.

4. Average by Dimension

Implement dimension_averages(rows, dimensions).

This should return one average per rubric dimension. It helps answer questions like:

• Which tutoring quality was strongest overall?
• Which rubric dimension was hardest for models?
• Which dimension did the judge score differently from the human scorer?

5. Compare Human and Judge Scores

Implement judge_agreement(human_rows, judge_rows, dimensions).

Match rows by:

(task_id, model)

For each matched row and each rubric dimension, compare the human score with the judge score.

Each (task_id, model, dimension) comparison counts separately. Skip comparisons where either the human score or judge score is None.

Report:

• total compared scores
• exact agreement count and percent
• within-one-point agreement count and percent

For the fixture files, the expected agreement is:

16/18 exact
18/18 within one point

6. Render Markdown Tables

Implement markdown_table(headers, rows).

Use it inside build_report(...) to print tables for:

• human averages by model
• human averages by task
• human averages by rubric dimension
• judge averages by rubric dimension
• human/judge agreement

Round averages to two decimal places.

Build the report in that order so the fixture output is easy to inspect before using real scores. The fixture files should be the first test target; they are small enough that incorrect grouping, missing dimensions, or agreement-count bugs show up quickly.

Checkpoints

Work in this order:

• Fixture score files load without errors.
• The fixture report prints Markdown tables.
• Human/judge agreement on fixtures is 16/18 exact and 18/18 within one.
• --output analysis.md writes the same report to disk.
• The default command uses evals/scores/human_scores.csv and evals/scores/judge_scores.csv.

Next: Research Workflow and Follow-Up