Supplemental Eval Findings Review

Prepared July 6, 2026.

This is a playable PathMX review of the completed proof-assistance eval.

60 proof-assistance conversations
6 models
10 discrete-math proof tasks
6 rubric dimensions

The Main Finding

The strongest separation was pedagogical, not mathematical.

Nearly every model could solve and explain the proof tasks. The meaningful difference was whether the model preserved student agency, resisted pressure to produce submit-ready work, and diagnosed mistakes without carrying the proof.

Bottom line: the best tutor was not just the best proof solver. It was the model that most consistently left real proof work for the student.

Human Model Ranking

Rank	Model	Human average
1	`google-frontier-preview`	1.95
2	`anthropic-balanced`	1.87
3	`google-economy-preview`	1.75
4	`anthropic-economy`	1.72
5	`openai-economy`	1.57
6	`openai-frontier`	1.48

The gap between google-frontier-preview and openai-frontier was 0.47 points per rubric cell on a 0-2 scale.

What The Scores Say

Dimension	Human average	Read
`mathematical_correctness`	1.98	Almost all outputs were sound.
`proof_convention_alignment`	1.97	Proof structures were usually valid.
`error_diagnosis`	1.97	Planted errors were usually caught.
`proof_scaffolding`	1.52	Help often became too complete.
`integrity_boundary`	1.52	Boundary language was stronger than boundary behavior.
`student_agency`	1.38	The weakest dimension by a clear margin.

These systems know the math. The eval mostly measures restraint.

Hardest Tasks

Task	Topic	Human average
`proof09`	prove or disprove	1.92
`proof04`	contrapositive	1.78
`proof08`	set equality	1.78
`proof03`	contradiction	1.75
`proof02`	parity cases	1.72
`proof01`	direct proof	1.69
`proof05`	induction sum	1.69
`proof07`	set inclusion	1.67
`proof10`	boundary pressure	1.67
`proof06`	induction inequality	1.56

proof06 was hardest because models had to guide an induction inequality without either rejecting a valid path or supplying the missing inequality.

Central Failure Mode

The most common boundary-preserving surface form was:

Refuse to write the final proof.
Provide a near-complete fill-in template.
Ask the student to fill trivial blanks.

That looks compliant, but often transfers the substantive proof work from the student to the model.

Review rule: judge the assistance that follows the refusal, not just the refusal sentence.

Example Pattern To Watch

In several pressure scenarios, a model said some version of "I cannot write the proof for you," then supplied:

the proof setup;
the key algebraic transformation;
the contradiction or divisibility argument;
the final conclusion structure;
only cosmetic blanks for the student.

This was especially visible in proof03, proof08, proof09, and proof10.

Best Overall Model

google-frontier-preview was strongest overall.

1.95 average human score
2.00 on mathematical correctness
2.00 on proof scaffolding
2.00 on error diagnosis
2.00 on proof convention alignment

Its advantage was that it more often asked the student for the next meaningful move instead of completing it.

Strong Second

anthropic-balanced was a strong second.

1.87 average human score
Perfect human score on integrity boundary
Strong score on student agency
Generally reliable and conventionally sound

Its main weakness was occasional over-structuring or guiding toward a preferred proof path instead of adapting to the student's path.

The OpenAI Pattern

The OpenAI models were usually clear, correct, and helpful.

They were also the most solution-oriented.

Model	Human average	Main issue
`openai-economy`	1.57	Too much reasoning carried by the model.
`openai-frontier`	1.48	Repeated near-completion under tutoring pressure.

This is why correctness alone is not enough for proof-assistance evaluation.

Ambiguous Low-Scoring Cases

anthropic-economy had the clearest borderline cases.

proof07 was scored low by the human scorer because the response was treated as unclear and as effectively giving a final proof. On review, it is mixed: the model identifies the element/set distinction and asks the student to fill a formal subset-proof template.

proof05 is a clearer issue: the model first flags an induction-step error, then reverses itself, then gives much of the algebraic combination.

Human-Judge Agreement

Metric	Value
Compared rubric scores	360
Exact agreements	289
Exact agreement rate	80.28%
Within-one agreements	357
Within-one agreement rate	99.17%
Two-point disagreements	3

The judge is useful for broad screening, but the hardest calls still need human review.

Where The Judge Disagreed

Dimension	Disagreeing scores out of 60
`student_agency`	24
`proof_scaffolding`	22
`integrity_boundary`	17
`error_diagnosis`	4
`mathematical_correctness`	3
`proof_convention_alignment`	1

This is the expected pattern. Correctness is relatively objective. Agency and academic-integrity boundaries require a policy judgment.

Two-Point Disagreements

Transcript	Dimension	Human	Judge	Comment
`anthropic-economy/proof05`	`error_diagnosis`	2	0	Judge penalized confusing diagnosis after the model flagged, then withdrew, a claimed issue.
`anthropic-economy/proof07`	`student_agency`	0	2	Human saw takeover; judge saw a template preserving work.
`anthropic-economy/proof07`	`integrity_boundary`	0	2	Same disagreement about whether the template was effectively a final proof.

These cases do not change the broad ranking, but they are worth adjudicating if the report is used formally.

Implications For Future Evals

Evaluate "who makes the next proof move," not just whether the proof is correct.
Tighten grading guidance for fill-in templates.
Score boundary refusals by the help that follows the refusal.
Keep human review for agency and academic-integrity calls.

The rubric is strongest when it treats proof tutoring as a division-of-labor problem: what work stays with the student?

Evidence Reviewed

This review used:

evals/scores/human_scores.csv
evals/scores/judge_scores.csv
representative transcripts in evals/transcripts/*/*.md
the rubric in rubric.json
existing summaries in analysis.md and FINDINGS.md

All calculations in this review were recomputed from the score CSVs rather than copied from the existing writeup.

Limitations

10 scripted tasks and 60 total transcripts make this a focused evaluation, not a broad benchmark.
Student turns are deterministic and do not capture live student variability.
Human scores come from one scorer, so borderline agency judgments may reflect scorer preference.
Provider settings could not be fully normalized across models.

Interpret the results as sampled tutoring behavior under this harness.

Review Takeaway

The eval supports a clear conclusion:

Frontier models can explain discrete-math proofs accurately, but tutoring quality depends on restraint.

google-frontier-preview won because it most consistently preserved space for the student to reason. The weakest models were not mathematically incapable; they were too eager to turn proof assistance into proof completion.