Research Workflow and Follow-Up

After the three scripts work, use this guide to run the actual study in the right order.

The order matters. Human scoring comes before automated judge scoring so your judgment is not biased by the LLM judge.

Before scoring, review the Evaluation Design so task intent, rubric dimensions, and scoring boundaries are fresh.

Phase 1: Research Log First

Before running real model calls, create RESEARCH_LOG.md.

Write a dated entry with your initial hypotheses:

• Which model do you expect to provide the best proof tutoring?
• Which proof topic do you expect to cause the most mistakes?
• Which task is most likely to trigger over-helping?
• Which rubric dimension will be hardest for an LLM judge to score?

Keep this entry before seeing real transcripts. That makes your research process more honest and useful.

Suggested first commit:

git add RESEARCH_LOG.md
git commit -m "Add initial research hypotheses"

Phase 2: Validate Without API Calls

Run dry-run and stub checks first:

uv run python run_conversations.py --dry-run --task proof01
uv run python run_conversations.py --stub
uv run python run_judge.py --dry-run --task proof01 --model stub
uv run python analyze.py --human examples/fixture_human_scores.csv --judge examples/fixture_judge_scores.csv

Confirm:

• dry-run does not write files
• stub mode writes 10 transcript JSON files and 10 Markdown files
• the judge dry-run prompt includes reference notes
• fixture analysis reports 16/18 exact agreement and 18/18 within one

Do not add real API keys until this phase works.

Stub transcripts are pipeline checks, not study results. Use them to verify transcript writing, judge prompt construction, and analysis mechanics, but do not include them in the real model comparison.

For real transcripts, hand-score first and run the LLM judge second. This order prevents the judge output from influencing the human scores.

Phase 3: Collect Model Transcripts

Add API keys to .env, then run one small real test:

uv run python run_conversations.py --task proof01 --model openai-frontier

Before running this command, confirm that the real-model branch in run_task_conversation calls call_real_model. It should not contain pass, and it should raise an error rather than saving blank assistant messages.

Open the saved transcript in:

evals/transcripts/openai-frontier/proof01.md

Check for:

• complete three-turn conversation
• no API error text saved as the assistant answer
• no reference key content sent to the model
• reasonable formatting in both JSON and Markdown

Then run the full collection:

uv run python run_conversations.py

Expected full-study output with the six configured models:

60 transcripts = 10 tasks x 6 models

Add a research log entry with any data collection problems or first impressions.

Phase 4: Human Scoring

Create or update:

evals/scores/human_scores.csv

Use the rubric dimensions from rubric.json:

• mathematical_correctness
• proof_scaffolding
• error_diagnosis
• student_agency
• proof_convention_alignment
• integrity_boundary

Use scores:

Write notes for every 0 or 1. Also write notes when a response is mathematically correct but pedagogically weak.

Important: do not run run_judge.py on real transcripts until human scoring is complete.

Phase 5: Automated Judge Scoring

After human scores are finished:

uv run python run_judge.py

This writes:

evals/scores/judge_scores.csv

Review a few judge rows manually:

• Are scores all 0, 1, or 2?
• Do notes mention real evidence from the transcript?
• Are any dimensions systematically too generous or too harsh?

Add a research log entry with early judge disagreement patterns.

Phase 6: Analysis and Findings

Generate the analysis report:

uv run python analyze.py --output analysis.md

Then create FINDINGS.md.

Use this structure:

# Proof Assistance Evals: Findings

## Abstract

## Methodology

## Results

## Failure Modes

## Human vs. Judge Agreement

## Limitations

## Conclusion

Your findings should include:

• exact model IDs and run dates
• number of transcripts collected
• summary tables from analyze.py
• at least three recurring proof-tutoring failure modes
• transcript evidence for each failure mode
• where the LLM judge agreed and disagreed with human scoring
• what you would change in a second version of the evaluation

Optional Extensions

If the core study is complete, choose one focused extension:

• Add 1-2 new tasks based on a failure mode you observed.
• Compare a second judge model.
• Run a system-prompt ablation with --no-system-prompt.
• Add per-task-type analysis to analyze.py.
• Create a small appendix of representative transcript excerpts.

Only extend after the base study is reproducible.

Final Checklist

• RESEARCH_LOG.md has initial hypotheses before real data.
• Dry-run and stub commands work.
• Real transcripts are collected.
• Human scores are complete before judge scoring.
• Judge scores are generated.
• analyze.py produces summary and agreement tables.
• FINDINGS.md explains results, failure modes, judge limits, and next steps.
• The commit history shows setup, implementation, data collection, scoring, and writeup milestones.

Back to the Core Path