Project Setup

This guide gets the project running on your computer before you implement the harness.

Do these steps first. You should be able to run the help commands without API keys.

What You Need

  • Git
  • Python 3.12 or higher
  • uv for Python environments and dependencies
  • Cursor or another code editor
  • API keys for real model runs, later
  • Optional: PathMX CLI if you want to preview this guide locally

You do not need API keys for the first implementation milestones. Start with dry-run and stub behavior.

Install Cursor

Cursor is the recommended editor for this project.

  1. Go to cursor.com/download.
  2. Download the installer for your computer:
    • Mac: choose the Mac download. If you are unsure which chip you have, choose the universal Mac download.
    • Windows: choose the Windows download. The user installer is usually fine for a personal laptop.
  3. Install it like a normal app.
  4. Open Cursor once to make sure it launches.

You can use another editor if you prefer, but the guide assumes Cursor for opening the project folder and using the integrated terminal.

Install uv

uv manages this project's Python environment and dependencies.

Mac

Open the Terminal app and run:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then close and reopen Terminal. Verify the install:

uv --version

If you use Homebrew, this also works:

brew install uv

Windows

Open PowerShell and run:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Then close and reopen PowerShell. Verify the install:

uv --version

If the command is blocked by Windows security settings, ask Professor Johnson for help before changing system settings.

Clone the Repo

First, make sure you have permission to the GitHub repository. Ask Professor Johnson if you do not have access.

Open a terminal in the folder where you keep coding projects, then run:

git clone git@github.com:pathmx/proof-assistance-evals-project.git
cd proof-assistance-evals-project

If Git says your SSH key is not configured, ask your instructor for help setting up GitHub SSH access or for the HTTPS clone URL.

Open the cloned folder in Cursor:

cursor .

If cursor . is not configured, open Cursor manually and choose File > Open Folder.

Install Dependencies

From the project root:

uv sync

This installs the Python dependencies listed in pyproject.toml using the locked versions in uv.lock.

Then create your local environment file:

cp .env.example .env

Leave the API key values blank for now. You will request the completed .env values from Professor Johnson after the local dry-run and stub paths work.

Verify the Starter Commands

These commands should show help text:

uv run python run_conversations.py --help
uv run python run_judge.py --help
uv run python analyze.py --help

At the start of the project, commands that actually run the scripts will fail with TODO messages. That is expected. Your implementation work is to replace those TODO failures one guide at a time.

Try the first target command:

uv run python run_conversations.py --dry-run --task proof01

Your first milestone is to make this command build a transcript preview without API calls or file writes.

Add API Keys Later

After dry-run and stub transcript generation work, ask Professor Johnson for the API keys or .env contents for this project. Then edit .env and add only the keys you need:

OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GEMINI_API_KEY=

The default model IDs live in models.json. Optional model override variables are already listed in .env.example.

The default suite uses six evaluated models: a frontier or balanced model from each provider and a lower-cost model from each provider. The judge model is configured separately.

Do not commit .env, paste API keys into chat, or include API keys in screenshots.

Recommended key-sharing process:

  • Professor Johnson should share keys through a password manager, university-approved encrypted channel, or provider project invite rather than GitHub, email, or plain chat.
  • Use project-scoped keys with spending limits when possible.
  • If a key has to be sent directly, rotate it after the study or after any accidental exposure.
  • Before every commit, run git status --short and confirm .env is not listed.
  • Leave optional model override variables blank unless Professor Johnson explicitly asks you to test a different model ID.

Preview This Path

If you have the PathMX CLI installed, preview the student path with:

pmx dev -p 3000 --build

Then open:

http://localhost:3000

Use this preview when you want to read the project guides as a linked path instead of separate Markdown files.

Setup Checkpoint

Before moving on, confirm:

  • The repo is cloned locally.
  • Cursor opens the project folder.
  • uv --version works.
  • uv sync completes.
  • .env exists but is not committed.
  • All three --help commands work.
  • You understand that real API keys should be requested from Professor Johnson after dry-run and stub behavior.

Next: Project Overview