GPT-5.4

Hi! I can help with things like: - Answering questions - Explaining concepts - Writing or editing emails, essays, and messages - Brainstorming ideas - Coding and debugging - Summarizing text - Planning trips, schedules, or projects - Math and problem-solving What would you like help with?

Thanks — I like that. We do make a good team. Let’s build something. Send me any of these: - a question you want answered - a draft you want rewritten - a tricky problem - code that’s misbehaving - a vague idea you want to turn into something real If you want, we can start with something simple: 1. Improve a paragraph 2. Brainstorm an idea 3. Solve a problem 4. Debug code 5. Plan something Pick one — or just paste whatever you’ve got.

Great idea. It’s clear, useful, and easy to instantly understand. My vote: **start with 4. write an MVP spec**, because that makes the other directions easier. Once the MVP is sharp, we can turn it into a startup idea, sketch the flow, or build it. ## Working title - **Thoughts → Plan** - **ClearNext** - **Mind2Action** - **Untangle** - **From Brain Dump to Plan** ## Core value proposition A tool that takes an unstructured mental mess and turns it into a **practical, prioritized action plan**. In one sentence: > “Paste your messy thoughts, and get back a clear list of priorities, next steps, timeline, and optional draft messages.” --- # MVP Spec ## 1) Target user Start narrow. Best early users might be: - busy professionals juggling too many tasks - founders/solo builders - students managing projects and deadlines - ADHD / overwhelm-prone users - managers trying to turn meeting notes into action Best initial wedge: > People who already keep messy notes but struggle to convert them into action. --- ## 2) Core user problem Users have: - too many thoughts - unclear priorities - mixed personal/professional tasks - anxiety from ambiguity - notes spread across docs, chats, and apps They need: - clarity - ordering - momentum - a first step --- ## 3) MVP input User pastes a “brain dump” containing any mix of: - notes - tasks - ideas - deadlines - worries - goals Example input: > Need to launch the landing page by next Friday. Still haven’t asked Sam for the pricing copy. Also should book dentist appointment and figure out budget for ads. Worried the onboarding flow is too confusing. Need to email investors sometime this week. Maybe push the webinar? --- ## 4) MVP output The tool returns 4 sections: ### A. Priority list Sorted by urgency + importance + dependency Example: 1. Ask Sam for pricing copy 2. Finalize landing page structure 3. Email investors this week 4. Review onboarding flow confusion 5. Decide whether to postpone webinar 6. Book dentist appointment 7. Draft ad budget ### B. Action plan Break larger items into concrete next steps Example: - **Landing page** - Confirm pricing copy owner - Finalize headline and CTA - Review mobile layout - Publish by Thursday for buffer ### C. Timeline / schedule suggestion Simple “today / this week / later” grouping ### D. Optional generated artifacts - draft email - calendar block suggestions - meeting agenda - checklist --- ## 5) MVP features Keep it tight. ### Must-have - text input box for messy thoughts - AI processing - output structured into: - priorities - next steps - deadlines/time buckets - ability to edit results - copy/export output ### Nice-to-have - choose mode: - work project - personal planning - mixed life admin - tone/style options: - gentle - direct - executive - regenerate section - convert output to Notion / markdown / checklist ### Not yet - full calendar integration - task sync with Asana/Todoist - long-term memory across sessions - collaboration - mobile app --- ## 6) UX flow Very simple first version: ### Step 1: Dump Prompt: > “What’s on your mind? Paste notes, worries, tasks, and deadlines.” ### Step 2: Clarify Optional follow-up questions if needed: - What deadline matters most? - Is this mostly work, personal, or mixed? - Do you want a quick plan or a detailed plan? ### Step 3: Organize System generates: - key goals detected - urgent items - blockers/dependencies - suggested sequence ### Step 4: Act User can: - copy to-do list - export plan - generate email - generate schedule --- ## 7) “Magic moment” The product wins if the user feels: > “Wow, this understands my chaos and gives me a path forward.” That means the output should feel: - calm - intelligent - specific - non-judgmental - immediately usable --- ## 8) Differentiator This should not just be “ChatGPT but with a text box.” Possible differentiators: ### Option A: Action-first structure Always returns the same clear framework: - what matters - what to do next - what can wait - what to communicate ### Option B: Anxiety-aware planning Designed for users who feel overwhelmed: - breaks things small - identifies ambiguity - suggests first tiny step ### Option C: Decision support Not just task extraction — also: - identifies unresolved decisions - flags blockers - suggests tradeoffs Example: > “You mentioned both launching next Friday and concern about onboarding confusion. You may need to choose between speed and polish.” That’s strong. --- ## 9) MVP prompt logic If built as a prompt system, the internal structure could be something like: ### System behavior - Extract tasks, goals, concerns, deadlines, and dependencies - Infer priorities conservatively - Separate actionable items from vague worries - Ask clarifying questions if key info is missing - Return concise structured output - Avoid overcommitting the user ### Output template - Summary of what you’re trying to do - Top priorities - Next 3 actions - Plan by time horizon - Open questions/blockers - Optional drafts --- ## 10) Success metrics For an MVP, measure: - % of users who paste input and get output - % who copy/export result - % who return within 7 days - time from paste to “useful plan” - user rating: “Did this reduce overwhelm?” 1–5 Best early qualitative signal: > “I actually used the output.” --- ## 11) Fastest ways to build it ### Option 1: Repeatable prompt system Fastest possible version. - one well-designed prompt - maybe inside ChatGPT / Claude / custom GPT - manual copy-paste workflow Best for validating usefulness. ### Option 2: Simple web app - textarea input - submit button - structured output cards - copy buttons - save session Tech stack: - Next.js - simple API route - LLM backend - basic auth later ### Option 3: Notion template + AI - user dumps text into page - AI button transforms into plan - less flexible, but easy for Notion-heavy users ### Option 4: CLI for power users Surprisingly good for builders: ```bash braindump "notes here..." ``` Returns markdown plan. --- ## 12) Best MVP choice I’d recommend: ### Phase 1 **Prompt system** Validate whether people actually want this transformation. ### Phase 2 **Simple web app** Textarea in, structured plan out. Why: - simplest UX - shareable - easy to test - easier to monetize later than a prompt alone --- ## 13) Early feature examples ### Example outputs From one dump, generate: - **Focus mode:** “Just tell me the next 3 things.” - **Planner mode:** “Make a full week plan.” - **Communication mode:** “Draft emails/messages for anything delegated.” - **Decision mode:** “Highlight choices I need to make first.” That could become a very nice product shape. --- ## 14) Risks A few product risks: - output feels generic - priorities are wrong - users want actual task syncing - messy inputs can be too ambiguous - people use it once but don’t build habit How to reduce risk: - optimize for the first-use wow moment - keep output editable - ask minimal clarifying questions - focus on “next step quality” --- ## 15) Possible positioning A few ways to frame it: ### Productivity angle > Turn brain dumps into organized action plans. ### Emotional relief angle > When your mind is crowded, get a calm plan in minutes. ### Executive assistant angle > Your AI thought-organizer for priorities, planning, and follow-through. ### ADHD-friendly angle > Go from overwhelm to next steps. That last one is strong but should be handled thoughtfully. --- # My recommendation The strongest initial concept is: > **An AI planning assistant that transforms messy brain dumps into prioritized, realistic next steps, with blocker detection and optional draft communications.** That’s a real product. ## Best next step Choose one of these: 1. **I turn this into a one-page startup concept** 2. **I write the actual MVP spec in product-doc format** 3. **I sketch the user flow screen by screen** 4. **I write the first prompt that powers the tool** 5. **I build a first version in code** If you want momentum, I’d suggest **4 next**: I can write the actual prompt so you can test the idea immediately.

Yes — **5 next** is the best move if you want this to become buildable fast. A stable JSON shape will make it much easier to: - render clean UI cards - compare outputs across prompt versions - store sessions - export to Notion/checklists - test reliability - add quick/full modes later So let’s turn the prompt into a **JSON-first spec**. --- # JSON Output Spec for the MVP ## Goal The model should return: - human-useful planning output - in a **predictable machine-friendly structure** - with minimal formatting cleanup --- # 1) Recommended JSON schema Here’s a practical first version: ```json { "summary": { "goal": "string", "main_tension": "string", "confidence": "high | medium | low" }, "top_priorities": [ { "title": "string", "reason": "string", "urgency": "high | medium | low", "importance": "high | medium | low", "depends_on": ["string"], "inferred": false } ], "next_actions": [ { "action": "string", "why_this_first": "string", "estimated_effort": "under_20_min | 20_to_60_min | more_than_1_hour | unknown", "inferred": false } ], "time_horizon": { "today": ["string"], "this_week": ["string"], "later": ["string"] }, "open_questions": [ { "question": "string", "type": "decision | ambiguity | tradeoff", "impact": "high | medium | low" } ], "blockers": [ { "blocker": "string", "type": "dependency | missing_info | internal_uncertainty | external_waiting", "blocked_items": ["string"] } ], "communication_drafts": [ { "recipient": "string", "channel": "email | message | unknown", "subject": "string", "draft": "string", "purpose": "string" } ], "detected_items": { "tasks": ["string"], "goals": ["string"], "deadlines": [ { "item": "string", "deadline": "string", "confidence": "high | medium | low" } ], "concerns": ["string"], "decisions": ["string"] }, "clarifying_questions": ["string"] } ``` --- # 2) Why this schema works This version is useful because it separates: - **what the model detected** - **what the model recommends** - **what remains unresolved** - **what can be rendered directly in product UI** It also gives you room for: - confidence - inferred vs explicit items - dependency tracking - quick/full mode --- # 3) Simpler MVP JSON schema If you want the leanest possible version for a first build, use this instead: ```json { "summary": "string", "top_priorities": [ { "title": "string", "reason": "string" } ], "next_actions": ["string"], "time_horizon": { "today": ["string"], "this_week": ["string"], "later": ["string"] }, "open_questions": ["string"], "blockers": ["string"], "communication_drafts": [ { "recipient": "string", "draft": "string" } ] } ``` This is probably the best **true MVP** schema. --- # 4) JSON-first system prompt Here’s a version of your prompt rewritten for structured output. ## JSON-first system prompt ```text You are an AI planning assistant that transforms messy brain dumps into clear, realistic action plans. Your goal is to reduce overwhelm and create momentum. Given unstructured thoughts, notes, tasks, deadlines, worries, and ideas, you should: 1. Extract and classify: - tasks - goals - deadlines - concerns - decisions - blockers - dependencies - delegated items - communication needs 2. Prioritize using: - urgency - importance - dependencies - effort-to-unblock ratio 3. Distinguish between: - concrete actions - unresolved decisions - vague worries - future/lower-priority items 4. Break large or unclear items into small, concrete next steps. 5. Be realistic: - never invent deadlines - clearly label inferred items - avoid over-ambitious plans - prefer momentum over completeness 6. If the user seems overwhelmed: - simplify - reduce immediate action count - choose the smallest useful next steps - keep tone calm and non-judgmental 7. Actively detect tradeoffs and hidden decisions. 8. If a task depends on another person, include a short draft message when useful. 9. If critical information is missing, ask up to 3 brief clarifying questions, but still provide the best possible first-pass plan unless impossible. Return valid JSON only. Do not include markdown. Do not include explanatory text outside the JSON. If something is unknown, use an empty string, empty array, or "unknown" where appropriate. Do not hallucinate deadlines or facts. ``` --- # 5) JSON user prompt wrapper ```text Turn the following brain dump into a structured action plan. Preferences: - Mode: {{mode}} - Style: {{style}} - Depth: {{depth}} Return JSON using the required schema. Brain dump: {{user_input}} ``` --- # 6) Exact schema instructions for the model To improve consistency, include the desired structure directly in the prompt. Example: ```text Return JSON with exactly these top-level keys: { "summary": { "goal": string, "main_tension": string, "confidence": "high" | "medium" | "low" }, "top_priorities": [ { "title": string, "reason": string, "urgency": "high" | "medium" | "low", "importance": "high" | "medium" | "low", "depends_on": [string], "inferred": boolean } ], "next_actions": [ { "action": string, "why_this_first": string, "estimated_effort": "under_20_min" | "20_to_60_min" | "more_than_1_hour" | "unknown", "inferred": boolean } ], "time_horizon": { "today": [string], "this_week": [string], "later": [string] }, "open_questions": [ { "question": string, "type": "decision" | "ambiguity" | "tradeoff", "impact": "high" | "medium" | "low" } ], "blockers": [ { "blocker": string, "type": "dependency" | "missing_info" | "internal_uncertainty" | "external_waiting", "blocked_items": [string] } ], "communication_drafts": [ { "recipient": string, "channel": "email" | "message" | "unknown", "subject": string, "draft": string, "purpose": string } ], "detected_items": { "tasks": [string], "goals": [string], "deadlines": [ { "item": string, "deadline": string, "confidence": "high" | "medium" | "low" } ], "concerns": [string], "decisions": [string] }, "clarifying_questions": [string] } ``` That explicitness usually helps a lot. --- # 7) Example JSON output Using your sample input, a good output might look like this: ```json { "summary": { "goal": "Launch the landing page by next Friday while managing a few related marketing, communication, and personal tasks.", "main_tension": "There is a tradeoff between shipping on time and addressing concerns about onboarding quality.", "confidence": "high" }, "top_priorities": [ { "title": "Ask Sam for pricing copy", "reason": "The landing page likely depends on this, and delays here could block launch progress.", "urgency": "high", "importance": "high", "depends_on": ["Sam responds with pricing copy"], "inferred": false }, { "title": "Finalize the landing page for next Friday's launch", "reason": "This is the clearest deadline mentioned.", "urgency": "high", "importance": "high", "depends_on": [], "inferred": false }, { "title": "Email investors this week", "reason": "This is a time-sensitive communication task with external stakeholders.", "urgency": "high", "importance": "medium", "depends_on": [], "inferred": false }, { "title": "Decide whether to postpone the webinar", "reason": "This may affect workload and launch focus.", "urgency": "medium", "importance": "medium", "depends_on": [], "inferred": true } ], "next_actions": [ { "action": "Send Sam a short message today asking for pricing copy and expected delivery timing.", "why_this_first": "It removes a dependency and could unblock the landing page quickly.", "estimated_effort": "under_20_min", "inferred": false }, { "action": "Make a short landing-page checklist covering headline, pricing section, CTA, mobile review, and publish date.", "why_this_first": "It turns the launch into a visible plan and reduces ambiguity.", "estimated_effort": "under_20_min", "inferred": true }, { "action": "Draft and send the investor email, or schedule a calendar block for it today.", "why_this_first": "It prevents an important communication task from drifting.", "estimated_effort": "20_to_60_min", "inferred": false } ], "time_horizon": { "today": [ "Ask Sam for pricing copy", "Draft or send investor email", "Create landing page launch checklist", "Decide whether webinar timing needs an immediate decision" ], "this_week": [ "Finalize landing page structure and content", "Review onboarding flow issues and decide whether any fix is required before launch", "Draft ad budget", "Book dentist appointment" ], "later": [ "Make broader onboarding improvements if not required before launch", "Resume webinar planning if postponed" ] }, "open_questions": [ { "question": "Does the onboarding issue need to be fixed before launch, or can it wait until after launch?", "type": "tradeoff", "impact": "high" }, { "question": "Should the webinar be postponed to protect focus on the launch?", "type": "decision", "impact": "medium" }, { "question": "When can Sam realistically provide pricing copy?", "type": "ambiguity", "impact": "high" } ], "blockers": [ { "blocker": "Waiting on pricing copy from Sam", "type": "external_waiting", "blocked_items": ["Finalize landing page content"] }, { "blocker": "Uncertainty about whether onboarding issues must be addressed before launch", "type": "internal_uncertainty", "blocked_items": ["Scope final launch plan"] } ], "communication_drafts": [ { "recipient": "Sam", "channel": "message", "subject": "", "draft": "Hey Sam — could you send over the pricing copy for the landing page? I’m aiming to launch by next Friday, so it would help to know when you think you can get it to me.", "purpose": "Unblock landing page work" }, { "recipient": "Investors", "channel": "email", "subject": "Quick update", "draft": "Hi [Name], quick update from my side: we’re targeting next Friday for the landing page launch and I wanted to share progress this week. [Add 1–2 concise updates here]. Happy to send a fuller note soon.", "purpose": "Maintain stakeholder communication" } ], "detected_items": { "tasks": [ "Launch the landing page by next Friday", "Ask Sam for pricing copy", "Book dentist appointment", "Figure out budget for ads", "Email investors this week", "Decide whether to push the webinar" ], "goals": [ "Launch the landing page" ], "deadlines": [ { "item": "Launch the landing page", "deadline": "next Friday", "confidence": "high" }, { "item": "Email investors", "deadline": "this week", "confidence": "high" } ], "concerns": [ "Onboarding flow may be too confusing" ], "decisions": [ "Whether to push the webinar", "Whether onboarding issues must be resolved before launch" ] }, "clarifying_questions": [] } ``` --- # 8) Quick mode vs full mode in JSON You mentioned two modes earlier. Good idea. ## Quick Plan JSON ```json { "summary": "string", "top_priorities": [ { "title": "string", "reason": "string" } ], "next_actions": ["string"], "time_horizon": { "today": ["string"], "this_week": ["string"], "later": ["string"] } } ``` ## Full Organize JSON Use the richer schema above. This lets your frontend switch between: - a clean simple result - a more detailed planning result --- # 9) Recommended MVP implementation detail If you’re building this in code, I’d strongly recommend: - store the **raw brain dump** - store the **raw model output** - store the **parsed JSON** - log parse failures Why: - you’ll want to compare prompt versions later - you’ll discover where the schema breaks - you can retry or repair malformed JSON --- # 10) Practical prompt tips to improve JSON reliability These matter a lot: ## Add strict output rules Use lines like: ```text Return valid JSON only. Do not wrap the JSON in markdown fences. Do not include commentary before or after the JSON. ``` ## Keep enum values small For example: - high / medium / low - decision / ambiguity / tradeoff This reduces weird drift. ## Prefer arrays over mixed freeform text Much easier to render and validate. ## Avoid too many nested objects at first Keep it rich enough to be useful, but not so dense that the model starts breaking structure. --- # 11) If you want even better reliability If your stack supports structured outputs / JSON schema enforcement, use it. For example: - OpenAI structured outputs - JSON schema validation - Zod / Pydantic on the server - automatic retry if parse fails A common pattern is: 1. ask model for JSON 2. validate 3. if invalid, run a repair pass or retry --- # 12) Suggested frontend card mapping This JSON maps cleanly to a UI: - `summary` → intro card - `top_priorities` → ranked priorities card - `next_actions` → “Start here” card - `time_horizon.today` / `this_week` / `later` → planning columns - `open_questions` → decisions card - `blockers` → unblockers card - `communication_drafts` → ready-to-send drafts card That’s already enough for a really nice MVP. --- # 13) My recommendation from here Best path: 1. **Start with the simpler MVP JSON schema** 2. test it on 10–20 messy inputs 3. check where outputs fail or feel generic 4. only then add richer fields like confidence and detected items So if you want the fastest route to working software, I’d use this **lean production schema** first: ```json { "summary": "string", "top_priorities": [ { "title": "string", "reason": "string" } ], "next_actions": ["string"], "time_horizon": { "today": ["string"], "this_week": ["string"], "later": ["string"] }, "open_questions": ["string"], "blockers": ["string"], "communication_drafts": [ { "recipient": "string", "draft": "string" } ] } ``` That’s enough to build and validate. --- # 14) Best next move I’d suggest one of these: 1. **I write the exact production prompt for the lean JSON schema** 2. **I generate 10 test brain dumps + expected JSON outputs** 3. **I build a simple TypeScript interface/Zod schema for this JSON** 4. **I scaffold the backend/frontend code for a first web app** 5. **I design a rating rubric for prompt quality testing** My recommendation: **3 next**, then **4**. That gets you from concept to implementation fast. If you want, I can do **3 right now** and give you: - TypeScript types - Zod schema - example parser usage - a fake response object for frontend development

Yes — let’s do **1: Minimal Next.js MVP**. Your package starter is solid. The next best step is to make it runnable with the smallest full loop: - paste brain dump - submit - call API route - get structured plan - render as cards - support mock mode before model integration is perfect Below is a **first-pass Next.js scaffold** using the App Router. --- # Stack - Next.js - TypeScript - App Router - Zod - your planning schema/parsers - mock fallback support --- # Suggested file structure ```txt src/ app/ api/ plan/ route.ts page.tsx layout.tsx globals.css components/ BrainDumpForm.tsx PlanView.tsx lib/ planning/ schema.ts parse.ts normalize.ts prompts.ts mock.ts service.ts ``` --- # 1) `src/lib/planning/schema.ts` ```ts import { z } from "zod"; export const PrioritySchema = z.object({ title: z.string(), reason: z.string(), }); export const TimeHorizonSchema = z.object({ today: z.array(z.string()), this_week: z.array(z.string()), later: z.array(z.string()), }); export const CommunicationDraftSchema = z.object({ recipient: z.string(), draft: z.string(), }); export const BrainDumpPlanSchema = z.object({ summary: z.string(), top_priorities: z.array(PrioritySchema), next_actions: z.array(z.string()), time_horizon: TimeHorizonSchema, open_questions: z.array(z.string()), blockers: z.array(z.string()), communication_drafts: z.array(CommunicationDraftSchema), }); export type BrainDumpPlan = z.infer<typeof BrainDumpPlanSchema>; ``` --- # 2) `src/lib/planning/parse.ts` ```ts import { BrainDumpPlanSchema, type BrainDumpPlan } from "./schema"; export type ParseResult = | { success: true; data: BrainDumpPlan } | { success: false; error: string; raw: string }; export function safeParseBrainDumpPlan(raw: string): ParseResult { try { const parsed = JSON.parse(raw); const validated = BrainDumpPlanSchema.safeParse(parsed); if (!validated.success) { return { success: false, error: validated.error.message, raw, }; } return { success: true, data: validated.data, }; } catch (error) { return { success: false, error: error instanceof Error ? error.message : "Unknown parse error", raw, }; } } ``` --- # 3) `src/lib/planning/normalize.ts` Optional, but useful. ```ts function asStringArray(value: unknown): string[] { if (Array.isArray(value)) return value.map((v) => String(v)); if (typeof value === "string" && value.trim()) return [value]; return []; } function asPriorityArray(value: unknown): { title: string; reason: string }[] { if (!Array.isArray(value)) return []; return value.map((item) => { const obj = item as Record<string, unknown>; return { title: String(obj?.title ?? ""), reason: String(obj?.reason ?? ""), }; }); } function asCommunicationDraftArray( value: unknown ): { recipient: string; draft: string }[] { if (!Array.isArray(value)) return []; return value.map((item) => { const obj = item as Record<string, unknown>; return { recipient: String(obj?.recipient ?? ""), draft: String(obj?.draft ?? ""), }; }); } export function normalizeBrainDumpPlan(input: unknown) { const obj = (input ?? {}) as Record<string, unknown>; const time = (obj.time_horizon ?? {}) as Record<string, unknown>; return { summary: String(obj.summary ?? ""), top_priorities: asPriorityArray(obj.top_priorities), next_actions: asStringArray(obj.next_actions), time_horizon: { today: asStringArray(time.today), this_week: asStringArray(time.this_week), later: asStringArray(time.later), }, open_questions: asStringArray(obj.open_questions), blockers: asStringArray(obj.blockers), communication_drafts: asCommunicationDraftArray(obj.communication_drafts), }; } ``` --- # 4) `src/lib/planning/prompts.ts` ```ts export const SYSTEM_PROMPT = ` You are an AI planning assistant that transforms messy brain dumps into clear, realistic action plans. Your goal is to reduce overwhelm and create momentum. Given unstructured thoughts, notes, tasks, deadlines, worries, and ideas: - identify priorities - suggest next steps - group items by time horizon - surface blockers and open questions - include draft messages if clearly useful Rules: - return valid JSON only - do not include markdown - do not include explanatory text before or after the JSON - do not invent deadlines - keep the output realistic, concise, and practical - if uncertain, make the best reasonable interpretation without pretending certainty Return JSON with exactly this shape: { "summary": "string", "top_priorities": [ { "title": "string", "reason": "string" } ], "next_actions": ["string"], "time_horizon": { "today": ["string"], "this_week": ["string"], "later": ["string"] }, "open_questions": ["string"], "blockers": ["string"], "communication_drafts": [ { "recipient": "string", "draft": "string" } ] } `.trim(); export function buildUserPrompt(input: { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }) { return ` Turn the following brain dump into a structured action plan. Preferences: - Mode: ${input.mode ?? "mixed"} - Style: ${input.style ?? "direct"} - Depth: ${input.depth ?? "standard"} Brain dump: ${input.brainDump} `.trim(); } ``` --- # 5) `src/lib/planning/mock.ts` ```ts import type { BrainDumpPlan } from "./schema"; export const mockBrainDumpPlan: BrainDumpPlan = { summary: "You’re trying to launch a landing page by next Friday while balancing a few related work tasks and personal admin.", top_priorities: [ { title: "Ask Sam for pricing copy", reason: "This may block completion of the landing page.", }, { title: "Finalize the landing page", reason: "This is tied to the clearest deadline mentioned.", }, { title: "Email investors this week", reason: "This is a time-sensitive communication task.", }, ], next_actions: [ "Send Sam a message asking for pricing copy and timing.", "Create a short landing-page checklist.", "Draft and send the investor email.", ], time_horizon: { today: [ "Ask Sam for pricing copy", "Draft or send investor email", "Create landing page checklist", ], this_week: [ "Finalize landing page structure and content", "Review onboarding flow issues", "Draft ad budget", "Book dentist appointment", ], later: [ "Improve onboarding after launch if not launch-critical", "Resume webinar planning if postponed", ], }, open_questions: [ "Does the onboarding issue need to be fixed before launch?", "Should the webinar be postponed to protect launch focus?", ], blockers: [ "Waiting on pricing copy from Sam", "Uncertainty about whether onboarding changes are required before launch", ], communication_drafts: [ { recipient: "Sam", draft: "Hey Sam — could you send over the pricing copy for the landing page? I’m aiming to launch by next Friday, so it would help to know when you think you can get it to me.", }, { recipient: "Investors", draft: "Hi [Name], quick update from my side: we’re targeting next Friday for the landing page launch and I wanted to share progress this week.", }, ], }; ``` --- # 6) `src/lib/planning/service.ts` This includes: - mock mode - placeholder model call - parsing - normalization fallback ```ts import { safeParseBrainDumpPlan } from "./parse"; import { normalizeBrainDumpPlan } from "./normalize"; import { BrainDumpPlanSchema, type BrainDumpPlan } from "./schema"; import { SYSTEM_PROMPT, buildUserPrompt } from "./prompts"; import { mockBrainDumpPlan } from "./mock"; type GenerateOptions = { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }; async function callYourModel(_args: { system: string; user: string; }): Promise<string> { throw new Error("Model integration not implemented yet"); } export async function generateBrainDumpPlan( options: GenerateOptions ): Promise<BrainDumpPlan> { const useMock = process.env.USE_MOCK_PLAN === "true"; if (useMock) { return mockBrainDumpPlan; } const responseText = await callYourModel({ system: SYSTEM_PROMPT, user: buildUserPrompt(options), }); const result = safeParseBrainDumpPlan(responseText); if (result.success) { return result.data; } // fallback attempt: normalize then validate try { const parsed = JSON.parse(responseText); const normalized = normalizeBrainDumpPlan(parsed); return BrainDumpPlanSchema.parse(normalized); } catch (error) { console.error("Plan parse failed:", result.error); console.error("Raw model output:", result.raw); throw new Error( error instanceof Error ? error.message : "Failed to parse plan response" ); } } ``` --- # 7) `src/app/api/plan/route.ts` ```ts import { NextRequest, NextResponse } from "next/server"; import { generateBrainDumpPlan } from "@/lib/planning/service"; export async function POST(req: NextRequest) { try { const body = await req.json(); const brainDump = String(body.brainDump ?? "").trim(); const mode = body.mode; const style = body.style; const depth = body.depth; if (!brainDump) { return NextResponse.json( { error: "brainDump is required" }, { status: 400 } ); } const plan = await generateBrainDumpPlan({ brainDump, mode, style, depth, }); return NextResponse.json({ plan }); } catch (error) { console.error("POST /api/plan failed", error); return NextResponse.json( { error: "Failed to generate plan" }, { status: 500 } ); } } ``` --- # 8) `src/components/BrainDumpForm.tsx` ```tsx "use client"; import { useState } from "react"; import type { BrainDumpPlan } from "@/lib/planning/schema"; type Props = { onResult: (plan: BrainDumpPlan) => void; }; export function BrainDumpForm({ onResult }: Props) { const [brainDump, setBrainDump] = useState(""); const [mode, setMode] = useState<"work" | "personal" | "mixed">("mixed"); const [style, setStyle] = useState<"gentle" | "direct" | "executive">("direct"); const [depth, setDepth] = useState<"quick" | "standard" | "detailed">("standard"); const [loading, setLoading] = useState(false); const [error, setError] = useState(""); async function handleSubmit(e: React.FormEvent) { e.preventDefault(); setLoading(true); setError(""); try { const res = await fetch("/api/plan", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ brainDump, mode, style, depth, }), }); const data = await res.json(); if (!res.ok) { throw new Error(data?.error || "Something went wrong"); } onResult(data.plan); } catch (err) { setError(err instanceof Error ? err.message : "Unknown error"); } finally { setLoading(false); } } return ( <form onSubmit={handleSubmit} className="space-y-4 rounded-xl border p-4 bg-white"> <div> <label className="block text-sm font-medium mb-2"> What’s on your mind? </label> <textarea value={brainDump} onChange={(e) => setBrainDump(e.target.value)} placeholder="Paste scattered notes, worries, tasks, and deadlines..." rows={10} className="w-full rounded-lg border p-3 outline-none" /> </div> <div className="grid gap-3 sm:grid-cols-3"> <div> <label className="block text-sm mb-1">Mode</label> <select value={mode} onChange={(e) => setMode(e.target.value as typeof mode)} className="w-full rounded-lg border p-2" > <option value="mixed">Mixed</option> <option value="work">Work</option> <option value="personal">Personal</option> </select> </div> <div> <label className="block text-sm mb-1">Style</label> <select value={style} onChange={(e) => setStyle(e.target.value as typeof style)} className="w-full rounded-lg border p-2" > <option value="direct">Direct</option> <option value="gentle">Gentle</option> <option value="executive">Executive</option> </select> </div> <div> <label className="block text-sm mb-1">Depth</label> <select value={depth} onChange={(e) => setDepth(e.target.value as typeof depth)} className="w-full rounded-lg border p-2" > <option value="standard">Standard</option> <option value="quick">Quick</option> <option value="detailed">Detailed</option> </select> </div> </div> <button type="submit" disabled={loading || !brainDump.trim()} className="rounded-lg bg-black text-white px-4 py-2 disabled:opacity-50" > {loading ? "Organizing..." : "Turn into plan"} </button> {error ? <p className="text-sm text-red-600">{error}</p> : null} </form> ); } ``` --- # 9) `src/components/PlanView.tsx` ```tsx import type { BrainDumpPlan } from "@/lib/planning/schema"; function Section({ title, children, }: { title: string; children: React.ReactNode; }) { return ( <section className="rounded-xl border bg-white p-4"> <h2 className="mb-3 text-lg font-semibold">{title}</h2> {children} </section> ); } export function PlanView({ plan }: { plan: BrainDumpPlan | null }) { if (!plan) return null; return ( <div className="mt-6 space-y-4"> <Section title="Summary"> <p className="text-sm leading-6">{plan.summary}</p> </Section> <Section title="Top Priorities"> <ol className="space-y-3 list-decimal pl-5"> {plan.top_priorities.map((item, i) => ( <li key={`${item.title}-${i}`}> <p className="font-medium">{item.title}</p> <p className="text-sm text-gray-600">{item.reason}</p> </li> ))} </ol> </Section> <Section title="Next Actions"> <ul className="space-y-2 list-disc pl-5"> {plan.next_actions.map((item, i) => ( <li key={`${item}-${i}`}>{item}</li> ))} </ul> </Section> <Section title="Plan by Time Horizon"> <div className="grid gap-4 md:grid-cols-3"> <div> <h3 className="font-medium mb-2">Today</h3> <ul className="space-y-2 list-disc pl-5"> {plan.time_horizon.today.map((item, i) => ( <li key={`${item}-${i}`}>{item}</li> ))} </ul> </div> <div> <h3 className="font-medium mb-2">This Week</h3> <ul className="space-y-2 list-disc pl-5"> {plan.time_horizon.this_week.map((item, i) => ( <li key={`${item}-${i}`}>{item}</li> ))} </ul> </div> <div> <h3 className="font-medium mb-2">Later</h3> <ul className="space-y-2 list-disc pl-5"> {plan.time_horizon.later.map((item, i) => ( <li key={`${item}-${i}`}>{item}</li> ))} </ul> </div> </div> </Section> <Section title="Open Questions"> {plan.open_questions.length ? ( <ul className="space-y-2 list-disc pl-5"> {plan.open_questions.map((item, i) => ( <li key={`${item}-${i}`}>{item}</li> ))} </ul> ) : ( <p className="text-sm text-gray-500">No major open questions detected.</p> )} </Section> <Section title="Blockers"> {plan.blockers.length ? ( <ul className="space-y-2 list-disc pl-5"> {plan.blockers.map((item, i) => ( <li key={`${item}-${i}`}>{item}</li> ))} </ul> ) : ( <p className="text-sm text-gray-500">No blockers detected.</p> )} </Section> <Section title="Draft Messages"> {plan.communication_drafts.length ? ( <div className="space-y-4"> {plan.communication_drafts.map((draft, i) => ( <div key={`${draft.recipient}-${i}`} className="rounded-lg border p-3"> <p className="font-medium mb-2">To: {draft.recipient}</p> <p className="text-sm whitespace-pre-wrap">{draft.draft}</p> </div> ))} </div> ) : ( <p className="text-sm text-gray-500">No draft messages suggested.</p> )} </Section> </div> ); } ``` --- # 10) `src/app/page.tsx` ```tsx "use client"; import { useState } from "react"; import { BrainDumpForm } from "@/components/BrainDumpForm"; import { PlanView } from "@/components/PlanView"; import type { BrainDumpPlan } from "@/lib/planning/schema"; export default function HomePage() { const [plan, setPlan] = useState<BrainDumpPlan | null>(null); return ( <main className="min-h-screen bg-gray-50 text-gray-900"> <div className="mx-auto max-w-4xl px-4 py-10"> <div className="mb-8"> <h1 className="text-3xl font-bold">Thoughts → Plan</h1> <p className="mt-2 text-gray-600"> Paste a messy brain dump and get back a clear action plan. </p> </div> <BrainDumpForm onResult={setPlan} /> <PlanView plan={plan} /> </div> </main> ); } ``` --- # 11) `src/app/layout.tsx` ```tsx import "./globals.css"; import type { Metadata } from "next"; export const metadata: Metadata = { title: "Thoughts → Plan", description: "Turn messy thoughts into clear action plans", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode }>) { return ( <html lang="en"> <body>{children}</body> </html> ); } ``` --- # 12) `src/app/globals.css` Minimal defaults if you’re using Tailwind already can stay light. If not, here’s a simple base: ```css html, body { padding: 0; margin: 0; font-family: Arial, Helvetica, sans-serif; } * { box-sizing: border-box; } body { background: #f8fafc; color: #111827; } textarea, select, button { font: inherit; } ``` --- # 13) If you are using Tailwind The JSX above uses Tailwind-style classes. If your app doesn’t have Tailwind yet: ```bash npm install -D tailwindcss postcss autoprefixer npx tailwindcss init -p ``` Then configure `tailwind.config.ts` and add: ```css @tailwind base; @tailwind components; @tailwind utilities; ``` to `globals.css`. If you want, I can also give you a **non-Tailwind version**. --- # 14) OpenAI integration drop-in If you want to wire the model next, replace `callYourModel()` in `service.ts` with something like this. ## Example with OpenAI SDK ```ts import OpenAI from "openai"; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); async function callYourModel(args: { system: string; user: string; }): Promise<string> { const response = await client.responses.create({ model: "gpt-4.1-mini", input: [ { role: "system", content: args.system, }, { role: "user", content: args.user, }, ], }); return response.output_text; } ``` If you want maximum reliability, I’d later move this to: - structured output enforcement - or function/tool schema output But for MVP, raw JSON + Zod is enough. --- # 15) Environment variables Add `.env.local`: ```env USE_MOCK_PLAN=true OPENAI_API_KEY=your_key_here ``` Start with mock mode: - `USE_MOCK_PLAN=true` Then switch to: - `USE_MOCK_PLAN=false` when model integration is ready. --- # 16) Nice small upgrades right after this Once this basic app works, the best next improvements are: 1. **Copy buttons** - copy next actions - copy drafts - copy whole plan 2. **Example brain dump button** - makes first-run testing easier 3. **Loading skeleton** - better UX while waiting on model 4. **Parse error fallback UI** - if model output breaks, show raw output or friendly retry 5. **Save runs** - store input + parsed output in SQLite/Postgres/Supabase --- # 17) Best immediate build order I’d implement in this order: 1. create files above 2. run in mock mode 3. verify UI flow 4. connect real model 5. test 10 messy inputs 6. log parse failures 7. tighten prompt/schema as needed That gets you to a usable prototype fast. --- # 18) My recommendation This is enough for a real first version. If you want, the strongest next thing I can do is one of: 1. **add copy/export buttons + polished UX** 2. **wire real OpenAI structured output into this scaffold** 3. **add persistence with Supabase/Postgres** 4. **generate 10 real test cases for prompt evaluation** 5. **convert this scaffold into a single pasteable repo structure** My recommendation: **2 next**. That would turn this from scaffold to working AI app.

Yes — **1 next** is the right move. Once the app is live, **reliability beats almost everything else**. So let’s upgrade from: - “please return JSON” to: - **schema-enforced structured output** That reduces: - malformed JSON - missing keys - markdown fences - weird formatting drift --- # What we’ll do We’ll create a stricter `service.ts` that: 1. calls OpenAI 2. requests structured JSON matching your schema 3. validates with Zod 4. falls back cleanly if needed I’ll give you: - a production-ready `service.ts` - a JSON schema object - one small helper - notes on how to use it safely --- # Important note Exact SDK helpers can vary by version. So I’ll show the **most practical pattern**: - define a JSON schema manually - pass it as `response_format` - still validate the returned object with Zod This gives you: - API-level structure enforcement - app-level validation That combo is strong. --- # 1) Add a JSON schema constant Create `src/lib/planning/jsonSchema.ts`: ```ts export const brainDumpPlanJsonSchema = { name: "brain_dump_plan", schema: { type: "object", additionalProperties: false, required: [ "summary", "top_priorities", "next_actions", "time_horizon", "open_questions", "blockers", "communication_drafts", ], properties: { summary: { type: "string", }, top_priorities: { type: "array", items: { type: "object", additionalProperties: false, required: ["title", "reason"], properties: { title: { type: "string" }, reason: { type: "string" }, }, }, }, next_actions: { type: "array", items: { type: "string", }, }, time_horizon: { type: "object", additionalProperties: false, required: ["today", "this_week", "later"], properties: { today: { type: "array", items: { type: "string" }, }, this_week: { type: "array", items: { type: "string" }, }, later: { type: "array", items: { type: "string" }, }, }, }, open_questions: { type: "array", items: { type: "string", }, }, blockers: { type: "array", items: { type: "string", }, }, communication_drafts: { type: "array", items: { type: "object", additionalProperties: false, required: ["recipient", "draft"], properties: { recipient: { type: "string" }, draft: { type: "string" }, }, }, }, }, }, strict: true, } as const; ``` --- # 2) Slightly simplify the system prompt Once schema enforcement is active, the prompt can focus more on behavior and less on formatting. Update `src/lib/planning/prompts.ts`: ```ts export const SYSTEM_PROMPT = ` You are an AI planning assistant that transforms messy brain dumps into clear, realistic action plans. Your goal is to reduce overwhelm and create momentum. Given unstructured thoughts, notes, tasks, deadlines, worries, and ideas: - identify priorities - suggest next steps - group items by time horizon - surface blockers and open questions - include draft messages if clearly useful Rules: - do not invent deadlines - keep the output realistic, concise, and practical - if uncertain, make the best reasonable interpretation without pretending certainty - if a field has no content, return an empty string or empty array - always include every required field `.trim(); export function buildUserPrompt(input: { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }) { return ` Turn the following brain dump into a structured action plan. Preferences: - Mode: ${input.mode ?? "mixed"} - Style: ${input.style ?? "direct"} - Depth: ${input.depth ?? "standard"} Brain dump: ${input.brainDump} `.trim(); } ``` --- # 3) Schema-enforced `service.ts` Replace `src/lib/planning/service.ts` with this: ```ts import OpenAI from "openai"; import { z } from "zod"; import { BrainDumpPlanSchema, type BrainDumpPlan } from "./schema"; import { normalizeBrainDumpPlan } from "./normalize"; import { SYSTEM_PROMPT, buildUserPrompt } from "./prompts"; import { mockBrainDumpPlan } from "./mock"; import { brainDumpPlanJsonSchema } from "./jsonSchema"; type GenerateOptions = { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); function extractStructuredJson(outputText: string): unknown { try { return JSON.parse(outputText); } catch { return null; } } async function callStructuredModel(args: { system: string; user: string; }): Promise<unknown> { const model = process.env.OPENAI_MODEL || "gpt-4.1-mini"; const response = await client.responses.create({ model, input: [ { role: "system", content: args.system, }, { role: "user", content: args.user, }, ], text: { format: { type: "json_schema", name: brainDumpPlanJsonSchema.name, schema: brainDumpPlanJsonSchema.schema, strict: true, }, }, }); // Best case: the SDK gives structured JSON text directly if (response.output_text && response.output_text.trim()) { const parsed = extractStructuredJson(response.output_text); if (parsed !== null) return parsed; } // Fallback for unexpected SDK response shapes during debugging return null; } export async function generateBrainDumpPlan( options: GenerateOptions ): Promise<BrainDumpPlan> { const useMock = process.env.USE_MOCK_PLAN === "true"; if (useMock) { return mockBrainDumpPlan; } if (!process.env.OPENAI_API_KEY) { throw new Error("Missing OPENAI_API_KEY"); } const structured = await callStructuredModel({ system: SYSTEM_PROMPT, user: buildUserPrompt(options), }); if (!structured) { throw new Error("Model did not return valid structured output"); } // First try strict validation const validated = BrainDumpPlanSchema.safeParse(structured); if (validated.success) { return validated.data; } // Then try normalization + validation as a backup try { const normalized = normalizeBrainDumpPlan(structured); return BrainDumpPlanSchema.parse(normalized); } catch (error) { console.error("Structured output validation failed"); console.error( "Validation error:", validated.error instanceof z.ZodError ? validated.error.flatten() : validated.error ); console.error("Raw structured output:", JSON.stringify(structured, null, 2)); throw new Error( error instanceof Error ? error.message : "Failed to validate structured plan response" ); } } ``` --- # 4) Why this version is better Compared with raw JSON prompting, this gives you: ## Better output consistency The model is constrained to your schema. ## Fewer parse failures No more hoping it avoids markdown fences or commentary. ## Cleaner backend logic You’re validating an object, not cleaning up freeform text first. ## Easier prompt iteration You can change behavior without disturbing output shape as much. --- # 5) If your SDK version complains about `text.format` Depending on SDK version, the exact property names may differ. If that happens, you have two practical options: ## Option A: Update SDK ```bash npm install openai@latest ``` ## Option B: Keep your current raw-JSON path as fallback You can support both modes in one service. Here’s a simple hybrid pattern: ```ts async function callYourModelWithFallback(args: { system: string; user: string; }): Promise<unknown> { const model = process.env.OPENAI_MODEL || "gpt-4.1-mini"; try { const response = await client.responses.create({ model, input: [ { role: "system", content: args.system }, { role: "user", content: args.user }, ], text: { format: { type: "json_schema", name: brainDumpPlanJsonSchema.name, schema: brainDumpPlanJsonSchema.schema, strict: true, }, }, }); if (response.output_text?.trim()) { return JSON.parse(response.output_text); } } catch (error) { console.warn("Structured output request failed, falling back:", error); } // Fallback: plain text JSON prompt const fallbackResponse = await client.responses.create({ model, input: [ { role: "system", content: args.system }, { role: "user", content: args.user }, ], }); if (!fallbackResponse.output_text?.trim()) { throw new Error("No response text returned from model"); } return JSON.parse(fallbackResponse.output_text); } ``` That’s a good MVP safety net. --- # 6) Add better route validation Your route is already fine, but this small improvement helps: ```ts import { NextRequest, NextResponse } from "next/server"; import { generateBrainDumpPlan } from "@/lib/planning/service"; export async function POST(req: NextRequest) { try { const body = await req.json(); const brainDump = String(body.brainDump ?? "").trim(); const mode = body.mode; const style = body.style; const depth = body.depth; if (!brainDump) { return NextResponse.json( { error: "brainDump is required" }, { status: 400 } ); } if (brainDump.length < 10) { return NextResponse.json( { error: "Please enter a bit more detail." }, { status: 400 } ); } const plan = await generateBrainDumpPlan({ brainDump, mode, style, depth, }); return NextResponse.json({ plan }); } catch (error) { console.error("POST /api/plan failed", error); return NextResponse.json( { error: error instanceof Error ? error.message : "Failed to generate plan", }, { status: 500 } ); } } ``` --- # 7) Recommended logging during this phase For a live prototype, log: - input length - model name - whether mock mode was used - whether structured parse succeeded - validation failures Example: ```ts console.log("Generating plan", { model: process.env.OPENAI_MODEL || "gpt-4.1-mini", useMock: process.env.USE_MOCK_PLAN === "true", inputLength: options.brainDump.length, }); ``` This will help a lot when outputs feel weak or inconsistent. --- # 8) What to test right away Use 5 kinds of inputs: 1. **clear work chaos** 2. **mixed work + personal** 3. **vague overwhelmed dump** 4. **deadline-heavy project** 5. **dependency-heavy planning** Good test questions: - Did it return every field? - Did priorities make sense? - Were next steps concrete? - Did drafts feel useful or forced? - Did it reduce overwhelm? --- # 9) Best next safeguard: typed request validation You may also want to validate incoming API input with Zod. Something like: ```ts import { z } from "zod"; export const GeneratePlanRequestSchema = z.object({ brainDump: z.string().min(10), mode: z.enum(["work", "personal", "mixed"]).optional(), style: z.enum(["gentle", "direct", "executive"]).optional(), depth: z.enum(["quick", "standard", "detailed"]).optional(), }); ``` Then parse `req.json()` against it in the route. Not necessary today, but very worthwhile soon. --- # 10) My practical recommendation For your MVP, I’d now run this exact progression: 1. **wire schema-enforced output** 2. test 10 real brain dumps 3. log failures + weak outputs 4. refine prompt wording 5. add copy/export 6. add persistence That gives you a real product loop. --- # Best next move I’d suggest one of these: 1. **I add request validation + better typed API errors** 2. **I generate a 10-case prompt evaluation suite** 3. **I add copy/export UX to the frontend** 4. **I add persistence with Supabase** 5. **I refactor this into a single pasteable repo snapshot** My recommendation: **2 next**. Now that the app can actually run, you want a **prompt evaluation suite** so you can improve quality systematically instead of by gut feel. If you want, I can generate: - 10 realistic brain dumps - expected qualities of good output - a scoring rubric - edge cases that should break weak prompts

Agreed — **golden outputs for 3–5 cases** are the sharpest next step. Why this helps: - scores tell you **something is off** - golden outputs tell you **what “good” looks like** - they make prompt tuning much faster - they help you compare models without relying on gut feel So below is a **golden output set for 5 cases** using your lean MVP schema. --- # Golden Outputs for Evaluation These are not the only acceptable answers. They are **reference-quality targets** for: - prioritization - actionability - realism - decision detection - blocker handling - useful drafts Use them to ask: > “Is the model output at least this good in spirit?” --- ## Case 1: Clear work chaos ### Input > Need to launch the landing page by next Friday. Still haven’t asked Sam for the pricing copy. Also should book dentist appointment and figure out budget for ads. Worried the onboarding flow is too confusing. Need to email investors sometime this week. Maybe push the webinar? ### Golden output ```json { "summary": "You’re trying to launch a landing page by next Friday while juggling a few related marketing, communication, and personal admin tasks. The main tension is whether onboarding concerns or webinar planning will interfere with the launch.", "top_priorities": [ { "title": "Ask Sam for pricing copy", "reason": "This is a likely dependency for the landing page and could delay launch if not handled immediately." }, { "title": "Finalize the landing page for next Friday", "reason": "This is tied to the clearest deadline in the brain dump." }, { "title": "Email investors this week", "reason": "This is a time-sensitive external communication task that should not drift." }, { "title": "Decide whether to push the webinar", "reason": "This affects workload and may reduce pressure on the launch timeline." } ], "next_actions": [ "Send Sam a message today asking for pricing copy and expected timing.", "Make a short landing page checklist covering headline, pricing, CTA, mobile review, and publish date.", "Draft and send the investor email today or block time for it on the calendar." ], "time_horizon": { "today": [ "Ask Sam for pricing copy", "Draft or send investor email", "Create a landing page launch checklist" ], "this_week": [ "Finalize landing page content and structure", "Review whether onboarding issues need any pre-launch fixes", "Decide whether to postpone the webinar", "Draft ad budget", "Book dentist appointment" ], "later": [ "Make broader onboarding improvements if they are not launch-critical", "Resume webinar planning if it gets postponed" ] }, "open_questions": [ "Does the onboarding issue need to be fixed before launch, or can it wait until after?", "Should the webinar be postponed to protect focus on the launch?", "When can Sam deliver the pricing copy?" ], "blockers": [ "Landing page completion may depend on pricing copy from Sam", "It is unclear whether onboarding concerns need to be addressed before launch" ], "communication_drafts": [ { "recipient": "Sam", "draft": "Hey Sam — could you send over the pricing copy for the landing page? I’m aiming to launch by next Friday, so it would help to know when you think you can get it to me." }, { "recipient": "Investors", "draft": "Hi [Name], quick update from my side: we’re targeting next Friday for the landing page launch and I wanted to share progress this week. Happy to send a fuller update soon." } ] } ``` ### What makes this “golden” - launch and dependency come first - webinar is treated as a **decision** - onboarding is treated as **uncertain scope** - dentist stays lower priority - next actions are concrete and low-ambiguity --- ## Case 3: Founder ambiguity / hidden decisions ### Input > I’m not sure whether to keep pushing on the enterprise pilot or focus on self-serve. The pilot might bring revenue faster, but it’s eating all our time. Need to follow up with two prospects, talk to the designer about onboarding, and figure out pricing page copy. Also investor update is late. ### Golden output ```json { "summary": "You’re balancing a strategic choice between pursuing an enterprise pilot for near-term revenue or focusing on self-serve for scalability. At the same time, you have a few execution tasks and a late investor update that need attention.", "top_priorities": [ { "title": "Send the late investor update", "reason": "This is already overdue and affects stakeholder trust." }, { "title": "Clarify whether the enterprise pilot or self-serve path is the current priority", "reason": "This decision affects how the rest of the work should be prioritized." }, { "title": "Follow up with the two prospects", "reason": "These are active opportunities and likely time-sensitive." }, { "title": "Resolve onboarding and pricing page copy tasks", "reason": "These support go-to-market execution, but priority depends on the strategic direction." } ], "next_actions": [ "Draft and send the investor update today.", "Write down the decision criteria for enterprise pilot vs self-serve so you can choose intentionally.", "Send follow-up messages to the two prospects." ], "time_horizon": { "today": [ "Send investor update", "Follow up with the two prospects", "Outline the decision criteria for enterprise pilot vs self-serve" ], "this_week": [ "Decide which path gets primary focus this week", "Talk to the designer about onboarding", "Figure out pricing page copy" ], "later": [ "Revisit the secondary path after the immediate focus is clear" ] }, "open_questions": [ "Is the enterprise pilot worth the time cost compared with focusing on self-serve?", "Are onboarding and pricing page updates needed for the current top priority, or are they premature?", "What do investors need to hear now versus in a fuller later update?" ], "blockers": [ "The enterprise vs self-serve decision is making execution priorities less clear" ], "communication_drafts": [ { "recipient": "Investors", "draft": "Hi [Name], quick update from my side: I’m working through a near-term strategy decision around enterprise versus self-serve focus, while continuing progress on prospects and onboarding. I’ll send a fuller update shortly, but wanted to avoid going quiet in the meantime." } ] } ``` ### What makes this “golden” - investor update is not buried - strategic tension is explicit - model does **not** pretend it knows the right strategy - next actions reduce ambiguity first --- ## Case 5: Overwhelm-heavy emotional dump ### Input > Everything feels behind. Inbox is a mess, taxes still aren’t done, I need to send that application in, I’m avoiding texting Alex back, apartment is a disaster, and I haven’t worked out in weeks. I don’t even know where to start and I keep opening tabs and doing nothing. ### Golden output ```json { "summary": "You seem overloaded and stuck between urgent admin tasks, personal avoidance, and general backlog stress. The goal here is not to do everything at once, but to reduce pressure by identifying the smallest high-value starting points.", "top_priorities": [ { "title": "Send the application", "reason": "This is likely the most time-sensitive item and usually has more consequence than general backlog tasks." }, { "title": "Move taxes forward", "reason": "This is an important admin task that tends to get heavier the longer it is delayed." }, { "title": "Reply to Alex", "reason": "This is a small but emotionally sticky task that may be draining attention." } ], "next_actions": [ "Open the application and identify the exact remaining step needed to submit it.", "Set a 20-minute timer and gather the documents or information needed for taxes.", "Send Alex a short reply, even if it is just to acknowledge the delay and say you will follow up properly later." ], "time_horizon": { "today": [ "Move the application one step closer to submission", "Do one short taxes session", "Reply to Alex" ], "this_week": [ "Continue taxes", "Triage the inbox into urgent, waiting, and later", "Do a basic apartment reset" ], "later": [ "Rebuild workout routine", "Do a deeper apartment clean" ] }, "open_questions": [ "Does the application have a deadline that should make it the immediate top priority?", "What is the smallest meaningful taxes task you can complete today instead of trying to finish everything at once?" ], "blockers": [ "Overwhelm and avoidance are making it hard to choose a starting point", "The application may be hard to prioritize without knowing its deadline" ], "communication_drafts": [ { "recipient": "Alex", "draft": "Hey — sorry for the delay. I’ve been a bit overloaded. Wanted to reply now instead of leaving you hanging, and I’ll follow up properly soon." } ] } ``` ### What makes this “golden” - tone is calming, not preachy - only a few immediate actions - “work out” is not falsely elevated - avoidance is handled practically - uncertainty about the application deadline is acknowledged --- ## Case 6: Dependency-heavy manager workflow ### Input > Need to finalize next quarter hiring plan, but I’m waiting on finance for headcount numbers. Also need feedback from Jordan on the engineering org proposal and legal still hasn’t approved the contractor agreement. I should prep Monday’s team meeting agenda and send one follow-up to a candidate. ### Golden output ```json { "summary": "You have several important planning tasks, but multiple items are currently blocked by finance, Jordan, and legal. The clearest active work is handling near-term communication and preparing for Monday’s team meeting.", "top_priorities": [ { "title": "Prep Monday’s team meeting agenda", "reason": "This is the clearest near-term commitment you fully control." }, { "title": "Send the candidate follow-up", "reason": "This is a live external communication that should not drift." }, { "title": "Unblock the hiring plan and contractor agreement", "reason": "Progress depends on getting responses from finance, Jordan, and legal." } ], "next_actions": [ "Draft the Monday team meeting agenda today.", "Send the candidate follow-up message.", "Send short nudges to finance, Jordan, and legal to clarify timing." ], "time_horizon": { "today": [ "Prepare Monday team meeting agenda", "Send candidate follow-up", "Send nudges to finance, Jordan, and legal" ], "this_week": [ "Revise hiring plan once headcount numbers arrive", "Update the engineering org proposal after Jordan’s feedback", "Advance the contractor agreement once legal responds" ], "later": [ "Finalize next quarter hiring plan after dependencies are resolved" ] }, "open_questions": [ "When will finance send the headcount numbers?", "Is Jordan’s feedback required before the engineering org proposal can move forward?", "Does legal have a timeline for contractor agreement approval?" ], "blockers": [ "Hiring plan is blocked by missing headcount numbers from finance", "Engineering org proposal is blocked by feedback from Jordan", "Contractor agreement is blocked by legal approval" ], "communication_drafts": [ { "recipient": "Finance", "draft": "Hi — quick follow-up on the headcount numbers for next quarter planning. Do you have an updated timeline for when those will be ready?" }, { "recipient": "Jordan", "draft": "Hey Jordan — just checking in on the engineering org proposal. When you have a moment, could you send your feedback so I can move it forward?" }, { "recipient": "Legal", "draft": "Hi — following up on the contractor agreement review. Is there an estimated timeline for approval or any outstanding issue I should address?" } ] } ``` ### What makes this “golden” - blocked items are not treated as fully actionable - controllable work comes first - communication drafts are highly relevant - dependencies are surfaced clearly --- ## Case 10: Sparse, messy, realistic ambiguity ### Input > ok so this week: taxes, deck, maybe doctor, send that follow-up, fix homepage thing, ask jen?, reschedule dinner, maybe move friday, not sure ### Golden output ```json { "summary": "You have several possible tasks for this week, but the input is ambiguous and some items are unclear. A useful first pass is to separate the likely important tasks from the items that need clarification.", "top_priorities": [ { "title": "Work on taxes", "reason": "This is likely an important admin task and appears to be one of the clearer items listed." }, { "title": "Move the deck forward", "reason": "A deck often implies a time-sensitive work deliverable, though the deadline is unclear." }, { "title": "Send the follow-up", "reason": "This is a concrete action that is quick to complete and may unblock something else." }, { "title": "Clarify the homepage issue and the Jen item", "reason": "These may matter, but they are too vague to prioritize confidently without more detail." } ], "next_actions": [ "Send the follow-up message.", "Define what the deck is for and whether it has a deadline this week.", "Write one line of clarification for 'fix homepage thing' and 'ask Jen?' so they can become real tasks." ], "time_horizon": { "today": [ "Send the follow-up", "Clarify the deck deadline or purpose", "Clarify what 'homepage thing' and 'ask Jen?' mean" ], "this_week": [ "Work on taxes", "Work on the deck", "Reschedule dinner", "Decide whether the doctor appointment matters this week" ], "later": [ "Revisit whether to move Friday once that item is clearer" ] }, "open_questions": [ "What is the deadline or purpose of the deck?", "Who is Jen and what do you need to ask her?", "What does 'move Friday' mean here?", "Is the doctor item urgent or optional?" ], "blockers": [ "Several items are too vague to prioritize confidently" ], "communication_drafts": [] } ``` ### What makes this “golden” - no hallucinated details - ambiguity is explicit - still gives a useful best-effort plan - does not force drafts where they are not clearly useful --- # How to use these golden outputs For each real run, compare against the reference on 3 levels: ## 1. Structure Did it return all required keys correctly? ## 2. Judgment Did it prioritize roughly the same way? ## 3. Usefulness Would a user act on it immediately? The output does **not** need to match line-for-line. It should match in: - priority direction - decision awareness - blocker awareness - realism - tone --- # Good comparison method When evaluating a run against a golden output, ask: ### Priority match - Did it get the top 1–3 priorities roughly right? - Did it avoid clearly wrong promotions? ### Action match - Are next steps equally concrete or better? ### Decision match - Did it catch the same big unresolved questions? ### Blocker match - Did it identify the key dependencies? ### Tone match - Is it at least as calm and usable? --- # Suggested “golden comparison” scoring add-on You can add one more score: ## 9. Golden alignment How close is the output to the intended reference quality for this case? - **1** = badly misaligned - **3** = mixed quality, some key misses - **5** = strongly aligned with the reference That makes your eval sheet even more useful. --- # Best next step Now that you have: - rubric - test suite - golden outputs the strongest next move is: 1. run the 5 golden cases through your app 2. compare output against these references 3. identify the top recurring weakness 4. revise the prompt once 5. rerun If you want, I can do either of these next: 1. **write a stronger v2 prompt optimized against these golden cases** 2. **create a spreadsheet-ready evaluation table with all 10 tests and scoring columns** 3. **write a small batch-testing script to run prompts and save outputs locally** 4. **generate golden outputs for the remaining 5 test cases** My recommendation: **1 next** — a **v2 prompt tuned against these failure modes**.

Agreed — **2 next** is the highest-leverage move now. A **batch test script** gives you a repeatable eval loop: - run all 10 cases - save outputs - compare prompt/model versions - score manually or semi-manually - stop testing by vibes Below is a practical MVP version. --- # Goal Create a script that: 1. loads test cases 2. sends each case through your planner 3. saves raw results to disk 4. records timing / failures 5. makes it easy to compare prompt versions later We’ll keep it simple and useful. --- # Recommended structure ```txt src/ eval/ cases.ts runEval.ts types.ts utils.ts ``` You can also keep this in a top-level `scripts/` folder if you prefer. --- # 1) `src/eval/types.ts` ```ts export type EvalCase = { id: string; name: string; category: string; brainDump: string; }; export type EvalResult = { caseId: string; caseName: string; category: string; brainDump: string; success: boolean; durationMs: number; error?: string; output?: unknown; createdAt: string; model: string; promptVersion: string; }; ``` --- # 2) `src/eval/cases.ts` Start with your 10 test cases. ```ts import type { EvalCase } from "./types"; export const evalCases: EvalCase[] = [ { id: "case_01", name: "Clear work chaos", category: "work_chaos", brainDump: "Need to launch the landing page by next Friday. Still haven’t asked Sam for the pricing copy. Also should book dentist appointment and figure out budget for ads. Worried the onboarding flow is too confusing. Need to email investors sometime this week. Maybe push the webinar?", }, { id: "case_02", name: "Mixed work + life overload", category: "mixed_overload", brainDump: "I need to finish the QBR deck, respond to my landlord about the lease renewal, call my mom back, schedule a performance review with Nina, and figure out whether I can make the wedding next month. I also haven’t submitted reimbursement from the last trip and I keep putting off the dentist. QBR is Thursday morning.", }, { id: "case_03", name: "Founder ambiguity / hidden decisions", category: "founder_strategy", brainDump: "I’m not sure whether to keep pushing on the enterprise pilot or focus on self-serve. The pilot might bring revenue faster, but it’s eating all our time. Need to follow up with two prospects, talk to the designer about onboarding, and figure out pricing page copy. Also investor update is late.", }, { id: "case_04", name: "Student project overload", category: "student_planning", brainDump: "I have a group presentation Monday, a stats assignment due tomorrow night, and I still haven’t read two articles for class. Need to message the group because nobody has decided who’s presenting what. I’m also behind on laundry and groceries and I kind of want to drop one club because it’s too much.", }, { id: "case_05", name: "Overwhelm-heavy emotional dump", category: "overwhelm", brainDump: "Everything feels behind. Inbox is a mess, taxes still aren’t done, I need to send that application in, I’m avoiding texting Alex back, apartment is a disaster, and I haven’t worked out in weeks. I don’t even know where to start and I keep opening tabs and doing nothing.", }, { id: "case_06", name: "Dependency-heavy manager workflow", category: "dependencies", brainDump: "Need to finalize next quarter hiring plan, but I’m waiting on finance for headcount numbers. Also need feedback from Jordan on the engineering org proposal and legal still hasn’t approved the contractor agreement. I should prep Monday’s team meeting agenda and send one follow-up to a candidate.", }, { id: "case_07", name: "Personal planning with ambiguity", category: "personal_decision", brainDump: "Need to plan the trip to Chicago, but not sure if I should stay with friends or get a hotel. Should probably book flights soon if prices keep rising. Also need to renew my passport, figure out dog care, and ask my manager if I can take Friday off.", }, { id: "case_08", name: "Meeting notes to action plan", category: "meeting_notes", brainDump: "From today’s meeting: Priya will send revised onboarding copy, we need to decide whether to cut scope for V1, analytics event tracking is incomplete, someone needs to confirm if support can cover weekend launch, and I said I’d share an updated timeline tomorrow.", }, { id: "case_09", name: "Messy delegated work", category: "delegation", brainDump: "Need to get the blog post live, but marketing still owes final images. I can write the intro tonight if needed. Also should check with Dev about the broken signup form, ping support about the customer complaint, and maybe just cancel Friday’s internal demo if we’re not ready.", }, { id: "case_10", name: "Sparse, messy, realistic ambiguity", category: "sparse_ambiguity", brainDump: "ok so this week: taxes, deck, maybe doctor, send that follow-up, fix homepage thing, ask jen?, reschedule dinner, maybe move friday, not sure", }, ]; ``` --- # 3) `src/eval/utils.ts` Helpers for writing files and timestamping. ```ts import fs from "node:fs/promises"; import path from "node:path"; export function timestamp() { return new Date().toISOString().replace(/[:.]/g, "-"); } export async function ensureDir(dir: string) { await fs.mkdir(dir, { recursive: true }); } export async function writeJsonFile(filePath: string, data: unknown) { const dir = path.dirname(filePath); await ensureDir(dir); await fs.writeFile(filePath, JSON.stringify(data, null, 2), "utf8"); } export async function writeTextFile(filePath: string, data: string) { const dir = path.dirname(filePath); await ensureDir(dir); await fs.writeFile(filePath, data, "utf8"); } ``` --- # 4) Export a prompt version constant In `src/lib/planning/prompts.ts`, add: ```ts export const PROMPT_VERSION = "v2"; ``` That way your eval outputs are tagged. --- # 5) `src/eval/runEval.ts` This is the main script. ```ts import path from "node:path"; import { evalCases } from "./cases"; import type { EvalResult } from "./types"; import { writeJsonFile, writeTextFile, timestamp, ensureDir } from "./utils"; import { generateBrainDumpPlan } from "@/lib/planning/service"; import { PROMPT_VERSION } from "@/lib/planning/prompts"; const MODEL = process.env.OPENAI_MODEL || "gpt-4.1-mini"; const OUTPUT_DIR = path.join(process.cwd(), "eval-results", timestamp()); async function run() { console.log(`Running eval suite with model=${MODEL}, prompt=${PROMPT_VERSION}`); await ensureDir(OUTPUT_DIR); const summary: EvalResult[] = []; for (const testCase of evalCases) { console.log(`\n→ ${testCase.id}: ${testCase.name}`); const started = Date.now(); try { const output = await generateBrainDumpPlan({ brainDump: testCase.brainDump, mode: "mixed", style: "direct", depth: "standard", }); const durationMs = Date.now() - started; const result: EvalResult = { caseId: testCase.id, caseName: testCase.name, category: testCase.category, brainDump: testCase.brainDump, success: true, durationMs, output, createdAt: new Date().toISOString(), model: MODEL, promptVersion: PROMPT_VERSION, }; summary.push(result); await writeJsonFile( path.join(OUTPUT_DIR, `${testCase.id}.json`), result ); console.log(` success (${durationMs}ms)`); } catch (error) { const durationMs = Date.now() - started; const result: EvalResult = { caseId: testCase.id, caseName: testCase.name, category: testCase.category, brainDump: testCase.brainDump, success: false, durationMs, error: error instanceof Error ? error.message : "Unknown error", createdAt: new Date().toISOString(), model: MODEL, promptVersion: PROMPT_VERSION, }; summary.push(result); await writeJsonFile( path.join(OUTPUT_DIR, `${testCase.id}.json`), result ); console.log(` failed (${durationMs}ms): ${result.error}`); } } await writeJsonFile(path.join(OUTPUT_DIR, "summary.json"), summary); const md = buildMarkdownSummary(summary); await writeTextFile(path.join(OUTPUT_DIR, "summary.md"), md); console.log(`\nDone. Results written to:\n${OUTPUT_DIR}`); } function buildMarkdownSummary(results: EvalResult[]) { const total = results.length; const successCount = results.filter((r) => r.success).length; const failureCount = total - successCount; const avgDuration = results.reduce((sum, r) => sum + r.durationMs, 0) / Math.max(total, 1); const lines: string[] = []; lines.push(`# Eval Summary`); lines.push(""); lines.push(`- Model: ${MODEL}`); lines.push(`- Prompt version: ${PROMPT_VERSION}`); lines.push(`- Total cases: ${total}`); lines.push(`- Successes: ${successCount}`); lines.push(`- Failures: ${failureCount}`); lines.push(`- Avg duration: ${Math.round(avgDuration)}ms`); lines.push(""); lines.push(`## Case Results`); lines.push(""); lines.push(`| Case | Category | Success | Duration | Error |`); lines.push(`|---|---|---:|---:|---|`); for (const r of results) { lines.push( `| ${r.caseId} - ${r.caseName} | ${r.category} | ${r.success ? "yes" : "no"} | ${r.durationMs}ms | ${r.error ?? ""} |` ); } return lines.join("\n"); } run().catch((error) => { console.error("Eval runner crashed:", error); process.exit(1); }); ``` --- # 6) Add a script command In `package.json`: ```json { "scripts": { "dev": "next dev", "eval": "tsx src/eval/runEval.ts" } } ``` If you don’t already have `tsx`: ```bash npm install -D tsx ``` --- # 7) Path alias note If `tsx` doesn’t like `@/lib/...` imports, you have two options: ## Option A: use relative imports in eval files Example: ```ts import { generateBrainDumpPlan } from "../lib/planning/service"; import { PROMPT_VERSION } from "../lib/planning/prompts"; ``` ## Option B: configure path aliases for runtime That’s fine later, but for speed I’d just use relative imports in the eval script. So this version may be safer: ```ts import { generateBrainDumpPlan } from "../lib/planning/service"; import { PROMPT_VERSION } from "../lib/planning/prompts"; ``` --- # 8) Example output folder After running: ```bash npm run eval ``` You’ll get something like: ```txt eval-results/ 2026-06-08T12-34-56-789Z/ case_01.json case_02.json ... case_10.json summary.json summary.md ``` Each case file contains: - input - success/failure - output - model - prompt version - timing That’s already enough for useful comparison. --- # 9) Recommended next improvement: save raw model output too Right now `generateBrainDumpPlan()` returns parsed data only. For eval work, it’s better if you can also capture: - raw model response - parsed output - validation success/failure So I’d suggest adding a debug variant in `service.ts`. For example: ```ts export type GeneratePlanDebugResult = { plan: BrainDumpPlan; raw?: unknown; }; export async function generateBrainDumpPlanDebug( options: GenerateOptions ): Promise<GeneratePlanDebugResult> { const plan = await generateBrainDumpPlan(options); return { plan }; } ``` Even better would be returning structured internals from the model call, but this simple shape is enough to start. If you want, I can help refactor that next. --- # 10) Add manual scoring template output A very useful upgrade: generate a CSV-ready scoring sheet beside the results. Add to `runEval.ts`: ```ts function buildScoringCsv(results: EvalResult[]) { const headers = [ "caseId", "caseName", "category", "model", "promptVersion", "success", "priorityQuality", "nextActionQuality", "timeHorizonUsefulness", "decisionDetection", "blockerDetection", "draftUsefulness", "overwhelmReduction", "schemaReliability", "goldenAlignment", "notes" ]; const rows = results.map((r) => [ r.caseId, `"${r.caseName}"`, r.category, r.model, r.promptVersion, r.success ? "true" : "false", "", "", "", "", "", "", "", "", "", '""', ].join(",") ); return [headers.join(","), ...rows].join("\n"); } ``` Then save it: ```ts const csv = buildScoringCsv(summary); await writeTextFile(path.join(OUTPUT_DIR, "scoring.csv"), csv); ``` That makes review much easier. --- # 11) Even better: compare two prompt versions Once this runner works, you can do: - run prompt v1 → save outputs - switch to v2 → save outputs - compare summaries and case JSON side-by-side A nice convention is: ```ts export const PROMPT_VERSION = "v2-golden-tuned"; ``` That way results folders are self-describing. --- # 12) Practical recommendation for your next hour Do this in order: 1. add these eval files 2. run with `USE_MOCK_PLAN=true` once to verify the loop 3. switch to real model 4. run all 10 cases 5. inspect `summary.md` 6. manually score in `scoring.csv` That gives you your first real evaluation cycle. --- # 13) Best next move after this Once you have batch outputs, the most useful next options are: 1. **refactor service to return raw + parsed debug data** 2. **add automatic heuristic checks** 3. **build a side-by-side prompt comparison script** 4. **generate golden outputs for the remaining 5 cases** My recommendation: **1 next**. Returning **raw + parsed debug data** will make your eval loop much more informative. If you want, I can do that next and give you: - a `generateBrainDumpPlanDebug()` version - eval output that includes raw model text / structured payload - cleaner failure analysis without changing your app API

Yes — **2 next** is the right move. A **side-by-side prompt comparison runner** is exactly what turns this from “I have outputs” into “I can improve the system on purpose.” It lets you compare: - **Prompt A vs Prompt B** - same test cases - same model - same schema - same app logic So differences are more likely due to the prompt, not random setup drift. --- # Goal Build a comparison runner that: 1. runs all eval cases against **two prompt versions** 2. stores outputs separately 3. writes a **comparison summary** 4. makes manual review much easier We’ll keep it practical and compatible with your current structure. --- # Recommended approach To compare prompts cleanly, the planner service should accept prompt overrides. So we’ll do this in 3 pieces: 1. refactor prompt handling slightly 2. let `generateBrainDumpPlanDebug()` accept prompt overrides 3. add a `comparePrompts.ts` eval runner --- # 1) Add prompt version exports Your existing `prompts.ts` should expose named prompt configs, not just one default. ## `src/lib/planning/prompts.ts` ```ts export type PromptConfig = { version: string; systemPrompt: string; buildUserPrompt: (input: { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }) => string; }; export const PROMPT_VERSION = "v2"; export const SYSTEM_PROMPT_V1 = ` You are an AI planning assistant that transforms messy brain dumps into clear, realistic action plans. Your goal is to reduce overwhelm and create momentum. Given unstructured thoughts, notes, tasks, deadlines, worries, and ideas: - identify priorities - suggest next steps - group items by time horizon - surface blockers and open questions - include draft messages if clearly useful Rules: - do not invent deadlines - keep the output realistic, concise, and practical - if uncertain, make the best reasonable interpretation without pretending certainty - if a field has no content, return an empty string or empty array - always include every required field `.trim(); export const SYSTEM_PROMPT_V2 = ` You are an AI planning assistant that turns messy brain dumps into clear, realistic action plans. Your goal is to reduce overwhelm and help the user act. You will receive unstructured notes that may include: - tasks - deadlines - worries - decisions - dependencies - personal admin - half-formed reminders - emotionally overloaded language Your job is to interpret the input carefully and produce a practical plan. Planning rules: 1. Prioritize using this order of judgment: - explicit deadlines - tasks that unblock other work - overdue or time-sensitive communication - decisions that affect other priorities - important but non-urgent admin - lower-stakes personal tasks 2. Treat different kinds of items differently: - actionable tasks -> put into priorities, next_actions, or time_horizon - unresolved decisions -> put into open_questions - dependencies/waiting-on-someone -> put into blockers - vague or unclear items -> acknowledge uncertainty rather than inventing detail 3. For next_actions: - choose the smallest useful actions - prefer actions that reduce ambiguity or unblock progress - prefer actions that can be started immediately - do not just restate a large project as a next step 4. For time_horizon: - keep "today" short and realistic - do not overload the user - put blocked items in "this_week" or "later" when appropriate - if something cannot move yet, do not pretend it belongs in "today" 5. For communication_drafts: - include drafts only when clearly useful - drafts should be short, natural, and practical - prefer drafts that unblock dependencies or address overdue communication - do not force drafts when none are obviously needed 6. For ambiguity: - do not hallucinate missing facts - do not invent deadlines - if something is unclear, surface it in open_questions or blockers - make the best reasonable interpretation, but stay honest about uncertainty 7. For overwhelmed users: - reduce the number of immediate actions - prioritize calm and momentum over completeness - avoid sounding judgmental, preachy, or overly cheerful - do not try to optimize their whole life at once 8. For mixed work and personal tasks: - prioritize based on urgency and consequence, not category alone - avoid letting small personal admin outrank real deadlines unless clearly warranted 9. Watch for hidden decisions and tradeoffs, such as: - ship now vs improve quality - delegate vs do personally - postpone vs proceed - blocked vs actively actionable - strategic path A vs strategic path B 10. Respect the requested style and depth: - gentle: softer wording, supportive but still practical - direct: clear, plain, action-oriented wording - executive: concise, high-signal, structured wording - quick: minimal plan focused on momentum - standard: balanced level of detail - detailed: fuller coverage, but still realistic Output quality rules: - be concise - be specific - be realistic - be immediately usable - include every required field - use empty arrays when appropriate Return structured output matching the required schema. `.trim(); export function buildUserPromptV1(input: { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }) { return ` Turn the following brain dump into a structured action plan. Preferences: - Mode: ${input.mode ?? "mixed"} - Style: ${input.style ?? "direct"} - Depth: ${input.depth ?? "standard"} Brain dump: ${input.brainDump} `.trim(); } export function buildUserPromptV2(input: { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }) { return ` Turn the following brain dump into a structured action plan. User preferences: - Mode: ${input.mode ?? "mixed"} - Style: ${input.style ?? "direct"} - Depth: ${input.depth ?? "standard"} Focus especially on: - identifying the top 1-3 real priorities - choosing next actions that are concrete and easy to start - separating decisions from tasks - identifying blockers clearly - keeping today's plan realistic Brain dump: ${input.brainDump} `.trim(); } export const PROMPT_V1: PromptConfig = { version: "v1", systemPrompt: SYSTEM_PROMPT_V1, buildUserPrompt: buildUserPromptV1, }; export const PROMPT_V2: PromptConfig = { version: "v2", systemPrompt: SYSTEM_PROMPT_V2, buildUserPrompt: buildUserPromptV2, }; export const DEFAULT_PROMPT_CONFIG = PROMPT_V2; export const SYSTEM_PROMPT = DEFAULT_PROMPT_CONFIG.systemPrompt; export const buildUserPrompt = DEFAULT_PROMPT_CONFIG.buildUserPrompt; ``` This gives you: - a default prompt for the app - reusable prompt configs for eval comparisons --- # 2) Update the planner service to accept prompt overrides Now let’s make the debug generator accept a custom prompt config. ## `src/lib/planning/service.ts` Here’s the key refactor. I’ll show the full file so it’s easy to paste. ```ts import OpenAI from "openai"; import { z } from "zod"; import { BrainDumpPlanSchema, type BrainDumpPlan } from "./schema"; import { normalizeBrainDumpPlan } from "./normalize"; import { SYSTEM_PROMPT, buildUserPrompt, PROMPT_VERSION, type PromptConfig, DEFAULT_PROMPT_CONFIG, } from "./prompts"; import { mockBrainDumpPlan } from "./mock"; import { brainDumpPlanJsonSchema } from "./jsonSchema"; type GenerateOptions = { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; }; type GenerateDebugOptions = GenerateOptions & { promptConfig?: PromptConfig; }; export type StructuredModelCallResult = { rawResponse: unknown; outputText: string; parsedStructured: unknown | null; model: string; }; export type GeneratePlanDebugResult = { plan: BrainDumpPlan; debug: { usedMock: boolean; model: string; promptVersion?: string; strictValidationPassed: boolean; normalizationUsed: boolean; rawResponse: unknown; outputText: string; parsedStructured: unknown | null; systemPrompt: string; userPrompt: string; }; }; const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }); function tryParseJson(text: string): unknown | null { try { return JSON.parse(text); } catch { return null; } } async function callStructuredModel(args: { system: string; user: string; }): Promise<StructuredModelCallResult> { const model = process.env.OPENAI_MODEL || "gpt-4.1-mini"; const response = await client.responses.create({ model, input: [ { role: "system", content: args.system, }, { role: "user", content: args.user, }, ], text: { format: { type: "json_schema", name: brainDumpPlanJsonSchema.name, schema: brainDumpPlanJsonSchema.schema, strict: true, }, }, }); const outputText = response.output_text ?? ""; const parsedStructured = outputText.trim() ? tryParseJson(outputText) : null; return { rawResponse: response, outputText, parsedStructured, model, }; } export async function generateBrainDumpPlanDebug( options: GenerateDebugOptions ): Promise<GeneratePlanDebugResult> { const useMock = process.env.USE_MOCK_PLAN === "true"; const model = process.env.OPENAI_MODEL || "gpt-4.1-mini"; const promptConfig = options.promptConfig ?? DEFAULT_PROMPT_CONFIG; const systemPrompt = promptConfig.systemPrompt; const userPrompt = promptConfig.buildUserPrompt(options); if (useMock) { return { plan: mockBrainDumpPlan, debug: { usedMock: true, model, promptVersion: promptConfig.version, strictValidationPassed: true, normalizationUsed: false, rawResponse: { mock: true }, outputText: JSON.stringify(mockBrainDumpPlan, null, 2), parsedStructured: mockBrainDumpPlan, systemPrompt, userPrompt, }, }; } if (!process.env.OPENAI_API_KEY) { throw new Error("Missing OPENAI_API_KEY"); } const structuredResult = await callStructuredModel({ system: systemPrompt, user: userPrompt, }); if (!structuredResult.parsedStructured) { console.error("Model did not return parseable structured output"); console.error("Output text:", structuredResult.outputText); throw new Error("Model did not return valid structured output"); } const strictValidated = BrainDumpPlanSchema.safeParse( structuredResult.parsedStructured ); if (strictValidated.success) { return { plan: strictValidated.data, debug: { usedMock: false, model: structuredResult.model, promptVersion: promptConfig.version, strictValidationPassed: true, normalizationUsed: false, rawResponse: structuredResult.rawResponse, outputText: structuredResult.outputText, parsedStructured: structuredResult.parsedStructured, systemPrompt, userPrompt, }, }; } try { const normalized = normalizeBrainDumpPlan(structuredResult.parsedStructured); const parsed = BrainDumpPlanSchema.parse(normalized); return { plan: parsed, debug: { usedMock: false, model: structuredResult.model, promptVersion: promptConfig.version, strictValidationPassed: false, normalizationUsed: true, rawResponse: structuredResult.rawResponse, outputText: structuredResult.outputText, parsedStructured: structuredResult.parsedStructured, systemPrompt, userPrompt, }, }; } catch (error) { console.error("Structured output validation failed"); console.error( "Validation error:", strictValidated.error instanceof z.ZodError ? strictValidated.error.flatten() : strictValidated.error ); console.error( "Parsed structured output:", JSON.stringify(structuredResult.parsedStructured, null, 2) ); throw new Error( error instanceof Error ? error.message : "Failed to validate structured plan response" ); } } export async function generateBrainDumpPlan( options: GenerateOptions ): Promise<BrainDumpPlan> { const result = await generateBrainDumpPlanDebug({ ...options, promptConfig: DEFAULT_PROMPT_CONFIG, }); return result.plan; } ``` --- # 3) Add comparison-specific eval types ## `src/eval/compareTypes.ts` ```ts export type PromptComparisonCaseResult = { caseId: string; caseName: string; category: string; brainDump: string; results: Array<{ promptVersion: string; success: boolean; durationMs: number; error?: string; output?: unknown; debug?: { strictValidationPassed: boolean; normalizationUsed: boolean; outputText: string; parsedStructured: unknown | null; systemPrompt: string; userPrompt: string; }; }>; }; export type PromptComparisonSummary = { model: string; comparedPromptVersions: string[]; createdAt: string; totalCases: number; cases: PromptComparisonCaseResult[]; }; ``` --- # 4) Build the side-by-side comparison runner ## `src/eval/comparePrompts.ts` ```ts import path from "node:path"; import { evalCases } from "./cases"; import { ensureDir, timestamp, writeJsonFile, writeTextFile } from "./utils"; import { generateBrainDumpPlanDebug } from "../lib/planning/service"; import { PROMPT_V1, PROMPT_V2, type PromptConfig, } from "../lib/planning/prompts"; import type { PromptComparisonCaseResult, PromptComparisonSummary, } from "./compareTypes"; const MODEL = process.env.OPENAI_MODEL || "gpt-4.1-mini"; const OUTPUT_DIR = path.join( process.cwd(), "eval-comparisons", timestamp() ); const PROMPTS_TO_COMPARE: PromptConfig[] = [PROMPT_V1, PROMPT_V2]; async function run() { console.log( `Comparing prompts [${PROMPTS_TO_COMPARE.map((p) => p.version).join(", ")}] with model=${MODEL}` ); await ensureDir(OUTPUT_DIR); const caseSummaries: PromptComparisonCaseResult[] = []; for (const testCase of evalCases) { console.log(`\n→ ${testCase.id}: ${testCase.name}`); const caseResult: PromptComparisonCaseResult = { caseId: testCase.id, caseName: testCase.name, category: testCase.category, brainDump: testCase.brainDump, results: [], }; for (const promptConfig of PROMPTS_TO_COMPARE) { const started = Date.now(); try { const resultDebug = await generateBrainDumpPlanDebug({ brainDump: testCase.brainDump, mode: "mixed", style: "direct", depth: "standard", promptConfig, }); const durationMs = Date.now() - started; caseResult.results.push({ promptVersion: promptConfig.version, success: true, durationMs, output: resultDebug.plan, debug: { strictValidationPassed: resultDebug.debug.strictValidationPassed, normalizationUsed: resultDebug.debug.normalizationUsed, outputText: resultDebug.debug.outputText, parsedStructured: resultDebug.debug.parsedStructured, systemPrompt: resultDebug.debug.systemPrompt, userPrompt: resultDebug.debug.userPrompt, }, }); console.log( ` ${promptConfig.version}: success (${durationMs}ms) strict=${resultDebug.debug.strictValidationPassed} normalized=${resultDebug.debug.normalizationUsed}` ); } catch (error) { const durationMs = Date.now() - started; caseResult.results.push({ promptVersion: promptConfig.version, success: false, durationMs, error: error instanceof Error ? error.message : "Unknown error", }); console.log( ` ${promptConfig.version}: failed (${durationMs}ms)` ); } } caseSummaries.push(caseResult); await writeJsonFile( path.join(OUTPUT_DIR, `${testCase.id}.comparison.json`), caseResult ); } const summary: PromptComparisonSummary = { model: MODEL, comparedPromptVersions: PROMPTS_TO_COMPARE.map((p) => p.version), createdAt: new Date().toISOString(), totalCases: evalCases.length, cases: caseSummaries, }; await writeJsonFile(path.join(OUTPUT_DIR, "comparison-summary.json"), summary); await writeTextFile( path.join(OUTPUT_DIR, "comparison-summary.md"), buildMarkdownComparison(summary) ); await writeTextFile( path.join(OUTPUT_DIR, "comparison-scoring.csv"), buildComparisonScoringCsv(summary) ); console.log(`\nDone. Comparison written to:\n${OUTPUT_DIR}`); } function buildMarkdownComparison(summary: PromptComparisonSummary) { const lines: string[] = []; lines.push(`# Prompt Comparison Summary`); lines.push(""); lines.push(`- Model: ${summary.model}`); lines.push(`- Prompt versions: ${summary.comparedPromptVersions.join(", ")}`); lines.push(`- Total cases: ${summary.totalCases}`); lines.push(`- Created at: ${summary.createdAt}`); lines.push(""); lines.push(`## Case Overview`); lines.push(""); lines.push(`| Case | Category | Prompt | Success | Strict | Normalized | Duration | Error |`); lines.push(`|---|---|---|---:|---:|---:|---:|---|`); for (const c of summary.cases) { for (const r of c.results) { lines.push( `| ${c.caseId} - ${c.caseName} | ${c.category} | ${r.promptVersion} | ${r.success ? "yes" : "no"} | ${r.debug?.strictValidationPassed ? "yes" : ""} | ${r.debug?.normalizationUsed ? "yes" : ""} | ${r.durationMs}ms | ${r.error ?? ""} |` ); } } return lines.join("\n"); } function buildComparisonScoringCsv(summary: PromptComparisonSummary) { const headers = [ "caseId", "caseName", "category", "model", "promptVersion", "success", "strictValidationPassed", "normalizationUsed", "priorityQuality", "nextActionQuality", "timeHorizonUsefulness", "decisionDetection", "blockerDetection", "draftUsefulness", "overwhelmReduction", "schemaReliability", "goldenAlignment", "notes", ]; const rows: string[] = []; for (const c of summary.cases) { for (const r of c.results) { rows.push( [ c.caseId, `"${c.caseName}"`, c.category, summary.model, r.promptVersion, r.success ? "true" : "false", r.debug?.strictValidationPassed ? "true" : "false", r.debug?.normalizationUsed ? "true" : "false", "", "", "", "", "", "", "", "", "", '""', ].join(",") ); } } return [headers.join(","), ...rows].join("\n"); } run().catch((error) => { console.error("Prompt comparison runner crashed:", error); process.exit(1); }); ``` --- # 5) Add npm script In `package.json`: ```json { "scripts": { "dev": "next dev", "eval": "tsx src/eval/runEval.ts", "eval:compare": "tsx src/eval/comparePrompts.ts" } } ``` --- # 6) What this gives you Running: ```bash npm run eval:compare ``` produces a folder like: ```txt eval-comparisons/ 2026-06-08T13-22-10-123Z/ case_01.comparison.json case_02.comparison.json ... comparison-summary.json comparison-summary.md comparison-scoring.csv ``` Each case file contains both prompt outputs, which makes manual review much easier. --- # 7) Example comparison file shape A case file will look roughly like this: ```json { "caseId": "case_01", "caseName": "Clear work chaos", "category": "work_chaos", "brainDump": "Need to launch the landing page by next Friday...", "results": [ { "promptVersion": "v1", "success": true, "durationMs": 1480, "output": { "summary": "...", "top_priorities": [] }, "debug": { "strictValidationPassed": true, "normalizationUsed": false, "outputText": "{...}", "parsedStructured": {}, "systemPrompt": "...", "userPrompt": "..." } }, { "promptVersion": "v2", "success": true, "durationMs": 1522, "output": { "summary": "...", "top_priorities": [] }, "debug": { "strictValidationPassed": true, "normalizationUsed": false, "outputText": "{...}", "parsedStructured": {}, "systemPrompt": "...", "userPrompt": "..." } } ] } ``` That’s enough for: - human review - future automation - prompt diffing --- # 8) Very useful next improvement: add winner columns Once you start using `comparison-scoring.csv`, add columns like: - `winnerOverall` - `winnerPriorityQuality` - `winnerNextActionQuality` You can do that manually in Sheets or add them to the CSV template. For example: ```csv caseId,caseName,category,model,promptVersion,success,...,goldenAlignment,overallScore,winnerOverall,notes ``` But I’d keep the first version simple. --- # 9) Important caveat: model variance Even with the same prompt and same model, outputs can vary. So: - one comparison run is useful - multiple runs are better if you’re making close calls If needed later, you can extend this runner to do: - `n=3` runs per prompt per case - average human scores across runs But for now, single-run side-by-side is enough. --- # 10) Best practice for prompt comparison discipline Use the runner like this: ## Round 1 Compare: - `v1` - `v2` Across all 10 cases. ## Round 2 Manually score only the important categories: - priority quality - next-action quality - overwhelm reduction - schema reliability - golden alignment ## Round 3 Choose the winner by pattern, not one flashy case. That avoids overfitting to a single example. --- # 11) Strong next improvement after this Once you have side-by-side outputs, the best next additions are: 1. **automatic heuristic checks** 2. **multi-run comparisons** 3. **HTML diff report** 4. **golden outputs for the remaining 5 cases** My recommendation: **1 next**. Why? Because heuristics can catch obvious regressions automatically, like: - too many items in `today` - no blockers when dependencies exist - empty drafts in dependency-heavy cases - too many generic next actions That gives you faster signal before manual review. If you want, I can do that next and give you: - a heuristic scoring module - per-case warnings - automatic flags in the comparison summary

Yes — **1 next** is the right move. General heuristics are good for sanity checks. **Case-specific heuristics** are what make the eval suite actually sharp. They help catch regressions like: - “case 01 stopped mentioning Sam/pricing copy” - “case 03 no longer surfaces enterprise vs self-serve” - “case 06 missed finance/legal blockers” - “case 10 started hallucinating Jen details” That’s exactly the kind of automated signal you want. --- # Goal Add **targeted heuristic rules for your eval cases** so the comparison runner can flag: - missed key priorities - missed decisions - missed blockers - forced drafts - hallucination risks We’ll do this in a way that layers cleanly on top of your existing general heuristics. --- # Plan We’ll add: 1. a case-specific heuristic module 2. rule helpers 3. integration into the existing heuristic evaluator 4. richer warnings in comparison output --- # 1) Update heuristic types Let’s slightly improve the type shape so checks can identify whether they are: - general - case-specific ## `src/eval/heuristicTypes.ts` ```ts export type HeuristicCheck = { id: string; passed: boolean; severity: "info" | "warning" | "error"; message: string; scope?: "general" | "case_specific"; }; export type HeuristicEvaluation = { score: number; // 0-100 checks: HeuristicCheck[]; warnings: string[]; }; ``` --- # 2) Add case-specific heuristics module ## `src/eval/caseSpecificHeuristics.ts` ```ts import type { BrainDumpPlan } from "../lib/planning/schema"; import type { EvalCase } from "./types"; import type { HeuristicCheck } from "./heuristicTypes"; function lower(text: string) { return text.toLowerCase(); } function includesAny(text: string, phrases: string[]) { const t = lower(text); return phrases.some((p) => t.includes(lower(p))); } function planText(plan: BrainDumpPlan) { return lower( [ plan.summary, ...plan.top_priorities.map((p) => `${p.title} ${p.reason}`), ...plan.next_actions, ...plan.time_horizon.today, ...plan.time_horizon.this_week, ...plan.time_horizon.later, ...plan.open_questions, ...plan.blockers, ...plan.communication_drafts.map((d) => `${d.recipient} ${d.draft}`), ].join("\n") ); } function priorityText(plan: BrainDumpPlan) { return lower( plan.top_priorities .map((p) => `${p.title} ${p.reason}`) .join("\n") ); } function openQuestionText(plan: BrainDumpPlan) { return lower(plan.open_questions.join("\n")); } function blockerText(plan: BrainDumpPlan) { return lower(plan.blockers.join("\n")); } function draftText(plan: BrainDumpPlan) { return lower( plan.communication_drafts .map((d) => `${d.recipient} ${d.draft}`) .join("\n") ); } function todayText(plan: BrainDumpPlan) { return lower(plan.time_horizon.today.join("\n")); } function nextActionsText(plan: BrainDumpPlan) { return lower(plan.next_actions.join("\n")); } function check(params: { id: string; passed: boolean; severity: "info" | "warning" | "error"; message: string; }): HeuristicCheck { return { ...params, scope: "case_specific", }; } export function evaluateCaseSpecificHeuristics( testCase: EvalCase, plan: BrainDumpPlan ): HeuristicCheck[] { const checks: HeuristicCheck[] = []; const allText = planText(plan); const priorities = priorityText(plan); const questions = openQuestionText(plan); const blockers = blockerText(plan); const drafts = draftText(plan); const today = todayText(plan); const next = nextActionsText(plan); switch (testCase.id) { case "case_01": { checks.push( check({ id: "must_notice_sam_or_pricing_copy", passed: includesAny(allText, ["sam", "pricing copy", "pricing"]), severity: "error", message: "Case 01 should mention Sam or pricing copy somewhere in the plan.", }) ); checks.push( check({ id: "must_treat_webinar_as_decision", passed: includesAny(questions, ["webinar", "push the webinar", "postpone"]), severity: "warning", message: "Case 01 should surface the webinar as an open question or decision.", }) ); checks.push( check({ id: "should_not_prioritize_dentist_above_launch_work", passed: !includesAny(priorities, ["dentist"]) || includesAny(priorities, ["landing page", "pricing copy", "investor"]), severity: "warning", message: "Case 01 should not let dentist/admin crowd out launch-critical work.", }) ); checks.push( check({ id: "should_include_sam_draft", passed: includesAny(drafts, ["sam"]), severity: "warning", message: "Case 01 should usually include a short draft to Sam.", }) ); break; } case "case_02": { checks.push( check({ id: "must_notice_qbr_deadline", passed: includesAny(allText, ["qbr", "thursday"]), severity: "error", message: "Case 02 should capture the QBR and its deadline pressure.", }) ); checks.push( check({ id: "qbr_should_appear_in_priorities", passed: includesAny(priorities, ["qbr", "deck"]), severity: "error", message: "Case 02 should prioritize the QBR deck.", }) ); checks.push( check({ id: "should_not_overload_today", passed: plan.time_horizon.today.length <= 5, severity: "warning", message: "Case 02 should avoid turning mixed overload into an unrealistic today list.", }) ); break; } case "case_03": { checks.push( check({ id: "must_surface_enterprise_vs_selfserve", passed: includesAny(allText, [ "enterprise", "self-serve", "self serve", "pilot", ]), severity: "error", message: "Case 03 should surface the enterprise vs self-serve strategic tension.", }) ); checks.push( check({ id: "must_treat_strategy_as_decision", passed: includesAny(questions, [ "enterprise", "self-serve", "self serve", "pilot", ]), severity: "warning", message: "Case 03 should place the core strategy tension in open questions.", }) ); checks.push( check({ id: "should_notice_investor_update", passed: includesAny(allText, ["investor update", "investor"]), severity: "warning", message: "Case 03 should include the late investor update.", }) ); break; } case "case_04": { checks.push( check({ id: "must_notice_stats_due_tomorrow", passed: includesAny(allText, ["stats", "tomorrow"]), severity: "error", message: "Case 04 should capture the stats assignment due tomorrow.", }) ); checks.push( check({ id: "must_notice_group_coordination", passed: includesAny(allText, ["group", "presentation", "presenting what"]), severity: "warning", message: "Case 04 should notice the group coordination issue.", }) ); checks.push( check({ id: "drop_club_should_be_decision", passed: includesAny(questions, ["club", "drop"]), severity: "warning", message: "Case 04 should treat dropping a club as a decision, not a settled task.", }) ); break; } case "case_05": { checks.push( check({ id: "should_limit_next_actions_for_overwhelm", passed: plan.next_actions.length <= 3, severity: "warning", message: "Case 05 should keep next actions very short due to overwhelm.", }) ); checks.push( check({ id: "must_notice_application_or_taxes", passed: includesAny(allText, ["application", "taxes"]), severity: "error", message: "Case 05 should prioritize the application or taxes rather than only guilt tasks.", }) ); checks.push( check({ id: "should_not_overprioritize_workout", passed: !includesAny(priorities, ["work out", "workout", "exercise"]), severity: "warning", message: "Case 05 should not elevate workout above likely urgent admin without evidence.", }) ); checks.push( check({ id: "alex_reply_should_be_handled_as_small_action_or_draft", passed: includesAny(allText, ["alex"]) && (includesAny(next, ["alex"]) || includesAny(drafts, ["alex"])), severity: "warning", message: "Case 05 should usually turn the Alex reply into a small manageable action or draft.", }) ); break; } case "case_06": { checks.push( check({ id: "must_notice_finance_jordan_or_legal_blockers", passed: includesAny(blockers, ["finance", "jordan", "legal"]), severity: "error", message: "Case 06 should explicitly identify blockers involving finance, Jordan, or legal.", }) ); checks.push( check({ id: "should_prioritize_meeting_or_candidate_followup", passed: includesAny(priorities, ["meeting", "candidate", "agenda"]), severity: "warning", message: "Case 06 should prioritize controllable near-term work like the meeting agenda or candidate follow-up.", }) ); checks.push( check({ id: "should_include_nudge_drafts", passed: includesAny(drafts, ["finance", "jordan", "legal", "candidate"]), severity: "warning", message: "Case 06 should usually include at least one useful follow-up draft.", }) ); break; } case "case_07": { checks.push( check({ id: "must_surface_trip_decision", passed: includesAny(questions, ["friends", "hotel", "stay with friends"]), severity: "warning", message: "Case 07 should surface the stay-with-friends vs hotel decision.", }) ); checks.push( check({ id: "should_notice_flights_or_manager_approval", passed: includesAny(allText, ["flights", "manager", "friday off"]), severity: "warning", message: "Case 07 should notice flight urgency or manager approval as near-term planning needs.", }) ); break; } case "case_08": { checks.push( check({ id: "must_notice_timeline_tomorrow", passed: includesAny(allText, ["timeline", "tomorrow"]), severity: "error", message: "Case 08 should clearly capture the updated timeline due tomorrow.", }) ); checks.push( check({ id: "must_surface_v1_scope_decision", passed: includesAny(questions, ["scope", "v1", "cut scope"]), severity: "warning", message: "Case 08 should surface the V1 scope decision.", }) ); checks.push( check({ id: "should_notice_support_or_priya_dependency", passed: includesAny(blockers, ["support", "priya", "copy"]), severity: "warning", message: "Case 08 should notice support coverage or Priya's copy as dependencies.", }) ); break; } case "case_09": { checks.push( check({ id: "must_notice_signup_or_customer_issue", passed: includesAny(allText, [ "signup form", "broken signup", "customer complaint", "support", ]), severity: "error", message: "Case 09 should notice the broken signup form or customer complaint as urgent work.", }) ); checks.push( check({ id: "must_treat_blog_images_as_dependency", passed: includesAny(blockers, ["images", "marketing"]), severity: "warning", message: "Case 09 should identify final images/marketing as a dependency for the blog post.", }) ); checks.push( check({ id: "should_surface_demo_as_decision", passed: includesAny(questions, ["demo", "cancel", "not ready"]), severity: "warning", message: "Case 09 should treat canceling Friday's internal demo as a decision.", }) ); break; } case "case_10": { checks.push( check({ id: "must_preserve_uncertainty", passed: plan.open_questions.length > 0 || plan.blockers.length > 0, severity: "error", message: "Case 10 should explicitly preserve ambiguity rather than pretending clarity.", }) ); checks.push( check({ id: "must_not_hallucinate_jen_identity", passed: !includesAny(allText, [ "jen is", "your manager jen", "your friend jen", "coworker jen", ]), severity: "error", message: "Case 10 should not hallucinate who Jen is.", }) ); checks.push( check({ id: "should_notice_deck_taxes_or_followup", passed: includesAny(allText, ["deck", "taxes", "follow-up", "follow up"]), severity: "warning", message: "Case 10 should extract at least some of the clearer items like taxes, deck, or follow-up.", }) ); checks.push( check({ id: "should_not_force_many_drafts", passed: plan.communication_drafts.length <= 1, severity: "warning", message: "Case 10 should not force lots of draft messages from sparse notes.", }) ); break; } default: break; } return checks; } ``` --- # 3) Integrate case-specific rules into the main heuristic evaluator Now update your general heuristic module so it appends case-specific checks. ## `src/eval/heuristics.ts` Replace it with this version, or merge the highlighted parts. ```ts import type { BrainDumpPlan } from "../lib/planning/schema"; import type { EvalCase } from "./types"; import type { HeuristicCheck, HeuristicEvaluation } from "./heuristicTypes"; import { evaluateCaseSpecificHeuristics } from "./caseSpecificHeuristics"; function normalizeText(text: string) { return text.toLowerCase().trim(); } function containsAny(text: string, phrases: string[]) { const lower = normalizeText(text); return phrases.some((p) => lower.includes(p.toLowerCase())); } function countGenericActions(actions: string[]) { const genericStarts = [ "work on", "handle", "deal with", "make progress on", "figure out", "continue", "review", "address", "fix things", ]; return actions.filter((a) => genericStarts.some((start) => normalizeText(a).startsWith(start)) ).length; } function hasDependencySignals(input: string) { const signals = [ "waiting on", "need feedback from", "still hasn’t", "still hasn't", "owes", "depends on", "legal", "finance", "send revised", "follow up", "ask", "send", "someone needs to confirm", ]; return containsAny(input, signals); } function hasDecisionSignals(input: string) { const signals = [ "not sure whether", "maybe", "should i", "should we", "whether to", "decide", "might", "push the webinar", "cut scope", "stay with friends or get a hotel", "drop one club", "enterprise pilot or focus on self-serve", ]; return containsAny(input, signals); } function hasExplicitDeadlineSignals(input: string) { const signals = [ "today", "tomorrow", "this week", "next friday", "monday", "thursday", "due", "late", "overdue", ]; return containsAny(input, signals); } function looksOverwhelmed(input: string) { const signals = [ "everything feels behind", "i don't even know where to start", "i don’t even know where to start", "too much", "overloaded", "behind", "doing nothing", "apartment is a disaster", "inbox is a mess", ]; return containsAny(input, signals); } function countTotalTimeHorizonItems(plan: BrainDumpPlan) { return ( plan.time_horizon.today.length + plan.time_horizon.this_week.length + plan.time_horizon.later.length ); } function hasLikelyCommunicationNeed(input: string) { const signals = [ "email", "message", "follow up", "follow-up", "ask", "respond", "reply", "send", "ping", "investor update", "candidate", "landlord", "manager", "sam", "finance", "legal", "jordan", "priya", "support", "dev", "alex", ]; return containsAny(input, signals); } function hasEmptyOrThinSummary(plan: BrainDumpPlan) { return plan.summary.trim().length < 30; } function generalCheck(params: { id: string; passed: boolean; severity: "info" | "warning" | "error"; message: string; }): HeuristicCheck { return { ...params, scope: "general", }; } export function evaluateHeuristics( testCase: EvalCase, plan: BrainDumpPlan ): HeuristicEvaluation { const checks: HeuristicCheck[] = []; const input = testCase.brainDump; const genericActionCount = countGenericActions(plan.next_actions); const dependencyHeavy = hasDependencySignals(input); const decisionHeavy = hasDecisionSignals(input); const hasDeadlines = hasExplicitDeadlineSignals(input); const overwhelmed = looksOverwhelmed(input); const communicationLikely = hasLikelyCommunicationNeed(input); checks.push( generalCheck({ id: "summary_present", passed: !hasEmptyOrThinSummary(plan), severity: "error", message: "Summary should be present and meaningful.", }) ); checks.push( generalCheck({ id: "has_priorities", passed: plan.top_priorities.length > 0, severity: "error", message: "Top priorities should not be empty.", }) ); checks.push( generalCheck({ id: "has_next_actions", passed: plan.next_actions.length > 0, severity: "error", message: "Next actions should not be empty.", }) ); checks.push( generalCheck({ id: "today_not_overloaded", passed: plan.time_horizon.today.length <= 5, severity: "warning", message: `Today has ${plan.time_horizon.today.length} items; keep it realistic.`, }) ); if (overwhelmed) { checks.push( generalCheck({ id: "overwhelm_limited_next_actions", passed: plan.next_actions.length <= 3, severity: "warning", message: "Overwhelm-heavy inputs should usually have 3 or fewer next actions.", }) ); checks.push( generalCheck({ id: "overwhelm_limited_today", passed: plan.time_horizon.today.length <= 4, severity: "warning", message: "Overwhelm-heavy inputs should keep today especially short.", }) ); } checks.push( generalCheck({ id: "next_actions_not_too_generic", passed: genericActionCount <= 1, severity: "warning", message: `Detected ${genericActionCount} overly generic next actions.`, }) ); if (dependencyHeavy) { checks.push( generalCheck({ id: "dependency_input_has_blockers", passed: plan.blockers.length > 0, severity: "warning", message: "Dependency-heavy input should usually produce at least one blocker.", }) ); } if (decisionHeavy) { checks.push( generalCheck({ id: "decision_input_has_open_questions", passed: plan.open_questions.length > 0, severity: "warning", message: "Decision-heavy input should usually produce open questions.", }) ); } if (hasDeadlines) { checks.push( generalCheck({ id: "deadline_input_has_near_term_plan", passed: plan.time_horizon.today.length > 0 || plan.time_horizon.this_week.length > 0, severity: "warning", message: "Deadline-sensitive input should produce a near-term plan.", }) ); } if (communicationLikely && dependencyHeavy) { checks.push( generalCheck({ id: "dependency_input_has_draft_or_blocker", passed: plan.communication_drafts.length > 0 || plan.blockers.length > 0, severity: "warning", message: "Dependency-heavy communication input should usually produce a draft or blocker.", }) ); } if (testCase.category === "sparse_ambiguity") { checks.push( generalCheck({ id: "sparse_input_not_overdrafted", passed: plan.communication_drafts.length <= 1, severity: "warning", message: "Sparse ambiguous input should not produce lots of forced drafts.", }) ); } checks.push( generalCheck({ id: "nontrivial_total_output", passed: countTotalTimeHorizonItems(plan) + plan.top_priorities.length >= 4, severity: "warning", message: "Output may be too thin to be useful.", }) ); // Add case-specific checks checks.push(...evaluateCaseSpecificHeuristics(testCase, plan)); const warnings = checks .filter((c) => !c.passed) .map((c) => `[${c.scope ?? "general"}][${c.severity}] ${c.message}`); let score = 100; for (const check of checks) { if (!check.passed) { if (check.severity === "error") score -= 25; if (check.severity === "warning") score -= 10; if (check.severity === "info") score -= 5; } } score = Math.max(0, score); return { score, checks, warnings, }; } ``` --- # 4) No major changes needed in `comparePrompts.ts` Because your heuristic evaluator still returns the same shape, your existing integration should keep working. You already have: ```ts const heuristics = evaluateHeuristics(testCase, resultDebug.plan); ``` That now includes: - general checks - case-specific checks No extra runner changes needed unless you want richer summaries. --- # 5) Optional: add failed check IDs to CSV This is useful for spotting recurring prompt regressions. Update your CSV builder in `comparePrompts.ts` to include a `failedCheckIds` column. ### Add header ```ts "failedCheckIds", ``` ### Add row value ```ts `"${(r.heuristics?.checks ?? []) .filter((c) => !c.passed) .map((c) => c.id) .join(" | ") .replace(/"/g, '""')}"`, ``` So your CSV can highlight patterns like: - `must_surface_enterprise_vs_selfserve` - `must_notice_finance_jordan_or_legal_blockers` - `must_not_hallucinate_jen_identity` That’s very useful. --- # 6) Optional: add special winner summary by case In the Markdown report, you can add a section like: ```ts lines.push(""); lines.push("## Heuristic Winner By Case"); lines.push(""); for (const c of summary.cases) { const successful = c.results.filter((r) => r.success && r.heuristics); if (successful.length === 0) { lines.push(`- ${c.caseId} - ${c.caseName}: no successful outputs`); continue; } const best = successful.sort( (a, b) => (b.heuristics?.score ?? 0) - (a.heuristics?.score ?? 0) )[0]; lines.push( `- ${c.caseId} - ${c.caseName}: ${best.promptVersion} (${best.heuristics?.score})` ); } ``` Still just a hint — but a useful one. --- # 7) What these new checks catch well ## Case 01 - losing Sam/pricing copy - missing webinar decision - over-prioritizing dentist - missing Sam draft ## Case 03 - missing the core enterprise vs self-serve tension - forgetting investor update ## Case 06 - missing finance/legal/Jordan blockers - not prioritizing controllable near-term work - missing useful follow-up drafts ## Case 10 - hallucinating who Jen is - pretending confidence where none exists - forcing too many drafts That’s exactly the kind of regression suite you want. --- # 8) A caution on brittleness Case-specific checks are powerful, but they can get too rigid if overdone. Example: - good output says “pricing content” instead of “pricing copy” - heuristic fails if phrasing is too narrow So keep them: - **concept-based** - not too exact-string dependent - tolerant of reasonable variation The versions above try to stay on the safe side. --- # 9) Best way to use case-specific heuristics I’d use them in this order: ## First pass Look for: - low heuristic scores - failed `error` checks - repeated failures across prompt versions ## Second pass Manually inspect only those flagged cases. ## Third pass Use golden outputs to decide if the heuristic is correct or too brittle. That keeps automation helpful without letting it overrule judgment. --- # 10) My recommendation This is enough to make your eval loop much sharper. The next best step after this is likely one of: 1. **add multi-run comparisons to reduce output randomness** 2. **generate golden outputs for the remaining 5 test cases** 3. **build an HTML diff report** 4. **add model comparison support, not just prompt comparison** My recommendation: **1 next**. Why: - prompt quality comparisons can be noisy with one run - multi-run evaluation gives you much more trustworthy signal - once heuristics exist, averaging across runs becomes really useful If you want, I can do **1 next** and give you: - `n` runs per prompt per case - aggregated heuristic averages - per-case variance summary - a cleaner comparison report that’s more robust to model randomness

Yes — **1 next** is the right move. At this point you already have: - per-case comparisons - heuristics - case-specific checks - multi-run averages What you still need is the **executive summary**: > “Which prompt is better overall?” So let’s add **overall prompt-level aggregates across all cases**. This will give you: - overall success rate per prompt - overall average heuristic score - overall worst/best heuristic score - total case wins - recurring failed checks across all cases - a cleaner top-line summary for decision-making --- # Goal Extend the multi-run comparison so it also computes: - **per-prompt rollups across all cases** - **case win counts** - **top recurring failed checks** - **top recurring warnings** We’ll do this with minimal disruption. --- # 1) Update multi-run types ## `src/eval/multiRunTypes.ts` Replace with this expanded version: ```ts import type { HeuristicEvaluation } from "./heuristicTypes"; export type PromptRunResult = { runIndex: number; promptVersion: string; success: boolean; durationMs: number; error?: string; output?: unknown; debug?: { strictValidationPassed: boolean; normalizationUsed: boolean; outputText: string; parsedStructured: unknown | null; systemPrompt: string; userPrompt: string; }; heuristics?: HeuristicEvaluation; }; export type PromptCasePromptAggregate = { promptVersion: string; totalRuns: number; successCount: number; failureCount: number; strictValidationPassCount: number; normalizationUsedCount: number; avgDurationMs: number; avgHeuristicScore: number; minHeuristicScore: number | null; maxHeuristicScore: number | null; failedCheckCounts: Record<string, number>; warningCounts: Record<string, number>; runs: PromptRunResult[]; }; export type MultiRunCaseComparison = { caseId: string; caseName: string; category: string; brainDump: string; promptAggregates: PromptCasePromptAggregate[]; }; export type OverallPromptAggregate = { promptVersion: string; totalCases: number; totalRuns: number; successCount: number; failureCount: number; successRate: number; caseWinCount: number; tiedCaseCount: number; strictValidationPassCount: number; normalizationUsedCount: number; avgDurationMs: number; avgHeuristicScore: number; minHeuristicScore: number | null; maxHeuristicScore: number | null; failedCheckCounts: Record<string, number>; warningCounts: Record<string, number>; }; export type MultiRunComparisonSummary = { model: string; runCount: number; comparedPromptVersions: string[]; createdAt: string; totalCases: number; cases: MultiRunCaseComparison[]; overallPromptAggregates: OverallPromptAggregate[]; }; ``` --- # 2) Add overall aggregation helpers ## `src/eval/aggregateOverall.ts` ```ts import type { MultiRunCaseComparison, OverallPromptAggregate, PromptCasePromptAggregate, } from "./multiRunTypes"; function average(nums: number[]) { if (nums.length === 0) return 0; return nums.reduce((sum, n) => sum + n, 0) / nums.length; } function mergeCountMaps(maps: Record<string, number>[]) { const merged: Record<string, number> = {}; for (const map of maps) { for (const [key, value] of Object.entries(map)) { merged[key] = (merged[key] ?? 0) + value; } } return merged; } function getCaseWinners( caseComparison: MultiRunCaseComparison ): { winners: string[]; topScore: number } { const sorted = [...caseComparison.promptAggregates].sort( (a, b) => b.avgHeuristicScore - a.avgHeuristicScore ); if (sorted.length === 0) { return { winners: [], topScore: 0 }; } const topScore = sorted[0].avgHeuristicScore; const winners = sorted .filter((agg) => agg.avgHeuristicScore === topScore) .map((agg) => agg.promptVersion); return { winners, topScore }; } export function aggregateOverallPromptResults( cases: MultiRunCaseComparison[], promptVersions: string[] ): OverallPromptAggregate[] { return promptVersions.map((promptVersion) => { const aggregatesForPrompt: PromptCasePromptAggregate[] = cases .map((c) => c.promptAggregates.find((agg) => agg.promptVersion === promptVersion) ) .filter((agg): agg is PromptCasePromptAggregate => Boolean(agg)); const totalCases = aggregatesForPrompt.length; const totalRuns = aggregatesForPrompt.reduce((sum, agg) => sum + agg.totalRuns, 0); const successCount = aggregatesForPrompt.reduce( (sum, agg) => sum + agg.successCount, 0 ); const failureCount = aggregatesForPrompt.reduce( (sum, agg) => sum + agg.failureCount, 0 ); const strictValidationPassCount = aggregatesForPrompt.reduce( (sum, agg) => sum + agg.strictValidationPassCount, 0 ); const normalizationUsedCount = aggregatesForPrompt.reduce( (sum, agg) => sum + agg.normalizationUsedCount, 0 ); const heuristicScores = aggregatesForPrompt .flatMap((agg) => agg.runs .filter((run) => run.success && typeof run.heuristics?.score === "number") .map((run) => run.heuristics!.score) ); const durations = aggregatesForPrompt.flatMap((agg) => agg.runs.map((run) => run.durationMs) ); let caseWinCount = 0; let tiedCaseCount = 0; for (const caseComparison of cases) { const { winners } = getCaseWinners(caseComparison); if (winners.includes(promptVersion)) { if (winners.length === 1) { caseWinCount += 1; } else { tiedCaseCount += 1; } } } return { promptVersion, totalCases, totalRuns, successCount, failureCount, successRate: totalRuns > 0 ? successCount / totalRuns : 0, caseWinCount, tiedCaseCount, strictValidationPassCount, normalizationUsedCount, avgDurationMs: average(durations), avgHeuristicScore: average(heuristicScores), minHeuristicScore: heuristicScores.length ? Math.min(...heuristicScores) : null, maxHeuristicScore: heuristicScores.length ? Math.max(...heuristicScores) : null, failedCheckCounts: mergeCountMaps( aggregatesForPrompt.map((agg) => agg.failedCheckCounts) ), warningCounts: mergeCountMaps( aggregatesForPrompt.map((agg) => agg.warningCounts) ), }; }); } ``` --- # 3) Update the multi-run runner Now wire this into your summary generation. ## `src/eval/comparePromptsMultiRun.ts` ### Add import ```ts import { aggregateOverallPromptResults } from "./aggregateOverall"; ``` ### Update summary creation Replace this section: ```ts const summary: MultiRunComparisonSummary = { model: MODEL, runCount: RUN_COUNT, comparedPromptVersions: PROMPTS_TO_COMPARE.map((p) => p.version), createdAt: new Date().toISOString(), totalCases: evalCases.length, cases: caseComparisons, }; ``` with: ```ts const comparedPromptVersions = PROMPTS_TO_COMPARE.map((p) => p.version); const summary: MultiRunComparisonSummary = { model: MODEL, runCount: RUN_COUNT, comparedPromptVersions, createdAt: new Date().toISOString(), totalCases: evalCases.length, cases: caseComparisons, overallPromptAggregates: aggregateOverallPromptResults( caseComparisons, comparedPromptVersions ), }; ``` --- # 4) Improve the Markdown summary Replace `buildMarkdownSummary()` with this version: ```ts function buildMarkdownSummary(summary: MultiRunComparisonSummary) { const lines: string[] = []; lines.push("# Multi-Run Prompt Comparison Summary"); lines.push(""); lines.push(`- Model: ${summary.model}`); lines.push(`- Prompt versions: ${summary.comparedPromptVersions.join(", ")}`); lines.push(`- Runs per prompt per case: ${summary.runCount}`); lines.push(`- Total cases: ${summary.totalCases}`); lines.push(`- Created at: ${summary.createdAt}`); lines.push(""); lines.push("## Overall Prompt Summary"); lines.push(""); lines.push( "| Prompt | Success Rate | Avg Heuristic | Min | Max | Avg Duration | Case Wins | Ties | Strict Passes | Normalized |" ); lines.push("|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|"); for (const overall of summary.overallPromptAggregates) { lines.push( `| ${overall.promptVersion} | ${(overall.successRate * 100).toFixed(1)}% (${overall.successCount}/${overall.totalRuns}) | ${overall.avgHeuristicScore.toFixed(1)} | ${overall.minHeuristicScore ?? ""} | ${overall.maxHeuristicScore ?? ""} | ${Math.round(overall.avgDurationMs)}ms | ${overall.caseWinCount} | ${overall.tiedCaseCount} | ${overall.strictValidationPassCount} | ${overall.normalizationUsedCount} |` ); } lines.push(""); lines.push("## Overall Frequent Failed Checks"); lines.push(""); for (const overall of summary.overallPromptAggregates) { lines.push(`### ${overall.promptVersion}`); lines.push(""); const failedEntries = Object.entries(overall.failedCheckCounts).sort( (a, b) => b[1] - a[1] ); if (failedEntries.length === 0) { lines.push("- No repeated failed checks"); } else { for (const [checkId, count] of failedEntries.slice(0, 10)) { lines.push(`- ${checkId}: ${count}`); } } lines.push(""); } lines.push("## Aggregate Overview by Case"); lines.push(""); lines.push( "| Case | Prompt | Successes | Avg Heuristic | Min | Max | Avg Duration | Strict Passes | Normalized |" ); lines.push("|---|---|---:|---:|---:|---:|---:|---:|---:|"); for (const c of summary.cases) { for (const agg of c.promptAggregates) { lines.push( `| ${c.caseId} - ${c.caseName} | ${agg.promptVersion} | ${agg.successCount}/${agg.totalRuns} | ${agg.avgHeuristicScore.toFixed(1)} | ${agg.minHeuristicScore ?? ""} | ${agg.maxHeuristicScore ?? ""} | ${Math.round(agg.avgDurationMs)}ms | ${agg.strictValidationPassCount} | ${agg.normalizationUsedCount} |` ); } } lines.push(""); lines.push("## Heuristic Winner by Case"); lines.push(""); for (const c of summary.cases) { const sorted = [...c.promptAggregates].sort( (a, b) => b.avgHeuristicScore - a.avgHeuristicScore ); if (sorted.length === 0) { lines.push(`- ${c.caseId} - ${c.caseName}: no results`); continue; } const topScore = sorted[0].avgHeuristicScore; const winners = sorted.filter((agg) => agg.avgHeuristicScore === topScore); if (winners.length === 1) { lines.push( `- ${c.caseId} - ${c.caseName}: ${winners[0].promptVersion} (avg ${winners[0].avgHeuristicScore.toFixed(1)})` ); } else { lines.push( `- ${c.caseId} - ${c.caseName}: tie (${winners .map((w) => w.promptVersion) .join(", ")}) at avg ${topScore.toFixed(1)}` ); } } lines.push(""); lines.push("## Frequent Failed Checks by Case"); lines.push(""); for (const c of summary.cases) { lines.push(`### ${c.caseId} - ${c.caseName}`); lines.push(""); for (const agg of c.promptAggregates) { const failedEntries = Object.entries(agg.failedCheckCounts).sort( (a, b) => b[1] - a[1] ); lines.push(`- **${agg.promptVersion}**`); if (failedEntries.length === 0) { lines.push(" - No repeated failed checks"); continue; } for (const [checkId, count] of failedEntries.slice(0, 8)) { lines.push(` - ${checkId}: ${count}/${agg.totalRuns}`); } } lines.push(""); } return lines.join("\n"); } ``` --- # 5) Improve the CSV export Replace `buildScoringCsv()` with this richer version: ```ts function buildScoringCsv(summary: MultiRunComparisonSummary) { const headers = [ "scope", "caseId", "caseName", "category", "model", "promptVersion", "runCount", "successCount", "failureCount", "successRate", "avgHeuristicScore", "minHeuristicScore", "maxHeuristicScore", "avgDurationMs", "caseWinCount", "tiedCaseCount", "strictValidationPassCount", "normalizationUsedCount", "topFailedCheckIds", "notes", ]; const rows: string[] = []; for (const overall of summary.overallPromptAggregates) { const topFailedChecks = Object.entries(overall.failedCheckCounts) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .map(([id, count]) => `${id}:${count}`) .join(" | "); rows.push( [ "overall", "", '""', "", summary.model, overall.promptVersion, overall.totalRuns, overall.successCount, overall.failureCount, overall.successRate.toFixed(3), overall.avgHeuristicScore.toFixed(1), overall.minHeuristicScore ?? "", overall.maxHeuristicScore ?? "", Math.round(overall.avgDurationMs), overall.caseWinCount, overall.tiedCaseCount, overall.strictValidationPassCount, overall.normalizationUsedCount, `"${topFailedChecks.replace(/"/g, '""')}"`, '""', ].join(",") ); } for (const c of summary.cases) { for (const agg of c.promptAggregates) { const topFailedChecks = Object.entries(agg.failedCheckCounts) .sort((a, b) => b[1] - a[1]) .slice(0, 8) .map(([id, count]) => `${id}:${count}/${agg.totalRuns}`) .join(" | "); rows.push( [ "case", c.caseId, `"${c.caseName}"`, c.category, summary.model, agg.promptVersion, agg.totalRuns, agg.successCount, agg.failureCount, (agg.totalRuns > 0 ? agg.successCount / agg.totalRuns : 0).toFixed(3), agg.avgHeuristicScore.toFixed(1), agg.minHeuristicScore ?? "", agg.maxHeuristicScore ?? "", Math.round(agg.avgDurationMs), "", "", agg.strictValidationPassCount, agg.normalizationUsedCount, `"${topFailedChecks.replace(/"/g, '""')}"`, '""', ].join(",") ); } } return [headers.join(","), ...rows].join("\n"); } ``` --- # 6) What this gives you Now your multi-run output will answer: ## Which prompt wins more cases? From `caseWinCount`. ## Which prompt is more reliable overall? From: - `successRate` - `minHeuristicScore` - `strictValidationPassCount` ## Which prompt is stronger on average? From `avgHeuristicScore`. ## What keeps going wrong across the whole suite? From `overall.failedCheckCounts`. That’s a much better decision surface. --- # 7) How to interpret the overall summary Use this order: ## 1. Success rate If one prompt structurally fails more often, that matters first. ## 2. Average heuristic score Good default overall signal. ## 3. Case wins Tells you breadth, not just mean score. ## 4. Min heuristic score Important for tail-risk / bad-run behavior. ## 5. Failed-check leaderboard Best clue for what to fix next. Example interpretation: - **V2 avg score 86 vs V1 74** - **V2 wins 7/10 cases** - **V2 min score 70 vs V1 min 45** - repeated V1 failures: `must_surface_enterprise_vs_selfserve`, `dependency_input_has_blockers` That’s strong evidence V2 is better. --- # 8) Nice next tweak: weighted case wins Right now every case counts equally. Later you may want to weight cases differently, for example: - overwhelm-heavy inputs matter more - dependency-heavy manager cases matter more - sparse ambiguity matters less initially But I would **not** do that yet. Equal weights are cleaner for now. --- # 9) Nice next tweak: overall standard deviation A strong next addition would be: - heuristic score stddev per prompt overall - heuristic score stddev per case/prompt That helps detect: - prompts that look good on average - but are inconsistent in practice Again: useful, but not the next must-have. --- # 10) Recommended run cycle now With this in place, a strong evaluation loop is: ```bash EVAL_RUN_COUNT=3 npm run eval:compare:multi ``` Then inspect: 1. `multirun-summary.md` 2. top overall prompt table 3. case wins 4. failed-check leaderboard 5. a few individual case JSON files where scores differ a lot That gives you a real prompt iteration workflow. --- # 11) My recommendation This is enough to choose between V1 and V2 with much more confidence. The best next steps from here are probably: 1. **add heuristic score variance / stddev** 2. **generate golden outputs for the remaining 5 cases** 3. **build an HTML report for easier review** 4. **add model-vs-model comparison using the same framework** My recommendation: **1 next** if you want stronger reliability analysis, or **4 next** if you want to compare models as well as prompts. If you want, I can do **1 next** and add: - per-case/prompt heuristic stddev - overall prompt stddev - a simple “stability” metric in the summary tables

Yes — **1 next** is the right move. At this point your evaluation framework is strong enough that **model choice** becomes a real lever, not a guess. You can now ask: - is `gpt-4.1-mini` already good enough? - does a stronger model improve priority judgment? - does a cheaper/faster model hold up on your golden cases? - is prompt tuning doing more work than model selection? So below is a clean extension for **model-vs-model comparison** using your existing multi-run + heuristic + stability framework. --- # Goal Compare: - multiple **models** - with the same **prompt(s)** - across the same **eval cases** - with the same **run count** - while tracking: - success rate - heuristic averages - heuristic stddev / stability - case wins - recurring failed checks This gives you a much better decision surface for: - cost vs quality - speed vs quality - prompt portability across models --- # Recommended design Keep this separate from prompt comparison. You now have: - prompt comparison runner - multi-run prompt comparison runner Add: - **multi-run model comparison runner** That keeps analysis clean. --- # What we’ll add 1. model comparison types 2. service support for model override 3. aggregation helpers for model runs 4. `compareModelsMultiRun.ts` 5. Markdown/CSV summaries --- # 1) Add model override support in the planner service Your current planner service reads the model from env. To compare models directly, let it accept an optional override. ## `src/lib/planning/service.ts` Update the option types: ```ts type GenerateOptions = { brainDump: string; mode?: "work" | "personal" | "mixed"; style?: "gentle" | "direct" | "executive"; depth?: "quick" | "standard" | "detailed"; modelOverride?: string; }; type GenerateDebugOptions = GenerateOptions & { promptConfig?: PromptConfig; }; ``` Then update `callStructuredModel()`: ```ts async function callStructuredModel(args: { system: string; user: string; modelOverride?: string; }): Promise<StructuredModelCallResult> { const model = args.modelOverride || process.env.OPENAI_MODEL || "gpt-4.1-mini"; const response = await client.responses.create({ model, input: [ { role: "system", content: args.system, }, { role: "user", content: args.user, }, ], text: { format: { type: "json_schema", name: brainDumpPlanJsonSchema.name, schema: brainDumpPlanJsonSchema.schema, strict: true, }, }, }); const outputText = response.output_text ?? ""; const parsedStructured = outputText.trim() ? tryParseJson(outputText) : null; return { rawResponse: response, outputText, parsedStructured, model, }; } ``` And pass the override through inside `generateBrainDumpPlanDebug()`: ```ts const model = options.modelOverride || process.env.OPENAI_MODEL || "gpt-4.1-mini"; ``` Then in the call: ```ts const structuredResult = await callStructuredModel({ system: systemPrompt, user: userPrompt, modelOverride: options.modelOverride, }); ``` That’s the only service change you need. --- # 2) Add model comparison types ## `src/eval/modelCompareTypes.ts` ```ts import type { HeuristicEvaluation } from "./heuristicTypes"; export type ModelRunResult = { runIndex: number; model: string; promptVersion: string; success: boolean; durationMs: number; error?: string; output?: unknown; debug?: { strictValidationPassed: boolean; normalizationUsed: boolean; outputText: string; parsedStructured: unknown | null; systemPrompt: string; userPrompt: string; }; heuristics?: HeuristicEvaluation; }; export type ModelCaseAggregate = { model: string; promptVersion: string; totalRuns: number; successCount: number; failureCount: number; strictValidationPassCount: number; normalizationUsedCount: number; avgDurationMs: number; avgHeuristicScore: number; heuristicStdDev: number; stabilityLabel: "high" | "medium" | "low"; minHeuristicScore: number | null; maxHeuristicScore: number | null; failedCheckCounts: Record<string, number>; warningCounts: Record<string, number>; runs: ModelRunResult[]; }; export type ModelCaseComparison = { caseId: string; caseName: string; category: string; brainDump: string; aggregates: ModelCaseAggregate[]; }; export type OverallModelAggregate = { model: string; promptVersion: string; totalCases: number; totalRuns: number; successCount: number; failureCount: number; successRate: number; caseWinCount: number; tiedCaseCount: number; strictValidationPassCount: number; normalizationUsedCount: number; avgDurationMs: number; avgHeuristicScore: number; heuristicStdDev: number; stabilityLabel: "high" | "medium" | "low"; minHeuristicScore: number | null; maxHeuristicScore: number | null; failedCheckCounts: Record<string, number>; warningCounts: Record<string, number>; }; export type MultiRunModelComparisonSummary = { promptVersion: string; runCount: number; comparedModels: string[]; createdAt: string; totalCases: number; cases: ModelCaseComparison[]; overallModelAggregates: OverallModelAggregate[]; }; ``` --- # 3) Add model aggregation helpers ## `src/eval/aggregateModels.ts` ```ts import type { ModelRunResult, ModelCaseAggregate, ModelCaseComparison, OverallModelAggregate, } from "./modelCompareTypes"; import { average, stddev, stabilityLabelFromStdDev } from "./stats"; function countBy(items: string[]) { const map: Record<string, number> = {}; for (const item of items) { map[item] = (map[item] ?? 0) + 1; } return map; } function mergeCountMaps(maps: Record<string, number>[]) { const merged: Record<string, number> = {}; for (const map of maps) { for (const [key, value] of Object.entries(map)) { merged[key] = (merged[key] ?? 0) + value; } } return merged; } export function aggregateModelRuns( model: string, promptVersion: string, runs: ModelRunResult[] ): ModelCaseAggregate { const successRuns = runs.filter((r) => r.success); const scores = successRuns .map((r) => r.heuristics?.score) .filter((v): v is number => typeof v === "number"); const failedCheckIds = successRuns.flatMap((r) => (r.heuristics?.checks ?? []).filter((c) => !c.passed).map((c) => c.id) ); const warnings = successRuns.flatMap((r) => r.heuristics?.warnings ?? []); const heuristicStdDev = stddev(scores); return { model, promptVersion, totalRuns: runs.length, successCount: successRuns.length, failureCount: runs.length - successRuns.length, strictValidationPassCount: successRuns.filter( (r) => r.debug?.strictValidationPassed ).length, normalizationUsedCount: successRuns.filter( (r) => r.debug?.normalizationUsed ).length, avgDurationMs: average(runs.map((r) => r.durationMs)), avgHeuristicScore: average(scores), heuristicStdDev, stabilityLabel: stabilityLabelFromStdDev(heuristicStdDev), minHeuristicScore: scores.length ? Math.min(...scores) : null, maxHeuristicScore: scores.length ? Math.max(...scores) : null, failedCheckCounts: countBy(failedCheckIds), warningCounts: countBy(warnings), runs, }; } function getCaseWinners(caseComparison: ModelCaseComparison) { const sorted = [...caseComparison.aggregates].sort( (a, b) => b.avgHeuristicScore - a.avgHeuristicScore ); if (sorted.length === 0) { return { winners: [], topScore: 0 }; } const topScore = sorted[0].avgHeuristicScore; const winners = sorted .filter((agg) => agg.avgHeuristicScore === topScore) .map((agg) => agg.model); return { winners, topScore }; } export function aggregateOverallModelResults( cases: ModelCaseComparison[], models: string[], promptVersion: string ): OverallModelAggregate[] { return models.map((model) => { const aggregatesForModel = cases .map((c) => c.aggregates.find((agg) => agg.model === model)) .filter((agg): agg is ModelCaseAggregate => Boolean(agg)); const totalCases = aggregatesForModel.length; const totalRuns = aggregatesForModel.reduce((sum, agg) => sum + agg.totalRuns, 0); const successCount = aggregatesForModel.reduce( (sum, agg) => sum + agg.successCount, 0 ); const failureCount = aggregatesForModel.reduce( (sum, agg) => sum + agg.failureCount, 0 ); const strictValidationPassCount = aggregatesForModel.reduce( (sum, agg) => sum + agg.strictValidationPassCount, 0 ); const normalizationUsedCount = aggregatesForModel.reduce( (sum, agg) => sum + agg.normalizationUsedCount, 0 ); const scores = aggregatesForModel.flatMap((agg) => agg.runs .filter((run) => run.success && typeof run.heuristics?.score === "number") .map((run) => run.heuristics!.score) ); const durations = aggregatesForModel.flatMap((agg) => agg.runs.map((run) => run.durationMs) ); let caseWinCount = 0; let tiedCaseCount = 0; for (const caseComparison of cases) { const { winners } = getCaseWinners(caseComparison); if (winners.includes(model)) { if (winners.length === 1) caseWinCount += 1; else tiedCaseCount += 1; } } const heuristicStdDev = stddev(scores); return { model, promptVersion, totalCases, totalRuns, successCount, failureCount, successRate: totalRuns > 0 ? successCount / totalRuns : 0, caseWinCount, tiedCaseCount, strictValidationPassCount, normalizationUsedCount, avgDurationMs: average(durations), avgHeuristicScore: average(scores), heuristicStdDev, stabilityLabel: stabilityLabelFromStdDev(heuristicStdDev), minHeuristicScore: scores.length ? Math.min(...scores) : null, maxHeuristicScore: scores.length ? Math.max(...scores) : null, failedCheckCounts: mergeCountMaps( aggregatesForModel.map((agg) => agg.failedCheckCounts) ), warningCounts: mergeCountMaps( aggregatesForModel.map((agg) => agg.warningCounts) ), }; }); } ``` --- # 4) Build the multi-run model comparison runner ## `src/eval/compareModelsMultiRun.ts` ```ts import path from "node:path"; import { evalCases } from "./cases"; import { ensureDir, timestamp, writeJsonFile, writeTextFile } from "./utils"; import { generateBrainDumpPlanDebug } from "../lib/planning/service"; import { PROMPT_V2 } from "../lib/planning/prompts"; import { evaluateHeuristics } from "./heuristics"; import { aggregateModelRuns, aggregateOverallModelResults, } from "./aggregateModels"; import type { ModelRunResult, ModelCaseComparison, MultiRunModelComparisonSummary, } from "./modelCompareTypes"; const RUN_COUNT = Number(process.env.EVAL_RUN_COUNT || "3"); const OUTPUT_DIR = path.join( process.cwd(), "eval-model-comparisons", timestamp() ); const MODELS_TO_COMPARE = ( process.env.EVAL_MODELS || "gpt-4.1-mini,gpt-4.1" ) .split(",") .map((s) => s.trim()) .filter(Boolean); async function run() { console.log( `Running model comparison [${MODELS_TO_COMPARE.join(", ")}] with prompt=${PROMPT_V2.version}, runs=${RUN_COUNT}` ); await ensureDir(OUTPUT_DIR); const caseComparisons: ModelCaseComparison[] = []; for (const testCase of evalCases) { console.log(`\n→ ${testCase.id}: ${testCase.name}`); const modelRunsMap = new Map<string, ModelRunResult[]>(); for (const model of MODELS_TO_COMPARE) { modelRunsMap.set(model, []); for (let runIndex = 1; runIndex <= RUN_COUNT; runIndex++) { const started = Date.now(); try { const resultDebug = await generateBrainDumpPlanDebug({ brainDump: testCase.brainDump, mode: "mixed", style: "direct", depth: "standard", promptConfig: PROMPT_V2, modelOverride: model, }); const durationMs = Date.now() - started; const heuristics = evaluateHeuristics(testCase, resultDebug.plan); const runResult: ModelRunResult = { runIndex, model, promptVersion: PROMPT_V2.version, success: true, durationMs, output: resultDebug.plan, debug: { strictValidationPassed: resultDebug.debug.strictValidationPassed, normalizationUsed: resultDebug.debug.normalizationUsed, outputText: resultDebug.debug.outputText, parsedStructured: resultDebug.debug.parsedStructured, systemPrompt: resultDebug.debug.systemPrompt, userPrompt: resultDebug.debug.userPrompt, }, heuristics, }; modelRunsMap.get(model)?.push(runResult); console.log( ` ${model} run ${runIndex}/${RUN_COUNT}: success (${durationMs}ms) heuristic=${heuristics.score}` ); } catch (error) { const durationMs = Date.now() - started; const runResult: ModelRunResult = { runIndex, model, promptVersion: PROMPT_V2.version, success: false, durationMs, error: error instanceof Error ? error.message : "Unknown error", }; modelRunsMap.get(model)?.push(runResult); console.log( ` ${model} run ${runIndex}/${RUN_COUNT}: failed (${durationMs}ms)` ); } } } const aggregates = MODELS_TO_COMPARE.map((model) => aggregateModelRuns( model, PROMPT_V2.version, modelRunsMap.get(model) ?? [] ) ); const caseComparison: ModelCaseComparison = { caseId: testCase.id, caseName: testCase.name, category: testCase.category, brainDump: testCase.brainDump, aggregates, }; caseComparisons.push(caseComparison); await writeJsonFile( path.join(OUTPUT_DIR, `${testCase.id}.models.json`), caseComparison ); } const summary: MultiRunModelComparisonSummary = { promptVersion: PROMPT_V2.version, runCount: RUN_COUNT, comparedModels: MODELS_TO_COMPARE, createdAt: new Date().toISOString(), totalCases: evalCases.length, cases: caseComparisons, overallModelAggregates: aggregateOverallModelResults( caseComparisons, MODELS_TO_COMPARE, PROMPT_V2.version ), }; await writeJsonFile(path.join(OUTPUT_DIR, "model-summary.json"), summary); await writeTextFile( path.join(OUTPUT_DIR, "model-summary.md"), buildMarkdownSummary(summary) ); await writeTextFile( path.join(OUTPUT_DIR, "model-scoring.csv"), buildScoringCsv(summary) ); console.log(`\nDone. Model comparison written to:\n${OUTPUT_DIR}`); } function buildMarkdownSummary(summary: MultiRunModelComparisonSummary) { const lines: string[] = []; lines.push("# Multi-Run Model Comparison Summary"); lines.push(""); lines.push(`- Prompt version: ${summary.promptVersion}`); lines.push(`- Models: ${summary.comparedModels.join(", ")}`); lines.push(`- Runs per model per case: ${summary.runCount}`); lines.push(`- Total cases: ${summary.totalCases}`); lines.push(`- Created at: ${summary.createdAt}`); lines.push(""); lines.push("## Overall Model Summary"); lines.push(""); lines.push( "| Model | Success Rate | Avg Score | StdDev | Stability | Min | Max | Avg Duration | Case Wins | Ties | Strict Passes | Normalized |" ); lines.push("|---|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|"); for (const overall of summary.overallModelAggregates) { lines.push( `| ${overall.model} | ${(overall.successRate * 100).toFixed(1)}% (${overall.successCount}/${overall.totalRuns}) | ${overall.avgHeuristicScore.toFixed(1)} | ${overall.heuristicStdDev.toFixed(1)} | ${overall.stabilityLabel} | ${overall.minHeuristicScore ?? ""} | ${overall.maxHeuristicScore ?? ""} | ${Math.round(overall.avgDurationMs)}ms | ${overall.caseWinCount} | ${overall.tiedCaseCount} | ${overall.strictValidationPassCount} | ${overall.normalizationUsedCount} |` ); } lines.push(""); lines.push("## Overall Frequent Failed Checks"); lines.push(""); for (const overall of summary.overallModelAggregates) { lines.push(`### ${overall.model}`); lines.push(""); const failedEntries = Object.entries(overall.failedCheckCounts).sort( (a, b) => b[1] - a[1] ); if (failedEntries.length === 0) { lines.push("- No repeated failed checks"); } else { for (const [checkId, count] of failedEntries.slice(0, 10)) { lines.push(`- ${checkId}: ${count}`); } } lines.push(""); } lines.push("## Aggregate Overview by Case"); lines.push(""); lines.push( "| Case | Model | Successes | Avg Score | StdDev | Stability | Min | Max | Avg Duration | Strict Passes | Normalized |" ); lines.push("|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|"); for (const c of summary.cases) { for (const agg of c.aggregates) { lines.push( `| ${c.caseId} - ${c.caseName} | ${agg.model} | ${agg.successCount}/${agg.totalRuns} | ${agg.avgHeuristicScore.toFixed(1)} | ${agg.heuristicStdDev.toFixed(1)} | ${agg.stabilityLabel} | ${agg.minHeuristicScore ?? ""} | ${agg.maxHeuristicScore ?? ""} | ${Math.round(agg.avgDurationMs)}ms | ${agg.strictValidationPassCount} | ${agg.normalizationUsedCount} |` ); } } lines.push(""); lines.push("## Model Winner by Case"); lines.push(""); for (const c of summary.cases) { const sorted = [...c.aggregates].sort( (a, b) => b.avgHeuristicScore - a.avgHeuristicScore ); if (sorted.length === 0) { lines.push(`- ${c.caseId} - ${c.caseName}: no results`); continue; } const topScore = sorted[0].avgHeuristicScore; const winners = sorted.filter((agg) => agg.avgHeuristicScore === topScore); if (winners.length === 1) { lines.push( `- ${c.caseId} - ${c.caseName}: ${winners[0].model} (avg ${winners[0].avgHeuristicScore.toFixed(1)}, stddev ${winners[0].heuristicStdDev.toFixed(1)})` ); } else { lines.push( `- ${c.caseId} - ${c.caseName}: tie (${winners .map((w) => w.model) .join(", ")}) at avg ${topScore.toFixed(1)}` ); } } return lines.join("\n"); } function buildScoringCsv(summary: MultiRunModelComparisonSummary) { const headers = [ "scope", "caseId", "caseName", "category", "promptVersion", "model", "runCount", "successCount", "failureCount", "successRate", "avgHeuristicScore", "heuristicStdDev", "stabilityLabel", "minHeuristicScore", "maxHeuristicScore", "avgDurationMs", "caseWinCount", "tiedCaseCount", "strictValidationPassCount", "normalizationUsedCount", "topFailedCheckIds", "notes", ]; const rows: string[] = []; for (const overall of summary.overallModelAggregates) { const topFailedChecks = Object.entries(overall.failedCheckCounts) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .map(([id, count]) => `${id}:${count}`) .join(" | "); rows.push( [ "overall", "", '""', "", summary.promptVersion, overall.model, overall.totalRuns, overall.successCount, overall.failureCount, overall.successRate.toFixed(3), overall.avgHeuristicScore.toFixed(1), overall.heuristicStdDev.toFixed(1), overall.stabilityLabel, overall.minHeuristicScore ?? "", overall.maxHeuristicScore ?? "", Math.round(overall.avgDurationMs), overall.caseWinCount, overall.tiedCaseCount, overall.strictValidationPassCount, overall.normalizationUsedCount, `"${topFailedChecks.replace(/"/g, '""')}"`, '""', ].join(",") ); } for (const c of summary.cases) { for (const agg of c.aggregates) { const topFailedChecks = Object.entries(agg.failedCheckCounts) .sort((a, b) => b[1] - a[1]) .slice(0, 8) .map(([id, count]) => `${id}:${count}/${agg.totalRuns}`) .join(" | "); rows.push( [ "case", c.caseId, `"${c.caseName}"`, c.category, summary.promptVersion, agg.model, agg.totalRuns, agg.successCount, agg.failureCount, (agg.totalRuns > 0 ? agg.successCount / agg.totalRuns : 0).toFixed(3), agg.avgHeuristicScore.toFixed(1), agg.heuristicStdDev.toFixed(1), agg.stabilityLabel, agg.minHeuristicScore ?? "", agg.maxHeuristicScore ?? "", Math.round(agg.avgDurationMs), "", "", agg.strictValidationPassCount, agg.normalizationUsedCount, `"${topFailedChecks.replace(/"/g, '""')}"`, '""', ].join(",") ); } } return [headers.join(","), ...rows].join("\n"); } run().catch((error) => { console.error("Multi-run model comparison crashed:", error); process.exit(1); }); ``` --- # 5) Add npm script In `package.json`: ```json { "scripts": { "dev": "next dev", "eval": "tsx src/eval/runEval.ts", "eval:compare": "tsx src/eval/comparePrompts.ts", "eval:compare:multi": "tsx src/eval/comparePromptsMultiRun.ts", "eval:models:multi": "tsx src/eval/compareModelsMultiRun.ts" } } ``` --- # 6) How to run it Example: ```bash EVAL_RUN_COUNT=3 EVAL_MODELS=gpt-4.1-mini,gpt-4.1 npm run eval:models:multi ``` Or try cheaper/faster variants too: ```bash EVAL_RUN_COUNT=3 EVAL_MODELS=gpt-4.1-mini,gpt-4o-mini npm run eval:models:multi ``` --- # 7) What this tells you Now you can compare models on: ## Quality - average heuristic score - case wins - recurring failures ## Reliability - success rate - strict validation passes - stddev / stability ## Speed - average duration This gives you the real tradeoff: > Is the quality gain worth the latency/cost? --- # 8) Suggested interpretation order When comparing models, I’d look at: 1. **success rate** 2. **avg heuristic score** 3. **stability** 4. **case wins** 5. **avg duration** 6. **failed-check leaderboard** Example: - `gpt-4.1` avg 90, stddev 4.5, 8 case wins, 2200ms - `gpt-4.1-mini` avg 84, stddev 6.8, 2 case wins, 900ms That suggests: - full model is stronger - mini may still be acceptable if speed/cost matters more --- # 9) Best practical usage A strong workflow is: ## Step 1 Pick your best prompt version first. ## Step 2 Run model comparison with that one prompt. ## Step 3 Choose your default model based on: - quality bar - speed budget - cost budget That avoids mixing prompt and model effects too early. --- # 10) My recommendation This is enough to make model choice evidence-based. The best next moves from here are probably: 1. **compare prompt+model combinations in one matrix** 2. **generate golden outputs for the remaining 5 cases** 3. **build an HTML report for easier review** 4. **add cost estimation into evaluation summaries** My recommendation: **4 next** if you care about product economics, or **1 next** if you want the most complete evaluation framework. If you want, I can do **4 next** and add: - estimated cost per run - estimated cost per full eval suite - quality-per-dollar style summary - a much clearer view of whether the better model is actually worth it

Yes — **1 next** is the strongest move now. You’ve already built the pieces separately: - prompt comparison - model comparison - multi-run eval - heuristics - stability - cost estimates The natural next step is a **prompt × model matrix runner**. That gives you one unified answer to: > “What is the best overall combination for this product?” Not just: - best prompt - best model …but best **pairing**. --- # Goal Compare: - multiple **prompts** - multiple **models** - across all eval cases - across multiple runs And summarize: - quality - stability - speed - estimated cost - case wins - recurring failed checks all in one place. --- # Recommended output For each **prompt + model combination**, you want: - success rate - avg heuristic score - heuristic stddev - stability label - avg duration - avg cost/run - total eval cost - quality per dollar - case win count - top failed checks That gives you a clean ranking surface. --- # What we’ll add 1. matrix comparison types 2. matrix aggregation helpers 3. `compareMatrixMultiRun.ts` 4. Markdown/CSV summaries 5. npm script We’ll reuse your existing heuristics, cost estimation, stats, and service debug path. --- # 1) Add matrix comparison types ## `src/eval/matrixTypes.ts` ```ts import type { HeuristicEvaluation } from "./heuristicTypes"; export type MatrixRunResult = { runIndex: number; model: string; promptVersion: string; success: boolean; durationMs: number; error?: string; output?: unknown; debug?: { strictValidationPassed: boolean; normalizationUsed: boolean; outputText: string; parsedStructured: unknown | null; systemPrompt: string; userPrompt: string; }; heuristics?: HeuristicEvaluation; estimatedUsage?: { inputTokens: number; outputTokens: number; totalTokens: number; estimatedCostUsd: number; }; }; export type MatrixCaseCombinationAggregate = { model: string; promptVersion: string; totalRuns: number; successCount: number; failureCount: number; strictValidationPassCount: number; normalizationUsedCount: number; avgDurationMs: number; avgHeuristicScore: number; heuristicStdDev: number; stabilityLabel: "high" | "medium" | "low"; minHeuristicScore: number | null; maxHeuristicScore: number | null; avgInputTokens: number; avgOutputTokens: number; avgTotalTokens: number; estimatedTotalCostUsd: number; estimatedAvgCostPerRunUsd: number; failedCheckCounts: Record<string, number>; warningCounts: Record<string, number>; runs: MatrixRunResult[]; }; export type MatrixCaseComparison = { caseId: string; caseName: string; category: string; brainDump: string; aggregates: MatrixCaseCombinationAggregate[]; }; export type OverallMatrixAggregate = { model: string; promptVersion: string; totalCases: number; totalRuns: number; successCount: number; failureCount: number; successRate: number; caseWinCount: number; tiedCaseCount: number; strictValidationPassCount: number; normalizationUsedCount: number; avgDurationMs: number; avgHeuristicScore: number; heuristicStdDev: number; stabilityLabel: "high" | "medium" | "low"; minHeuristicScore: number | null; maxHeuristicScore: number | null; avgInputTokens: number; avgOutputTokens: number; avgTotalTokens: number; estimatedTotalCostUsd: number; estimatedAvgCostPerRunUsd: number; qualityPerDollar: number; failedCheckCounts: Record<string, number>; warningCounts: Record<string, number>; }; export type MatrixComparisonSummary = { comparedModels: string[]; comparedPromptVersions: string[]; runCount: number; createdAt: string; totalCases: number; cases: MatrixCaseComparison[]; overallAggregates: OverallMatrixAggregate[]; }; ``` --- # 2) Add matrix aggregation helpers ## `src/eval/aggregateMatrix.ts` ```ts import type { MatrixRunResult, MatrixCaseCombinationAggregate, MatrixCaseComparison, OverallMatrixAggregate, } from "./matrixTypes"; import { average, stddev, stabilityLabelFromStdDev } from "./stats"; function countBy(items: string[]) { const map: Record<string, number> = {}; for (const item of items) { map[item] = (map[item] ?? 0) + 1; } return map; } function mergeCountMaps(maps: Record<string, number>[]) { const merged: Record<string, number> = {}; for (const map of maps) { for (const [key, value] of Object.entries(map)) { merged[key] = (merged[key] ?? 0) + value; } } return merged; } export function aggregateMatrixRuns(params: { model: string; promptVersion: string; runs: MatrixRunResult[]; }): MatrixCaseCombinationAggregate { const { model, promptVersion, runs } = params; const successRuns = runs.filter((r) => r.success); const scores = successRuns .map((r) => r.heuristics?.score) .filter((v): v is number => typeof v === "number"); const failedCheckIds = successRuns.flatMap((r) => (r.heuristics?.checks ?? []).filter((c) => !c.passed).map((c) => c.id) ); const warnings = successRuns.flatMap((r) => r.heuristics?.warnings ?? []); const inputTokens = successRuns .map((r) => r.estimatedUsage?.inputTokens) .filter((v): v is number => typeof v === "number"); const outputTokens = successRuns .map((r) => r.estimatedUsage?.outputTokens) .filter((v): v is number => typeof v === "number"); const totalTokens = successRuns .map((r) => r.estimatedUsage?.totalTokens) .filter((v): v is number => typeof v === "number"); const costs = successRuns .map((r) => r.estimatedUsage?.estimatedCostUsd) .filter((v): v is number => typeof v === "number"); const heuristicStdDev = stddev(scores); return { model, promptVersion, totalRuns: runs.length, successCount: successRuns.length, failureCount: runs.length - successRuns.length, strictValidationPassCount: successRuns.filter( (r) => r.debug?.strictValidationPassed ).length, normalizationUsedCount: successRuns.filter( (r) => r.debug?.normalizationUsed ).length, avgDurationMs: average(runs.map((r) => r.durationMs)), avgHeuristicScore: average(scores), heuristicStdDev, stabilityLabel: stabilityLabelFromStdDev(heuristicStdDev), minHeuristicScore: scores.length ? Math.min(...scores) : null, maxHeuristicScore: scores.length ? Math.max(...scores) : null, avgInputTokens: average(inputTokens), avgOutputTokens: average(outputTokens), avgTotalTokens: average(totalTokens), estimatedTotalCostUsd: costs.reduce((sum, c) => sum + c, 0), estimatedAvgCostPerRunUsd: average(costs), failedCheckCounts: countBy(failedCheckIds), warningCounts: countBy(warnings), runs, }; } function getCaseWinners(caseComparison: MatrixCaseComparison) { const sorted = [...caseComparison.aggregates].sort( (a, b) => b.avgHeuristicScore - a.avgHeuristicScore ); if (sorted.length === 0) { return { winners: [], topScore: 0 }; } const topScore = sorted[0].avgHeuristicScore; const winners = sorted .filter((agg) => agg.avgHeuristicScore === topScore) .map((agg) => `${agg.promptVersion}__${agg.model}`); return { winners, topScore }; } export function aggregateOverallMatrixResults( cases: MatrixCaseComparison[], combinations: Array<{ promptVersion: string; model: string }> ): OverallMatrixAggregate[] { return combinations.map(({ promptVersion, model }) => { const key = `${promptVersion}__${model}`; const aggregates = cases .map((c) => c.aggregates.find( (agg) => agg.promptVersion === promptVersion && agg.model === model ) ) .filter((agg): agg is MatrixCaseCombinationAggregate => Boolean(agg)); const totalCases = aggregates.length; const totalRuns = aggregates.reduce((sum, agg) => sum + agg.totalRuns, 0); const successCount = aggregates.reduce((sum, agg) => sum + agg.successCount, 0); const failureCount = aggregates.reduce((sum, agg) => sum + agg.failureCount, 0); const strictValidationPassCount = aggregates.reduce( (sum, agg) => sum + agg.strictValidationPassCount, 0 ); const normalizationUsedCount = aggregates.reduce( (sum, agg) => sum + agg.normalizationUsedCount, 0 ); const scores = aggregates.flatMap((agg) => agg.runs .filter((run) => run.success && typeof run.heuristics?.score === "number") .map((run) => run.heuristics!.score) ); const durations = aggregates.flatMap((agg) => agg.runs.map((run) => run.durationMs) ); const inputTokens = aggregates.flatMap((agg) => agg.runs .map((run) => run.estimatedUsage?.inputTokens) .filter((v): v is number => typeof v === "number") ); const outputTokens = aggregates.flatMap((agg) => agg.runs .map((run) => run.estimatedUsage?.outputTokens) .filter((v): v is number => typeof v === "number") ); const totalTokens = aggregates.flatMap((agg) => agg.runs .map((run) => run.estimatedUsage?.totalTokens) .filter((v): v is number => typeof v === "number") ); const costs = aggregates.flatMap((agg) => agg.runs .map((run) => run.estimatedUsage?.estimatedCostUsd) .filter((v): v is number => typeof v === "number") ); let caseWinCount = 0; let tiedCaseCount = 0; for (const caseComparison of cases) { const { winners } = getCaseWinners(caseComparison); if (winners.includes(key)) { if (winners.length === 1) caseWinCount += 1; else tiedCaseCount += 1; } } const heuristicStdDev = stddev(scores); const estimatedTotalCostUsd = costs.reduce((sum, c) => sum + c, 0); const estimatedAvgCostPerRunUsd = average(costs); return { model, promptVersion, totalCases, totalRuns, successCount, failureCount, successRate: totalRuns > 0 ? successCount / totalRuns : 0, caseWinCount, tiedCaseCount, strictValidationPassCount, normalizationUsedCount, avgDurationMs: average(durations), avgHeuristicScore: average(scores), heuristicStdDev, stabilityLabel: stabilityLabelFromStdDev(heuristicStdDev), minHeuristicScore: scores.length ? Math.min(...scores) : null, maxHeuristicScore: scores.length ? Math.max(...scores) : null, avgInputTokens: average(inputTokens), avgOutputTokens: average(outputTokens), avgTotalTokens: average(totalTokens), estimatedTotalCostUsd, estimatedAvgCostPerRunUsd, qualityPerDollar: estimatedTotalCostUsd > 0 ? average(scores) / estimatedTotalCostUsd : 0, failedCheckCounts: mergeCountMaps( aggregates.map((agg) => agg.failedCheckCounts) ), warningCounts: mergeCountMaps( aggregates.map((agg) => agg.warningCounts) ), }; }); } ``` --- # 3) Build the matrix runner ## `src/eval/compareMatrixMultiRun.ts` ```ts import path from "node:path"; import { evalCases } from "./cases"; import { ensureDir, timestamp, writeJsonFile, writeTextFile } from "./utils"; import { generateBrainDumpPlanDebug } from "../lib/planning/service"; import { PROMPT_V1, PROMPT_V2, type PromptConfig, } from "../lib/planning/prompts"; import { evaluateHeuristics } from "./heuristics"; import { estimateUsageAndCost } from "./costs"; import { aggregateMatrixRuns, aggregateOverallMatrixResults, } from "./aggregateMatrix"; import type { MatrixRunResult, MatrixCaseComparison, MatrixComparisonSummary, } from "./matrixTypes"; const RUN_COUNT = Number(process.env.EVAL_RUN_COUNT || "3"); const OUTPUT_DIR = path.join( process.cwd(), "eval-matrix-comparisons", timestamp() ); const MODELS_TO_COMPARE = ( process.env.EVAL_MODELS || "gpt-4.1-mini,gpt-4.1" ) .split(",") .map((s) => s.trim()) .filter(Boolean); const PROMPTS_TO_COMPARE: PromptConfig[] = [PROMPT_V1, PROMPT_V2]; async function run() { console.log( `Running matrix comparison with prompts=[${PROMPTS_TO_COMPARE .map((p) => p.version) .join(", ")}], models=[${MODELS_TO_COMPARE.join(", ")}], runs=${RUN_COUNT}` ); await ensureDir(OUTPUT_DIR); const caseComparisons: MatrixCaseComparison[] = []; for (const testCase of evalCases) { console.log(`\n→ ${testCase.id}: ${testCase.name}`); const runMap = new Map<string, MatrixRunResult[]>(); for (const promptConfig of PROMPTS_TO_COMPARE) { for (const model of MODELS_TO_COMPARE) { const key = `${promptConfig.version}__${model}`; runMap.set(key, []); for (let runIndex = 1; runIndex <= RUN_COUNT; runIndex++) { const started = Date.now(); try { const resultDebug = await generateBrainDumpPlanDebug({ brainDump: testCase.brainDump, mode: "mixed", style: "direct", depth: "standard", promptConfig, modelOverride: model, }); const durationMs = Date.now() - started; const heuristics = evaluateHeuristics(testCase, resultDebug.plan); const estimatedUsage = estimateUsageAndCost({ model, systemPrompt: resultDebug.debug.systemPrompt, userPrompt: resultDebug.debug.userPrompt, outputText: resultDebug.debug.outputText, }); const runResult: MatrixRunResult = { runIndex, model, promptVersion: promptConfig.version, success: true, durationMs, output: resultDebug.plan, debug: { strictValidationPassed: resultDebug.debug.strictValidationPassed, normalizationUsed: resultDebug.debug.normalizationUsed, outputText: resultDebug.debug.outputText, parsedStructured: resultDebug.debug.parsedStructured, systemPrompt: resultDebug.debug.systemPrompt, userPrompt: resultDebug.debug.userPrompt, }, heuristics, estimatedUsage, }; runMap.get(key)?.push(runResult); console.log( ` ${promptConfig.version} + ${model} run ${runIndex}/${RUN_COUNT}: success (${durationMs}ms) heuristic=${heuristics.score} cost=$${estimatedUsage.estimatedCostUsd.toFixed(4)}` ); } catch (error) { const durationMs = Date.now() - started; const runResult: MatrixRunResult = { runIndex, model, promptVersion: promptConfig.version, success: false, durationMs, error: error instanceof Error ? error.message : "Unknown error", }; runMap.get(key)?.push(runResult); console.log( ` ${promptConfig.version} + ${model} run ${runIndex}/${RUN_COUNT}: failed (${durationMs}ms)` ); } } } } const aggregates = PROMPTS_TO_COMPARE.flatMap((promptConfig) => MODELS_TO_COMPARE.map((model) => aggregateMatrixRuns({ model, promptVersion: promptConfig.version, runs: runMap.get(`${promptConfig.version}__${model}`) ?? [], }) ) ); const caseComparison: MatrixCaseComparison = { caseId: testCase.id, caseName: testCase.name, category: testCase.category, brainDump: testCase.brainDump, aggregates, }; caseComparisons.push(caseComparison); await writeJsonFile( path.join(OUTPUT_DIR, `${testCase.id}.matrix.json`), caseComparison ); } const combinations = PROMPTS_TO_COMPARE.flatMap((promptConfig) => MODELS_TO_COMPARE.map((model) => ({ promptVersion: promptConfig.version, model, })) ); const summary: MatrixComparisonSummary = { comparedModels: MODELS_TO_COMPARE, comparedPromptVersions: PROMPTS_TO_COMPARE.map((p) => p.version), runCount: RUN_COUNT, createdAt: new Date().toISOString(), totalCases: evalCases.length, cases: caseComparisons, overallAggregates: aggregateOverallMatrixResults( caseComparisons, combinations ), }; await writeJsonFile(path.join(OUTPUT_DIR, "matrix-summary.json"), summary); await writeTextFile( path.join(OUTPUT_DIR, "matrix-summary.md"), buildMarkdownSummary(summary) ); await writeTextFile( path.join(OUTPUT_DIR, "matrix-scoring.csv"), buildCsvSummary(summary) ); console.log(`\nDone. Matrix comparison written to:\n${OUTPUT_DIR}`); } ``` --- # 4) Add Markdown summary builder Continue in `src/eval/compareMatrixMultiRun.ts`: ```ts function buildMarkdownSummary(summary: MatrixComparisonSummary) { const lines: string[] = []; lines.push("# Multi-Run Prompt × Model Matrix Summary"); lines.push(""); lines.push(`- Prompts: ${summary.comparedPromptVersions.join(", ")}`); lines.push(`- Models: ${summary.comparedModels.join(", ")}`); lines.push(`- Runs per combination per case: ${summary.runCount}`); lines.push(`- Total cases: ${summary.totalCases}`); lines.push(`- Created at: ${summary.createdAt}`); lines.push(""); lines.push("## Overall Combination Summary"); lines.push(""); lines.push( "| Prompt | Model | Success Rate | Avg Score | StdDev | Stability | Avg Duration | Avg Cost/Run | Total Cost | Quality/$ | Case Wins |" ); lines.push("|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|"); const sortedOverall = [...summary.overallAggregates].sort((a, b) => { if (b.avgHeuristicScore !== a.avgHeuristicScore) { return b.avgHeuristicScore - a.avgHeuristicScore; } return a.estimatedAvgCostPerRunUsd - b.estimatedAvgCostPerRunUsd; }); for (const overall of sortedOverall) { lines.push( `| ${overall.promptVersion} | ${overall.model} | ${(overall.successRate * 100).toFixed(1)}% (${overall.successCount}/${overall.totalRuns}) | ${overall.avgHeuristicScore.toFixed(1)} | ${overall.heuristicStdDev.toFixed(1)} | ${overall.stabilityLabel} | ${Math.round(overall.avgDurationMs)}ms | $${overall.estimatedAvgCostPerRunUsd.toFixed(4)} | $${overall.estimatedTotalCostUsd.toFixed(4)} | ${overall.qualityPerDollar.toFixed(1)} | ${overall.caseWinCount} |` ); } lines.push(""); lines.push("## Best Combination by Case"); lines.push(""); for (const c of summary.cases) { const sorted = [...c.aggregates].sort((a, b) => { if (b.avgHeuristicScore !== a.avgHeuristicScore) { return b.avgHeuristicScore - a.avgHeuristicScore; } return a.estimatedAvgCostPerRunUsd - b.estimatedAvgCostPerRunUsd; }); if (sorted.length === 0) { lines.push(`- ${c.caseId} - ${c.caseName}: no results`); continue; } const best = sorted[0]; lines.push( `- ${c.caseId} - ${c.caseName}: ${best.promptVersion} + ${best.model} (avg ${best.avgHeuristicScore.toFixed(1)}, stddev ${best.heuristicStdDev.toFixed(1)}, cost/run $${best.estimatedAvgCostPerRunUsd.toFixed(4)})` ); } lines.push(""); lines.push("## Frequent Failed Checks by Combination"); lines.push(""); for (const overall of sortedOverall) { lines.push(`### ${overall.promptVersion} + ${overall.model}`); lines.push(""); const failedEntries = Object.entries(overall.failedCheckCounts).sort( (a, b) => b[1] - a[1] ); if (failedEntries.length === 0) { lines.push("- No repeated failed checks"); } else { for (const [checkId, count] of failedEntries.slice(0, 10)) { lines.push(`- ${checkId}: ${count}`); } } lines.push(""); } return lines.join("\n"); } ``` --- # 5) Add CSV summary builder Still in `src/eval/compareMatrixMultiRun.ts`: ```ts function buildCsvSummary(summary: MatrixComparisonSummary) { const headers = [ "scope", "caseId", "caseName", "category", "promptVersion", "model", "runCount", "successCount", "failureCount", "successRate", "avgHeuristicScore", "heuristicStdDev", "stabilityLabel", "avgDurationMs", "avgInputTokens", "avgOutputTokens", "avgTotalTokens", "estimatedAvgCostPerRunUsd", "estimatedTotalCostUsd", "qualityPerDollar", "caseWinCount", "tiedCaseCount", "strictValidationPassCount", "normalizationUsedCount", "topFailedCheckIds", "notes", ]; const rows: string[] = []; for (const overall of summary.overallAggregates) { const topFailedChecks = Object.entries(overall.failedCheckCounts) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .map(([id, count]) => `${id}:${count}`) .join(" | "); rows.push( [ "overall", "", '""', "", overall.promptVersion, overall.model, overall.totalRuns, overall.successCount, overall.failureCount, overall.successRate.toFixed(3), overall.avgHeuristicScore.toFixed(1), overall.heuristicStdDev.toFixed(1), overall.stabilityLabel, Math.round(overall.avgDurationMs), Math.round(overall.avgInputTokens), Math.round(overall.avgOutputTokens), Math.round(overall.avgTotalTokens), overall.estimatedAvgCostPerRunUsd.toFixed(6), overall.estimatedTotalCostUsd.toFixed(6), overall.qualityPerDollar.toFixed(2), overall.caseWinCount, overall.tiedCaseCount, overall.strictValidationPassCount, overall.normalizationUsedCount, `"${topFailedChecks.replace(/"/g, '""')}"`, '""', ].join(",") ); } for (const c of summary.cases) { for (const agg of c.aggregates) { const topFailedChecks = Object.entries(agg.failedCheckCounts) .sort((a, b) => b[1] - a[1]) .slice(0, 8) .map(([id, count]) => `${id}:${count}/${agg.totalRuns}`) .join(" | "); rows.push( [ "case", c.caseId, `"${c.caseName}"`, c.category, agg.promptVersion, agg.model, agg.totalRuns, agg.successCount, agg.failureCount, (agg.totalRuns > 0 ? agg.successCount / agg.totalRuns : 0).toFixed(3), agg.avgHeuristicScore.toFixed(1), agg.heuristicStdDev.toFixed(1), agg.stabilityLabel, Math.round(agg.avgDurationMs), Math.round(agg.avgInputTokens), Math.round(agg.avgOutputTokens), Math.round(agg.avgTotalTokens), agg.estimatedAvgCostPerRunUsd.toFixed(6), agg.estimatedTotalCostUsd.toFixed(6), "", "", "", agg.strictValidationPassCount, agg.normalizationUsedCount, `"${topFailedChecks.replace(/"/g, '""')}"`, '""', ].join(",") ); } } return [headers.join(","), ...rows].join("\n"); } run().catch((error) => { console.error("Matrix comparison crashed:", error); process.exit(1); }); ``` --- # 6) Add npm script In `package.json`: ```json { "scripts": { "dev": "next dev", "eval": "tsx src/eval/runEval.ts", "eval:compare": "tsx src/eval/comparePrompts.ts", "eval:compare:multi": "tsx src/eval/comparePromptsMultiRun.ts", "eval:models:multi": "tsx src/eval/compareModelsMultiRun.ts", "eval:matrix:multi": "tsx src/eval/compareMatrixMultiRun.ts" } } ``` --- # 7) How to run it Example: ```bash EVAL_RUN_COUNT=3 EVAL_MODELS=gpt-4.1-mini,gpt-4.1 npm run eval:matrix:multi ``` This compares: - `v1 + gpt-4.1-mini` - `v1 + gpt-4.1` - `v2 + gpt-4.1-mini` - `v2 + gpt-4.1` across all cases and runs. --- # 8) What this lets you answer Now you can decide things like: ## Best absolute quality Which combo gets the strongest average heuristic score? ## Best reliability Which combo has the best stability + success rate? ## Best economic default Which combo gives acceptable quality at the best cost/run? ## Best premium mode Maybe: - `v2 + gpt-4.1-mini` is best default - `v2 + gpt-4.1` is best premium/deep mode That’s exactly the kind of product decision this framework should support. --- # 9) Suggested interpretation order When reading the matrix summary, I’d use: 1. **Success rate** 2. **Avg heuristic score** 3. **Stddev / stability** 4. **Case wins** 5. **Avg cost/run** 6. **Quality/$** 7. **Failed check patterns** That balances: - reliability - output quality - economics --- # 10) My recommendation This is probably enough evaluation infrastructure for a very strong MVP iteration loop. The best next moves from here are likely: 1. **build an HTML report for easier review** 2. **switch from estimated costs to exact API usage where available** 3. **generate golden outputs for the remaining 5 cases** 4. **add CI-friendly regression thresholds** My recommendation: **4 next** if you want this to become a repeatable engineering tool, or **1 next** if you want much easier human review. If you want, I can do **4 next** and give you: - pass/fail thresholds for regressions - a CI-friendly summary JSON contract - an exit code strategy - recommended guardrails like “fail if avg score drops by > X”

Yes — **4 next** is the practical move. For CI, you usually care most about the **shipping combination**, not every research combo. So the right upgrade is: - keep full matrix eval for exploration - let CI focus on **one primary prompt/model pair** - use the same regression logic - reduce runtime, cost, and noise --- # Goal Add filtering so CI can evaluate only a target combo, like: - `v2 + gpt-4.1-mini` while still keeping your broader matrix tooling for local research. --- # What we’ll add 1. env-based filtering for the matrix runner 2. optional filtering in regression comparison 3. helper functions for consistent combo matching 4. recommended CI usage --- # 1) Add combo filter helpers ## `src/eval/comboFilter.ts` ```ts export type Combo = { promptVersion: string; model: string; }; export function comboKey(combo: Combo) { return `${combo.promptVersion}__${combo.model}`; } export function parsePrimaryComboFromEnv(): | Combo | null { const raw = process.env.EVAL_PRIMARY_COMBO?.trim(); if (!raw) return null; const [promptVersion, model] = raw.split("__").map((s) => s.trim()); if (!promptVersion || !model) { throw new Error( `Invalid EVAL_PRIMARY_COMBO format. Expected "promptVersion__model", got "${raw}"` ); } return { promptVersion, model }; } export function filterCombos<T extends Combo>( combos: T[], primaryCombo: Combo | null ): T[] { if (!primaryCombo) return combos; return combos.filter( (combo) => combo.promptVersion === primaryCombo.promptVersion && combo.model === primaryCombo.model ); } ``` This gives you one consistent format: ```env EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini ``` --- # 2) Filter combinations in the matrix runner Update `src/eval/compareMatrixMultiRun.ts`. ## Add import ```ts import { filterCombos, parsePrimaryComboFromEnv } from "./comboFilter"; ``` ## Replace prompt/model config section Instead of directly using all prompt/model combinations later, define them once. Add: ```ts const ALL_COMBINATIONS = PROMPTS_TO_COMPARE.flatMap((promptConfig) => MODELS_TO_COMPARE.map((model) => ({ promptVersion: promptConfig.version, model, promptConfig, })) ); const PRIMARY_COMBO = parsePrimaryComboFromEnv(); const COMBINATIONS_TO_COMPARE = filterCombos( ALL_COMBINATIONS, PRIMARY_COMBO ); ``` Because `filterCombos` only checks `promptVersion` and `model`, it will preserve `promptConfig` on the objects. ## Update startup log Replace: ```ts console.log( `Running matrix comparison with prompts=[${PROMPTS_TO_COMPARE .map((p) => p.version) .join(", ")}], models=[${MODELS_TO_COMPARE.join(", ")}], runs=${RUN_COUNT}` ); ``` with: ```ts console.log( `Running matrix comparison with combinations=[${COMBINATIONS_TO_COMPARE .map((c) => `${c.promptVersion} + ${c.model}`) .join(", ")}], runs=${RUN_COUNT}` ); ``` ## Update run loop Replace nested prompt/model loops: ```ts for (const promptConfig of PROMPTS_TO_COMPARE) { for (const model of MODELS_TO_COMPARE) { const key = `${promptConfig.version}__${model}`; runMap.set(key, []); for (let runIndex = 1; runIndex <= RUN_COUNT; runIndex++) { ... } } } ``` with: ```ts for (const combo of COMBINATIONS_TO_COMPARE) { const { promptConfig, model, promptVersion } = combo; const key = `${promptVersion}__${model}`; runMap.set(key, []); for (let runIndex = 1; runIndex <= RUN_COUNT; runIndex++) { const started = Date.now(); try { const resultDebug = await generateBrainDumpPlanDebug({ brainDump: testCase.brainDump, mode: "mixed", style: "direct", depth: "standard", promptConfig, modelOverride: model, }); const durationMs = Date.now() - started; const heuristics = evaluateHeuristics(testCase, resultDebug.plan); const estimatedUsage = estimateUsageAndCost({ model, systemPrompt: resultDebug.debug.systemPrompt, userPrompt: resultDebug.debug.userPrompt, outputText: resultDebug.debug.outputText, }); const runResult: MatrixRunResult = { runIndex, model, promptVersion, success: true, durationMs, output: resultDebug.plan, debug: { strictValidationPassed: resultDebug.debug.strictValidationPassed, normalizationUsed: resultDebug.debug.normalizationUsed, outputText: resultDebug.debug.outputText, parsedStructured: resultDebug.debug.parsedStructured, systemPrompt: resultDebug.debug.systemPrompt, userPrompt: resultDebug.debug.userPrompt, }, heuristics, estimatedUsage, }; runMap.get(key)?.push(runResult); console.log( ` ${promptVersion} + ${model} run ${runIndex}/${RUN_COUNT}: success (${durationMs}ms) heuristic=${heuristics.score} cost=$${estimatedUsage.estimatedCostUsd.toFixed(4)}` ); } catch (error) { const durationMs = Date.now() - started; const runResult: MatrixRunResult = { runIndex, model, promptVersion, success: false, durationMs, error: error instanceof Error ? error.message : "Unknown error", }; runMap.get(key)?.push(runResult); console.log( ` ${promptVersion} + ${model} run ${runIndex}/${RUN_COUNT}: failed (${durationMs}ms)` ); } } } ``` ## Update per-case aggregation Replace: ```ts const aggregates = PROMPTS_TO_COMPARE.flatMap((promptConfig) => MODELS_TO_COMPARE.map((model) => aggregateMatrixRuns({ model, promptVersion: promptConfig.version, runs: runMap.get(`${promptConfig.version}__${model}`) ?? [], }) ) ); ``` with: ```ts const aggregates = COMBINATIONS_TO_COMPARE.map((combo) => aggregateMatrixRuns({ model: combo.model, promptVersion: combo.promptVersion, runs: runMap.get(`${combo.promptVersion}__${combo.model}`) ?? [], }) ); ``` ## Update overall combinations list Replace: ```ts const combinations = PROMPTS_TO_COMPARE.flatMap((promptConfig) => MODELS_TO_COMPARE.map((model) => ({ promptVersion: promptConfig.version, model, })) ); ``` with: ```ts const combinations = COMBINATIONS_TO_COMPARE.map((combo) => ({ promptVersion: combo.promptVersion, model: combo.model, })); ``` That’s enough to make the runner support either: - full matrix - single shipping combo depending on env. --- # 3) Filter regression comparison too Now make the regression script optionally compare only the shipping combo. Update `src/eval/compareAgainstBaseline.ts`. ## Add import ```ts import { parsePrimaryComboFromEnv, comboKey } from "./comboFilter"; ``` ## Remove local `combinationKey()` function Delete: ```ts function combinationKey(promptVersion: string, model: string) { return `${promptVersion}__${model}`; } ``` ## Add primary combo filter near the top of `main()` After thresholds: ```ts const primaryCombo = parsePrimaryComboFromEnv(); ``` ## Build filtered maps Replace: ```ts const baselineMap = new Map( baseline.combinations.map((c) => [combinationKey(c.promptVersion, c.model), c]) ); const currentMap = new Map( current.combinations.map((c) => [combinationKey(c.promptVersion, c.model), c]) ); ``` with: ```ts const filteredBaselineCombinations = primaryCombo ? baseline.combinations.filter( (c) => c.promptVersion === primaryCombo.promptVersion && c.model === primaryCombo.model ) : baseline.combinations; const filteredCurrentCombinations = primaryCombo ? current.combinations.filter( (c) => c.promptVersion === primaryCombo.promptVersion && c.model === primaryCombo.model ) : current.combinations; const baselineMap = new Map( filteredBaselineCombinations.map((c) => [comboKey(c), c]) ); const currentMap = new Map( filteredCurrentCombinations.map((c) => [comboKey(c), c]) ); ``` ## Improve console output Add: ```ts if (primaryCombo) { console.log( `Primary combo filter: ${primaryCombo.promptVersion} + ${primaryCombo.model}` ); } ``` Now your regression gate can either: - compare all combos - compare only the shipping combo --- # 4) Recommended env usage ## Full matrix, for local research ```bash EVAL_RUN_COUNT=3 EVAL_MODELS=gpt-4.1-mini,gpt-4.1 npm run eval:matrix:multi npm run eval:regression ``` ## Shipping-combo-only CI ```bash EVAL_RUN_COUNT=3 \ EVAL_MODELS=gpt-4.1-mini,gpt-4.1 \ EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini \ npm run eval:matrix:multi EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini npm run eval:regression ``` That keeps CI lean and relevant. --- # 5) Recommended baseline strategy You now have two good options. ## Option A: one baseline for all combos Good for research branches. ## Option B: one baseline for the shipping combo only Best for CI. I recommend: - keep full matrix baselines for local experimentation - use **primary combo filtering in CI** That gives you flexibility without slowing deploy checks. --- # 6) Nice optional improvement: fail if primary combo missing In CI, if you specify `EVAL_PRIMARY_COMBO`, you may want to fail hard if the combo doesn’t exist in the current snapshot. In `compareAgainstBaseline.ts`, after building `currentMap`, add: ```ts if (primaryCombo && currentMap.size === 0) { console.error( `Primary combo ${primaryCombo.promptVersion} + ${primaryCombo.model} was not found in current snapshot.` ); process.exit(2); } ``` And similarly for baseline: ```ts if (primaryCombo && baselineMap.size === 0) { console.error( `Primary combo ${primaryCombo.promptVersion} + ${primaryCombo.model} was not found in baseline snapshot.` ); process.exit(2); } ``` That prevents silent false passes. --- # 7) Nice optional improvement: shipping combo in summary output You can also add a note in `matrix-summary.md` if a primary combo filter is active. In `compareMatrixMultiRun.ts`: ```ts const PRIMARY_COMBO = parsePrimaryComboFromEnv(); ``` Then in `buildMarkdownSummary()`, pass it in or just add a header line before writing: ```ts if (PRIMARY_COMBO) { lines.push( `- Primary combo filter: ${PRIMARY_COMBO.promptVersion} + ${PRIMARY_COMBO.model}` ); } ``` Not required, but very helpful. --- # 8) Practical CI recommendation For daily CI: - use one shipping combo - use `EVAL_RUN_COUNT=3` - fail on real regressions only For release validation: - use one shipping combo - use `EVAL_RUN_COUNT=5` - maybe compare one fallback/premium combo too That keeps the eval budget and runtime reasonable. --- # 9) My recommendation This is the right balance: - research remains broad - CI remains fast - regressions stay tied to what actually ships --- # Best next move The most useful next options are: 1. **build an HTML report for easier human review** 2. **switch estimated costs to exact API usage if available** 3. **generate golden outputs for the remaining 5 test cases** 4. **add per-case regression guarding for especially important cases** My recommendation: **4 next** if you want tighter quality protection for critical cases like: - founder strategy ambiguity - dependency-heavy workflow - sparse ambiguity / hallucination risk If you want, I can do **4 next** and give you: - per-case regression thresholds - “must not regress” protected cases - critical-check enforcement by case - CI rules that are stricter where product risk is highest

Yes — **4 next** is the right move. Right now the framework is strong, but still a bit too hardcoded for real ops. The next upgrade is to make it **environment-configurable** so you can support: - local experimentation - PR checks - main branch checks - release validation without editing code each time. --- # Goal Make these configurable via env: - regression thresholds - protected case selection - strictness mode - primary combo - run count / model list So you can do things like: - **PR mode:** faster, looser, cheaper - **release mode:** slower, stricter, safer --- # Recommended design Use **three layers** of config: 1. **defaults in code** 2. **mode presets** (`pr`, `ci`, `release`) 3. **env overrides** That keeps the system: - sane by default - easy to use - easy to tune --- # What we’ll add 1. `evalEnv.ts` 2. mode presets 3. protected-case filtering from env 4. threshold overrides from env 5. regression script + matrix runner integration --- # 1) Add env/config helpers ## `src/eval/evalEnv.ts` ```ts function parseNumber(value: string | undefined): number | undefined { if (value == null || value.trim() === "") return undefined; const n = Number(value); return Number.isFinite(n) ? n : undefined; } function parseStringList(value: string | undefined): string[] | undefined { if (!value || !value.trim()) return undefined; return value .split(",") .map((s) => s.trim()) .filter(Boolean); } function parseBoolean(value: string | undefined): boolean | undefined { if (!value || !value.trim()) return undefined; const lower = value.trim().toLowerCase(); if (["1", "true", "yes", "on"].includes(lower)) return true; if (["0", "false", "no", "off"].includes(lower)) return false; return undefined; } export function getEvalMode(): "local" | "pr" | "ci" | "release" { const raw = process.env.EVAL_MODE?.trim().toLowerCase(); if (raw === "pr" || raw === "ci" || raw === "release" || raw === "local") { return raw; } return "local"; } export function getEnvThresholdOverrides() { return { maxSuccessRateDrop: parseNumber(process.env.EVAL_MAX_SUCCESS_RATE_DROP), maxAvgScoreDrop: parseNumber(process.env.EVAL_MAX_AVG_SCORE_DROP), maxStdDevIncrease: parseNumber(process.env.EVAL_MAX_STDDEV_INCREASE), maxDurationIncreasePct: parseNumber(process.env.EVAL_MAX_DURATION_INCREASE_PCT), maxCostIncreasePct: parseNumber(process.env.EVAL_MAX_COST_INCREASE_PCT), maxCaseWinDrop: parseNumber(process.env.EVAL_MAX_CASE_WIN_DROP), maxStrictValidationPassDrop: parseNumber( process.env.EVAL_MAX_STRICT_VALIDATION_PASS_DROP ), disallowNewTopFailedChecks: parseStringList( process.env.EVAL_DISALLOW_NEW_TOP_FAILED_CHECKS ), }; } export function getProtectedCasesOverride(): string[] | undefined { return parseStringList(process.env.EVAL_PROTECTED_CASES); } export function getRunCountFromEnv(defaultValue: number): number { const parsed = parseNumber(process.env.EVAL_RUN_COUNT); return parsed ?? defaultValue; } export function getModelsFromEnv(defaultModels: string[]): string[] { const parsed = parseStringList(process.env.EVAL_MODELS); return parsed && parsed.length ? parsed : defaultModels; } export function getPromptsFromEnv(defaultPrompts: string[]): string[] { const parsed = parseStringList(process.env.EVAL_PROMPTS); return parsed && parsed.length ? parsed : defaultPrompts; } export function getFailOnWarnings(): boolean { return parseBoolean(process.env.EVAL_FAIL_ON_WARNINGS) ?? false; } ``` --- # 2) Add mode presets Now define reasonable defaults per mode. ## Update `src/eval/regressionConfig.ts` ```ts import type { MatrixComparisonSummary } from "./matrixTypes"; import type { EvalRegressionSnapshot, CombinationRegressionSummary, CaseCombinationRegressionSummary, } from "./regressionTypes"; import { getEnvThresholdOverrides, getEvalMode } from "./evalEnv"; export type RegressionThresholds = { maxSuccessRateDrop: number; maxAvgScoreDrop: number; maxStdDevIncrease: number; maxDurationIncreasePct: number; maxCostIncreasePct: number; maxCaseWinDrop: number; maxStrictValidationPassDrop: number; disallowNewTopFailedChecks?: string[]; }; export const DEFAULT_REGRESSION_THRESHOLDS: RegressionThresholds = { maxSuccessRateDrop: 0.05, maxAvgScoreDrop: 3, maxStdDevIncrease: 4, maxDurationIncreasePct: 0.35, maxCostIncreasePct: 0.5, maxCaseWinDrop: 2, maxStrictValidationPassDrop: 3, disallowNewTopFailedChecks: [ "must_surface_enterprise_vs_selfserve", "must_notice_finance_jordan_or_legal_blockers", "must_not_hallucinate_jen_identity", "must_notice_sam_or_pricing_copy", ], }; export const REGRESSION_THRESHOLDS_BY_MODE: Record< "local" | "pr" | "ci" | "release", RegressionThresholds > = { local: { ...DEFAULT_REGRESSION_THRESHOLDS, maxAvgScoreDrop: 5, maxStdDevIncrease: 6, maxDurationIncreasePct: 0.75, maxCostIncreasePct: 1.0, }, pr: { ...DEFAULT_REGRESSION_THRESHOLDS, maxAvgScoreDrop: 4, maxStdDevIncrease: 5, maxDurationIncreasePct: 0.5, maxCostIncreasePct: 0.75, }, ci: { ...DEFAULT_REGRESSION_THRESHOLDS, }, release: { ...DEFAULT_REGRESSION_THRESHOLDS, maxSuccessRateDrop: 0.03, maxAvgScoreDrop: 2, maxStdDevIncrease: 3, maxDurationIncreasePct: 0.25, maxCostIncreasePct: 0.35, maxCaseWinDrop: 1, maxStrictValidationPassDrop: 2, }, }; export function getRegressionThresholds(): RegressionThresholds { const mode = getEvalMode(); const base = REGRESSION_THRESHOLDS_BY_MODE[mode]; const overrides = getEnvThresholdOverrides(); return { ...base, ...Object.fromEntries( Object.entries(overrides).filter(([, v]) => v !== undefined) ), }; } ``` Keep your `toRegressionSnapshot()` function below that as before. --- # 3) Make protected cases configurable Update `src/eval/caseRegressionConfig.ts`. ## Replace with: ```ts import { getEvalMode, getProtectedCasesOverride } from "./evalEnv"; export type ProtectedCaseRule = { caseId: string; description: string; maxAvgScoreDrop?: number; maxStdDevIncrease?: number; minSuccessRate?: number; maxDurationIncreasePct?: number; maxCostIncreasePct?: number; maxStrictValidationPassDrop?: number; disallowFailedChecks?: string[]; requireChecksToStayAbsent?: string[]; severity?: "warning" | "error"; }; export const DEFAULT_PROTECTED_CASE_RULES: ProtectedCaseRule[] = [ { caseId: "case_03", description: "Founder strategy ambiguity must keep surfacing core strategic tension.", maxAvgScoreDrop: 4, maxStdDevIncrease: 5, minSuccessRate: 0.95, disallowFailedChecks: [ "must_surface_enterprise_vs_selfserve", "must_treat_strategy_as_decision", ], severity: "error", }, { caseId: "case_06", description: "Dependency-heavy manager workflow must keep detecting blockers.", maxAvgScoreDrop: 4, maxStdDevIncrease: 5, minSuccessRate: 0.95, disallowFailedChecks: [ "must_notice_finance_jordan_or_legal_blockers", ], severity: "error", }, { caseId: "case_10", description: "Sparse ambiguity must preserve uncertainty and avoid hallucination.", maxAvgScoreDrop: 3, maxStdDevIncrease: 4, minSuccessRate: 0.95, disallowFailedChecks: [ "must_preserve_uncertainty", "must_not_hallucinate_jen_identity", ], requireChecksToStayAbsent: [ "must_not_hallucinate_jen_identity", ], severity: "error", }, ]; export const RELEASE_PROTECTED_CASE_RULES: ProtectedCaseRule[] = [ ...DEFAULT_PROTECTED_CASE_RULES, { caseId: "case_05", description: "Overwhelm-heavy inputs must remain calming and low-overload.", maxAvgScoreDrop: 3, maxStdDevIncrease: 4, minSuccessRate: 0.95, disallowFailedChecks: [ "should_limit_next_actions_for_overwhelm", "must_notice_application_or_taxes", ], severity: "error", }, { caseId: "case_01", description: "Launch planning must preserve dependency prioritization.", maxAvgScoreDrop: 3, maxStdDevIncrease: 4, minSuccessRate: 0.95, disallowFailedChecks: [ "must_notice_sam_or_pricing_copy", "must_treat_webinar_as_decision", ], severity: "error", }, ]; export function getProtectedCaseRules(): ProtectedCaseRule[] { const mode = getEvalMode(); const envOverride = getProtectedCasesOverride(); const baseRules = mode === "release" ? RELEASE_PROTECTED_CASE_RULES : DEFAULT_PROTECTED_CASE_RULES; if (!envOverride || envOverride.length === 0) { return baseRules; } const allowed = new Set(envOverride); return baseRules.filter((rule) => allowed.has(rule.caseId)); } ``` This gives you: - normal CI defaults - stricter release defaults - optional env filtering Example: ```env EVAL_PROTECTED_CASES=case_03,case_06,case_10 ``` --- # 4) Update regression script to use dynamic config Update `src/eval/compareAgainstBaseline.ts`. ## Replace imports: ```ts import { DEFAULT_REGRESSION_THRESHOLDS, type RegressionThresholds, } from "./regressionConfig"; import { PROTECTED_CASE_RULES } from "./caseRegressionConfig"; ``` with: ```ts import { getRegressionThresholds, type RegressionThresholds, } from "./regressionConfig"; import { getProtectedCaseRules } from "./caseRegressionConfig"; import { getEvalMode, getFailOnWarnings } from "./evalEnv"; ``` ## Inside `main()`, replace: ```ts const thresholds: RegressionThresholds = DEFAULT_REGRESSION_THRESHOLDS; ``` with: ```ts const thresholds: RegressionThresholds = getRegressionThresholds(); const protectedCaseRules = getProtectedCaseRules(); const evalMode = getEvalMode(); const failOnWarnings = getFailOnWarnings(); ``` ## Replace: ```ts for (const rule of PROTECTED_CASE_RULES) { ``` with: ```ts for (const rule of protectedCaseRules) { ``` ## Improve header logging Add after loading config: ```ts console.log(`# Regression Check`); console.log(`Mode: ${evalMode}`); if (primaryCombo) { console.log( `Primary combo filter: ${primaryCombo.promptVersion} + ${primaryCombo.model}` ); } console.log(`Protected cases: ${protectedCaseRules.map((r) => r.caseId).join(", ") || "(none)"}`); console.log(`Fail on warnings: ${failOnWarnings ? "yes" : "no"}`); console.log(`Baseline: ${baselinePath}`); console.log(`Current: ${currentPath}`); console.log(""); ``` Then remove the duplicate older header block. ## Update exit logic Replace: ```ts process.exit(errorCount > 0 ? 1 : 0); ``` with: ```ts if (errorCount > 0) { process.exit(1); } if (warningCount > 0 && failOnWarnings) { process.exit(1); } process.exit(0); ``` Now you can make warnings fail in release mode if you want via env: ```env EVAL_FAIL_ON_WARNINGS=true ``` --- # 5) Make matrix runner mode-aware too You may want the runner to automatically use different prompt/model sets in different modes. This is optional, but useful. ## In `src/eval/compareMatrixMultiRun.ts` Add import: ```ts import { getEvalMode, getModelsFromEnv, getRunCountFromEnv } from "./evalEnv"; ``` Then replace: ```ts const RUN_COUNT = Number(process.env.EVAL_RUN_COUNT || "3"); ... const MODELS_TO_COMPARE = ( process.env.EVAL_MODELS || "gpt-4.1-mini,gpt-4.1" ) .split(",") .map((s) => s.trim()) .filter(Boolean); ``` with: ```ts const EVAL_MODE = getEvalMode(); const RUN_COUNT = EVAL_MODE === "release" ? getRunCountFromEnv(5) : EVAL_MODE === "pr" ? getRunCountFromEnv(2) : getRunCountFromEnv(3); const MODELS_TO_COMPARE = getModelsFromEnv(["gpt-4.1-mini", "gpt-4.1"]); ``` And optionally log mode: ```ts console.log(`Eval mode: ${EVAL_MODE}`); ``` That gives you mode-aware defaults without forcing them. --- # 6) Recommended env strategy Here’s a good setup. ## Local exploration ```env EVAL_MODE=local EVAL_RUN_COUNT=3 EVAL_MODELS=gpt-4.1-mini,gpt-4.1 ``` ## PR checks ```env EVAL_MODE=pr EVAL_RUN_COUNT=2 EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini EVAL_PROTECTED_CASES=case_03,case_06,case_10 ``` ## CI main ```env EVAL_MODE=ci EVAL_RUN_COUNT=3 EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini EVAL_PROTECTED_CASES=case_03,case_06,case_10 ``` ## Release validation ```env EVAL_MODE=release EVAL_RUN_COUNT=5 EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini EVAL_PROTECTED_CASES=case_01,case_03,case_05,case_06,case_10 EVAL_FAIL_ON_WARNINGS=true ``` That’s a very usable progression. --- # 7) Useful optional threshold env vars You can now override any threshold ad hoc: ```env EVAL_MAX_AVG_SCORE_DROP=2 EVAL_MAX_STDDEV_INCREASE=3 EVAL_MAX_SUCCESS_RATE_DROP=0.03 EVAL_MAX_DURATION_INCREASE_PCT=0.25 EVAL_MAX_COST_INCREASE_PCT=0.35 EVAL_MAX_CASE_WIN_DROP=1 EVAL_MAX_STRICT_VALIDATION_PASS_DROP=2 EVAL_DISALLOW_NEW_TOP_FAILED_CHECKS=must_not_hallucinate_jen_identity,must_notice_finance_jordan_or_legal_blockers ``` This is especially useful when tuning CI sensitivity. --- # 8) Recommended npm usage You don’t necessarily need new scripts; env-driven behavior is enough. Example: ## PR run ```bash EVAL_MODE=pr EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini npm run eval:matrix:multi EVAL_MODE=pr EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini npm run eval:regression ``` ## Release run ```bash EVAL_MODE=release EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini npm run eval:matrix:multi EVAL_MODE=release EVAL_PRIMARY_COMBO=v2__gpt-4.1-mini EVAL_FAIL_ON_WARNINGS=true npm run eval:regression ``` --- # 9) Why this config strategy is good It avoids: - hardcoding one CI behavior forever - duplicating scripts for every environment - editing TypeScript just to tune guardrails And it gives you: - stable defaults - predictable env control - easier CI evolution over time --- # 10) My recommendation This is enough to make the eval system genuinely operable. If I were prioritizing next steps, I’d pick one of these: 1. **build an HTML report for easier human review** 2. **switch estimated costs to exact API usage if available** 3. **generate golden outputs for the remaining 5 cases** 4. **add a single `eval doctor` script that validates config/baselines before CI runs** My recommendation: **4 next** if you want operational robustness, or **1 next** if your main pain is manual review. If you want, I can do **4 next** and give you a lightweight `eval doctor` script that checks: - env config validity - baseline presence - combo existence - pricing coverage - prompt/model names - protected case config sanity