If You’re Building AI Products Today…

You’ve probably experienced the “Vibes Phase.”

You build a prototype.
You test a few prompts.
You think:

“Wow… that’s actually pretty good.”

Then you change:

  • The model
  • The prompt
  • The RAG documents
  • The temperature

And suddenly you’re stuck wondering:

Did I improve the system… or did I just break it?

Without a rigorous evaluation system, you are flying blind.

Drawing on insights from Hamel Husain, Jason Liu, Eugene Yan, and research teams at Anthropic, this guide explains how to move from vibes to Evaluation-Driven Development (EDD).

Part 1 — The Mindset Shift

Ditch Generic Metrics

Engineers love numbers.

So it’s tempting to rely on:

  • ROUGE
  • BERTScore
  • “Helpfulness” ratings

But these metrics are a mirage.

They are painkillers — they make you feel better, but they don’t solve the problem.

Why They Fail

They lack context.

Example:

If a legal AI cites a non-existent case, a generic “coherence” metric might rate it highly because the sentence structure is perfect.

For a lawyer?

That’s a catastrophic failure.

The Rule:
Your product has specific constraints.
Your evaluations must measure those constraints — not general capability.

The Core Philosophy: Look at Your Data

The highest ROI activity in AI engineering is not:

  • Prompt tweaking
  • Model switching
  • Fine-tuning

It’s manually reading your logs.

You must build product intuition by reading:

  • Inputs
  • Retrieval traces
  • Outputs

You cannot outsource this.

Part 2 — The Error Analysis Workflow

Borrowed from qualitative research.

1️⃣ Open Coding (The “What”)

Look at production traces.

Don’t fix them.

Describe what happened.

Example:

User asked for gluten-free recipe. Model suggested regular soy sauce.

Label:

Constraint Violation: Dietary Restriction

2️⃣ Axial Coding (Build a Taxonomy)

After reviewing 50–100 logs, patterns emerge.

Group failures into categories:

  • Hallucination
  • Refusal
  • Tone Mismatch
  • Retrieval Failure
  • Constraint Violation

Now you have a Failure Mode Taxonomy.

3️⃣ The Ceiling on Performance

Take 100 failures.

Count categories.

Example:

  • 50 → Blurry Image Inputs
  • 5 → Dog misclassified as Cat

Fixing the dog problem perfectly improves system by only 5%.

Attack the highest-volume failure first.

This is how you prioritize like an engineer — not emotionally.

Part 3 — Metrics That Matter

The “5-Star Lie”

Likert scales introduce noise.

One annotator’s 4 is another annotator’s 3.

Annotators default to 3 to avoid hard decisions.

This creates useless data.

The Power of Binary (Pass/Fail)

Switch to binary metrics.

Instead of:

Rate this recipe 3/5.

Use:

  • Dietary Adherence? (Pass/Fail)
  • Ingredient Correctness? (Pass/Fail)
  • Tone Appropriate? (Pass/Fail)

Binary forces clarity.

If you can’t define Pass vs Fail — you don’t understand your product.

Part 4 — Evaluation Frameworks

For RAG Systems (The 6 Evals)

Three variables:

  • Q → Question
  • C → Context
  • A → Answer

Key relationships:

1️⃣ Context Relevance (C|Q)

Did retriever fetch relevant data?

2️⃣ Faithfulness (A|C)

Is answer supported only by retrieved context?

3️⃣ Answer Relevance (A|Q)

Did the answer actually address the user’s need?

If C fails, A is doomed.

For AI Agents

Agents act. They don’t just respond.

Evaluation must include:

Deterministic Checks

  • Did code execute?
  • Was file saved?
  • API call succeed?

Pass@k

If agent tries 10 times, did it succeed at least once?

Pass^k

Did it succeed 10 times in a row?

Reliability > Exploration.

Part 5 — Scaling with LLM-as-a-Judge

Manual review works for 100 logs.

Not for 10,000.

Use a strong LLM as a grader.

But treat the judge like a product.

The Critique Shadowing Protocol

  1. Domain Expert labels Pass/Fail + critique
  2. Create Golden Dataset
  3. Few-shot prompt the Judge with critiques
  4. Measure agreement rate

Do not trust the Judge until:

Human–LLM agreement > 90%

⚠️ Judge Limitations

LLM Judges struggle with:

  • Sneaky edge cases
  • Implicit knowledge
  • Predicting user engagement

They are scalable — not omniscient.

Part 6 — The Cold Start Problem

No users yet?

You still need evaluation.

Use Synthetic Data

Define:

  • Personas (Frustrated User, Legal Analyst, Banker)
  • Scenarios (Bug report, Compliance question, Retrieval test)

Use an LLM to generate test inputs.

Build a:

Minimum Viable Evaluation (MVE)

Every code change should trigger the eval harness.

Summary — Your Evaluation Is Your Moat

The moat is not:

  • The model
  • The prompt
  • The temperature

Everyone has access to the same models.

Your moat is:

  • Your data
  • Your failure taxonomy
  • Your binary metrics
  • Your eval harness

The Loop

  1. Log data
  2. Read traces
  3. Categorize failures
  4. Fix system
  5. Add to eval suite
  6. Repeat

Move from vibes → engineering.

References

  • https://www.decodingai.com/p/the-5-star-lie-you-are-doing-ai-evals
  • https://www.decodingai.com/p/the-mirage-of-generic-ai-metrics
  • https://www.youtube.com/watch?v=JoAxZsdw_3w
  • https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
  • https://www.youtube.com/watch?v=e2i6JbU2R-s
  • https://eugeneyan.com/writing/llm-evaluators/
  • https://www.decodingai.com/p/stop-launching-ai-apps-without-this
  • https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/
  • https://hamel.dev/blog/posts/llm-judge/
  • https://www.youtube.com/watch?v=sEMYSSS6Ims