Case Study · Project 02

AI Output Evaluation & Quality-Control System

A tested portfolio prototype for reviewing AI-generated outputs against explicit instructions, supplied evidence, quality criteria, severity rules, and human-review requirements.

LLM EvaluationAI Quality AssuranceSource GroundingRubric DesignAdversarial Testing

StatusTested portfolio prototype

The problem

AI-generated content can look polished while still containing factual errors, unsupported claims, missing requirements, instruction violations, or formatting defects. A reviewer who relies mainly on fluency can therefore approve an output that is confident, readable, and wrong.

The goal

I wanted to design a reusable evaluation workflow that turns vague quality judgments into evidence-backed review. The system needed to distinguish between incorrect, unsupported, incomplete, non-compliant, poorly formatted, and merely improvable outputs.

Core design

Score the response — but do not let the average hide a serious error.

The framework uses eight 0–4 evaluation dimensions, then applies a severity hierarchy before issuing the final PASS, REVISE, or FAIL decision. This keeps a polished response from passing when it contains a Critical factual or groundedness failure.

Instruction Compliance

Did the response follow the actual task?

Factual Accuracy

Are factual claims correct against available evidence?

Groundedness

Did the model add unsupported claims?

Completeness

Were required elements omitted?

Relevance

Did the response stay focused on the task?

Clarity & Organization

Is the output understandable and usable?

Format Compliance

Did it follow structural and machine-readable requirements?

Tone & Audience Fit

Is the language appropriate for the intended reader?

Decision hierarchy

Critical Error → Essential-Dimension Failure → Major-Error Pattern → Numerical Score → Minor Issues / Observations

Workflow

The system separates evidence collection, quality review, error classification, severity analysis, decision logic, revision, and final human review rather than treating evaluation as one impressionistic step.

Evaluation Flow

Evidence → evaluation → decision → human review

TESTED FLOW

Task + Evidence

Collect the instruction, reference material and candidate output.

Boundary Check

Separate operational instructions from source content.

Evaluate

Review eight quality dimensions against explicit criteria.

Classify

Assign one primary error category and severity.

Score

Calculate the applicable rubric score without double-counting.

Override

Let Critical and essential-dimension failures outrank averages.

Decide

Return PASS, REVISE or FAIL using the decision hierarchy.

Human Review

Keep final approval with an appropriate human reviewer.

Testing

Five deliberately different failure modes

I tested the framework against constructed cases designed to expose different evaluator weaknesses rather than repeating the same type of obvious error.

TEST 01

Standard Failure

Wrong facts, hallucination and omission

Result

FAIL

TEST 02

Subtle Fidelity

Meaning-changing qualifier shifts

Result

REVISE

TEST 03

Strict Format

Correct content but broken parser contract

Result

Severity-aware

TEST 04

Adversarial

Instructions embedded inside source material

Result

FAIL

TEST 05

Insufficient Evidence

Unsupported claim without proof of falsity

Result

REVISE

Failure discovered

The first evaluation design had a hidden scoring problem. A single fabricated statement could damage both Factual Accuracy and Groundedness, which made it easy to report and penalize the same underlying defect twice.

Example failure

“Several enterprise pilots converted into signed contracts” can be both factually false and ungrounded. Treating those as two independent issues would artificially amplify one defect.

The design was refined through failure-driven testing

Failure / Risk

Double-counting risk

Design Fix

Primary Category + Secondary Impact

Failure / Risk

Meaning lost through small qualifiers

Design Fix

Semantic Fidelity Rule

Failure / Risk

Cosmetic and parser-breaking formats treated alike

Design Fix

Format Criticality Rule

Failure / Risk

Source text could behave like instructions

Design Fix

Instruction / Reference Boundary

Failure / Risk

Unsupported could be mislabeled false

Design Fix

"Not verifiable from the supplied evidence"

Failure / Risk

High average score could hide severe defects

Design Fix

Severity Override

Failure / Risk

FAIL and REVISE could blur together

Design Fix

Regeneration vs Minimum Sufficient Correction

Failure / Risk

Style preference could become a fake error

Design Fix

Evidence Requirement

Evidence

What was actually tested and changed

The portfolio does not present the final evaluator prompt as proof by itself. The evidence is the sequence of constructed tests, the weaknesses they exposed, and the concrete rules added to address those weaknesses.

Test scenarios

Different evaluator failure modes

Quality dimensions

0–4 scoring when applicable

Severity levels

Critical / Major / Minor / Observation

Current validation status

Tested portfolio prototype

The framework has been tested against the defined constructed scenarios used in this project. It is not presented as a production-deployed, independently certified, or statistically benchmarked evaluation system.

Technical artifacts

Inspect the implementation behind the case study.

The case study is intentionally readable at a high level. The evaluator specification, test suite, failure/fix record, and representative annotated evaluation are available below for deeper inspection.

Artifact 01

Final Evaluator Prompt

The complete evaluation specification used in the final tested portfolio prototype.

Artifact 02

Five-Case Test Suite

A compact record of the standard, subtle, strict-format, adversarial, and insufficient-evidence tests.

Artifact 03

Failure / Fix Matrix

The design weaknesses discovered during testing and the rules added to address them.

Artifact 04

Annotated Sample Evaluation

A representative source-grounded evaluation showing evidence, severity, and decision logic.

What this project demonstrates

LLM EvaluationAI Quality AssurancePrompt TestingRubric DesignHallucination DetectionError ClassificationInstruction HierarchySource GroundingAdversarial TestingFailure AnalysisStructured RevisionHuman-in-the-Loop AI

Human-review position

The framework is designed as a decision-support system for human reviewers, not as an independent authority. High-stakes outputs still require appropriate human verification.

Limitations

This is an independent tested portfolio prototype. I do not claim production deployment, commercial client adoption, large-scale inter-rater studies, cross-model benchmarking, or independently measured accuracy improvements. The test suite is deliberately constructed to demonstrate evaluation design and failure analysis, not to establish universal reliability.