AI Response Evaluation & RLHF Dataset Creation for Financial Fraud Detection
Served as a subject matter expert and data labeling specialist on a contract AI training project focused on financial anomaly detection and fraud prevention. Responsibilities included: - Evaluating and ranking AI-generated responses using rubric-based frameworks, assessing outputs for accuracy, reasoning quality, coherence, and instruction-following - Writing structured preference justifications explaining ranking decisions to support downstream RLHF fine-tuning of large language models - Authoring complex domain-specific prompts and crafting high-quality reference responses to challenge and improve model reasoning in billing and fraud detection scenarios - Flagging edge cases, hallucinations, and systematic failure patterns across annotation batches, contributing actionable feedback to guideline refinement