Jennifer Moran

Key Skills

Software

Sama

Mindrift

Micro1

Scale AI

Snorkel AI

Surge AI

Toloka

Top Subject Matter

LLM (Large Language Model) Evaluation and Alignment

LLM Quality Assurance and Policy Evaluation

AI Quality Assurance and Evaluation

Top Data Types

Text

Document

Image

Top Task Types

Bounding Box

Entity Ner Classification

Question Answering

Text Summarization

Transcription

Data Collection

Classification

Freelancer Overview

AI Evaluation Analyst. Brings 6+ years of professional experience across legal operations, contract review, compliance, and structured analysis. Core strengths include Internal, Proprietary Tooling, and Sama. Education includes Doctor of Philosophy, University of Washington (2024) and Master of Science, Adelphi University (2024). AI-training focus includes data types such as Text and labeling workflows including Evaluation and Rating.

Labeling Experience

AI Evaluation Analyst

Text

As an AI Evaluation Analyst at RWS, I performed structured evaluation of large language model outputs for accuracy, relevance, and alignment. My work focused on designing and refining evaluation rubrics to guide objective model assessment, with particular attention to reasoning failures and policy compliance. I supported model training teams by generating analytical reports and collaborating on domain-specific evaluation criteria. • Assessed generative AI text outputs for logical consistency and behavioral standards. • Designed and customized evaluation rubrics for varied reasoning tasks. • Identified ambiguities, hallucinations, and safety issues in model responses. • Informed benchmarking and continuous model improvement through detailed documentation.

As an AI Evaluation Analyst at RWS, I performed structured evaluation of large language model outputs for accuracy, relevance, and alignment. My work focused on designing and refining evaluation rubrics to guide objective model assessment, with particular attention to reasoning failures and policy compliance. I supported model training teams by generating analytical reports and collaborating on domain-specific evaluation criteria. • Assessed generative AI text outputs for logical consistency and behavioral standards. • Designed and customized evaluation rubrics for varied reasoning tasks. • Identified ambiguities, hallucinations, and safety issues in model responses. • Informed benchmarking and continuous model improvement through detailed documentation.

2023 - Present

LLM Quality Assurance Research Assistant

Text

In my role as an LLM Quality Assurance Research Assistant at Databricks, I led research on evaluation methods for generative AI. I developed structured frameworks to measure alignment, policy adherence, and detect hallucinations in AI systems. My comparative studies contributed both to academic research and industry technical briefs. • Developed frameworks and rubrics to evaluate LLM reasoning and safety. • Conducted comparative evaluations to identify performance gaps. • Measured alignment and hallucination rates across multiple models. • Produced reports for PhD outputs and technical reviews.

In my role as an LLM Quality Assurance Research Assistant at Databricks, I led research on evaluation methods for generative AI. I developed structured frameworks to measure alignment, policy adherence, and detect hallucinations in AI systems. My comparative studies contributed both to academic research and industry technical briefs. • Developed frameworks and rubrics to evaluate LLM reasoning and safety. • Conducted comparative evaluations to identify performance gaps. • Measured alignment and hallucination rates across multiple models. • Produced reports for PhD outputs and technical reviews.

2023 - 2024

AI Quality & Evaluation Associate

SamaText

As an AI Quality & Evaluation Associate at Sama, I contributed to the assessment and improvement of AI systems through compliance monitoring and documentation. My work involved analyzing ambiguous output patterns, inconsistent reasoning, and integrating new insights into testing frameworks. Peer review of evaluation methodologies and participation in research were central to this role. • Participated in identifying failure modes and reasoning chains in AI outputs. • Documented ambiguous/inconsistent system behaviors for quality improvement. • Helped improve testing frameworks with research-backed recommendations. • Engaged in peer review and research documentation on AI assessment.

As an AI Quality & Evaluation Associate at Sama, I contributed to the assessment and improvement of AI systems through compliance monitoring and documentation. My work involved analyzing ambiguous output patterns, inconsistent reasoning, and integrating new insights into testing frameworks. Peer review of evaluation methodologies and participation in research were central to this role. • Participated in identifying failure modes and reasoning chains in AI outputs. • Documented ambiguous/inconsistent system behaviors for quality improvement. • Helped improve testing frameworks with research-backed recommendations. • Engaged in peer review and research documentation on AI assessment.

2021 - 2023

Data & AI Quality Review Assistant

Text

At Microsoft as a Data & AI Quality Review Assistant, I supported large-scale data quality validation for AI and ML systems. My main responsibilities included reviewing annotated datasets, detecting inconsistencies, and ensuring guideline compliance. I regularly documented findings and contributed to the improvement of annotation standards and model reliability. • Evaluated structured and unstructured datasets for annotation accuracy. • Detected data inconsistencies and labeling errors impacting model performance. • Assessed task completeness using review checklists and rubrics. • Provided feedback through remote reporting workflows for model enhancement.

At Microsoft as a Data & AI Quality Review Assistant, I supported large-scale data quality validation for AI and ML systems. My main responsibilities included reviewing annotated datasets, detecting inconsistencies, and ensuring guideline compliance. I regularly documented findings and contributed to the improvement of annotation standards and model reliability. • Evaluated structured and unstructured datasets for annotation accuracy. • Detected data inconsistencies and labeling errors impacting model performance. • Assessed task completeness using review checklists and rubrics. • Provided feedback through remote reporting workflows for model enhancement.

2019 - 2021

Education

Work History