Jonathan Pisone

Key Skills

Software

Scale AI

Top Subject Matter

Large Language Models (LLMs)

AI Safety

Text Generation

Top Data Types

Text

Top Task Types

Data Collection

Classification

Freelancer Overview

AI Prompt Evaluator & Data Annotator Specialist. Brings 5+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Scale AI, Invisible Technologies, and N. Education includes Master of Science, Georgia Institute of Technology (2023) and Bachelor of Arts, University of Georgia (2018). AI-training focus includes data types such as Text and labeling workflows including Evaluation, Rating, and Data Collection.

Labeling Experience

AI Literacy Volunteer

TextData Collection

As an AI Literacy Volunteer for the Atlanta Public Library Tech Connect Program, I led community workshops focused on understanding and responsibly using generative AI. I provided foundational AI literacy training, emphasizing ethical data usage and practical applications of AI tools. These sessions included introduction-level exercises using annotated conversational data. • Educated over 100 community members on safe and effective AI adoption. • Created hands-on, data labeling literacy materials for public use. • Facilitated responsible AI discussions and addressed participant questions. • Developed instructional content for broader AI and data annotation understanding.

As an AI Literacy Volunteer for the Atlanta Public Library Tech Connect Program, I led community workshops focused on understanding and responsibly using generative AI. I provided foundational AI literacy training, emphasizing ethical data usage and practical applications of AI tools. These sessions included introduction-level exercises using annotated conversational data. • Educated over 100 community members on safe and effective AI adoption. • Created hands-on, data labeling literacy materials for public use. • Facilitated responsible AI discussions and addressed participant questions. • Developed instructional content for broader AI and data annotation understanding.

2023 - Present

AI Prompt Evaluator & Data Annotator Specialist

Scale AIText

As an AI Prompt Evaluator & Data Annotator Specialist at Scale AI, I evaluated and ranked LLM-generated responses across various domains. I developed, refined, and applied complex prompt-response rubrics to assess model outputs and conducted adversarial testing to uncover model vulnerabilities. I also mentored junior annotators in correct annotation and taxonomy application. • Evaluated over 1,500 textual responses per month from models such as GPT-4, Claude, and Gemini. • Directly contributed to model alignment improvements and authored over 50 bug reports based on labeling findings. • Trained junior team members in edge-case adjudication and quality assurance best practices. • Achieved a 99% consistency score in quality audits during rating and annotation processes.

As an AI Prompt Evaluator & Data Annotator Specialist at Scale AI, I evaluated and ranked LLM-generated responses across various domains. I developed, refined, and applied complex prompt-response rubrics to assess model outputs and conducted adversarial testing to uncover model vulnerabilities. I also mentored junior annotators in correct annotation and taxonomy application. • Evaluated over 1,500 textual responses per month from models such as GPT-4, Claude, and Gemini. • Directly contributed to model alignment improvements and authored over 50 bug reports based on labeling findings. • Trained junior team members in edge-case adjudication and quality assurance best practices. • Achieved a 99% consistency score in quality audits during rating and annotation processes.

2023 - Present

Creator, LLM Response Quality Analysis Framework

TextClassification

Through the LLM Response Quality Analysis Framework project, I developed and applied a rubric to annotate a dataset of over 2,000 prompt-response pairs. This work focused on rubric-based scoring of AI-generated text for coherence, harmlessness, and instruction-following metrics. The resulting annotations supported university research on AI alignment and response quality. • Designed a detailed evaluation rubric for large-scale AI response annotation. • Scored and labeled responses systematically for research-grade dataset creation. • Contributed to academic insights on LLM behavior and output variability. • Enhanced best practices in prompt-based text evaluation for research use.

Through the LLM Response Quality Analysis Framework project, I developed and applied a rubric to annotate a dataset of over 2,000 prompt-response pairs. This work focused on rubric-based scoring of AI-generated text for coherence, harmlessness, and instruction-following metrics. The resulting annotations supported university research on AI alignment and response quality. • Designed a detailed evaluation rubric for large-scale AI response annotation. • Scored and labeled responses systematically for research-grade dataset creation. • Contributed to academic insights on LLM behavior and output variability. • Enhanced best practices in prompt-based text evaluation for research use.

2023 - 2023

AI Training & Evaluation Contractor

Text

While serving as an AI Training & Evaluation Contractor at Invisible Technologies, I participated in high-priority annotation projects for AI research labs. My work included prompt writing, long-form summarization, and the evaluation of factual consistency in AI outputs. I contributed to the development and refinement of annotation guidelines during weekly calibration sessions. • Collaborated within a dedicated pod on context-sensitive and technical prompt creation. • Maintained a 98% accuracy rate in annotation and evaluation tasks for LLM outputs. • Enhanced project alignment with evolving client objectives through guideline updates. • Focused extensively on text summarization and factual accuracy for AI-generated content.

While serving as an AI Training & Evaluation Contractor at Invisible Technologies, I participated in high-priority annotation projects for AI research labs. My work included prompt writing, long-form summarization, and the evaluation of factual consistency in AI outputs. I contributed to the development and refinement of annotation guidelines during weekly calibration sessions. • Collaborated within a dedicated pod on context-sensitive and technical prompt creation. • Maintained a 98% accuracy rate in annotation and evaluation tasks for LLM outputs. • Enhanced project alignment with evolving client objectives through guideline updates. • Focused extensively on text summarization and factual accuracy for AI-generated content.

2022 - 2022

Education

Work History