AI Training / Annotation Specialist – RLHF Tasks (Scale Labs)
Scale AITextRLHF
Led and contributed to Reinforcement Learning from Human Feedback (RLHF) tasks for training large language models. Developed and enforced detailed annotation guidelines and scoring rubrics to enhance evaluator consistency. Conducted hands-on annotation and evaluation including prompt evaluation, response ranking, safety review, and preference modeling. Improved training outcomes by tracking quality signals and refining guidelines.
• Evaluated and ranked model outputs for prompt response quality.
• Performed safety and policy-aligned judgement on ambiguous or edge-case outputs.
• Created guidelines to minimize subjective annotation variance.
• Analyzed error patterns and task rework to iteratively improve training results.
Led and contributed to Reinforcement Learning from Human Feedback (RLHF) tasks for training large language models. Developed and enforced detailed annotation guidelines and scoring rubrics to enhance evaluator consistency. Conducted hands-on annotation and evaluation including prompt evaluation, response ranking, safety review, and preference modeling. Improved training outcomes by tracking quality signals and refining guidelines. • Evaluated and ranked model outputs for prompt response quality. • Performed safety and policy-aligned judgement on ambiguous or edge-case outputs. • Created guidelines to minimize subjective annotation variance. • Analyzed error patterns and task rework to iteratively improve training results.
2025 - Present