AI Manager API/LLM Orchestration Lead (GlueX Platform) – Data Labeling & Evaluation
Owned orchestration, fine-tuning, and evaluation of LLMs powering the AI Manager API for GlueX platform's multi-provider text and conversational agents. Designed and deployed a semantic edit and embedding search flow that selects artifacts, generates labels, and propagates consistency patches based on model confidence scoring. Maintained authoritative JSON schemas and OpenSpec change requests to manage edit surface area, evaluating LLM and agent output for enterprise workflows. • Managed prompt engineering, semantic search, and edit propagation for AI-generated text. • Orchestrated and fine-tuned multiple LLM providers for label consistency and accuracy. • Used Arize Phoenix, Sentry, and Blinker for production tracing and monitoring. • Oversaw evaluation, red teaming, and continuous improvement of LLM outputs.