People In AI · 2 days ago
AI Data Partner
NYC Metro Area
Full-time
Hybrid
Mid, Senior Level
$300K/yr - $800K/yrPeople In AI is a specialist recruitment firm connecting exceptional talent with high-impact roles across the AI and machine learning ecosystem. The role involves working with researchers and domain experts to create trusted evaluation datasets and frameworks for AI systems, directly impacting their reliability in high-stakes environments.
Staffing & Recruiting
Hiring Manager
Cassie Wandell
Responsibilities
Work directly with researchers and domain experts to define what "good" looks like for AI outputs
Source, clean, and validate unstructured data for LLM evaluation and benchmarking
Build and maintain golden datasets grounded in expert qualitative judgment
Investigate failures, edge cases, and ambiguous outputs to improve data quality
Prepare datasets, annotations, and validation checks for offline evaluation
Iteratively refine data assets as understanding evolves—avoiding premature automation
Design and evolve evaluation data frameworks and scalable benchmarks
Define principles for translating qualitative judgment into metrics
Identify when workflows are ready to scale vs. remain experimental
Detect drift, regressions, and subtle failure modes in evaluation datasets
Mentor other Data Partners and uphold data quality standards
Lead efforts around provenance, auditability, and long-term data trust
Qualification
Required
Deep curiosity about AI behavior, failure modes, and evaluation rigor
Experience with unstructured data, human annotation, or research support
Comfort with ambiguity, iteration, and high-trust collaborative environments
Strong analytical and qualitative reasoning skills
Ability to operationalize expert insights into structured, scalable formats
Experience leading complex evaluation or data quality projects (for senior-level candidates)
Proven ability to mentor and set standards in fast-moving environments (for senior-level candidates)
Work directly with researchers and domain experts to define what 'good' looks like for AI outputs
Source, clean, and validate unstructured data for LLM evaluation and benchmarking
Build and maintain golden datasets grounded in expert qualitative judgment
Investigate failures, edge cases, and ambiguous outputs to improve data quality
Prepare datasets, annotations, and validation checks for offline evaluation
Iteratively refine data assets as understanding evolves—avoiding premature automation
Design and evolve evaluation data frameworks and scalable benchmarks
Define principles for translating qualitative judgment into metrics
Identify when workflows are ready to scale vs. remain experimental
Detect drift, regressions, and subtle failure modes in evaluation datasets
Mentor other Data Partners and uphold data quality standards
Lead efforts around provenance, auditability, and long-term data trust
Preferred
Experience leading complex evaluation or data quality projects (for senior-level candidates)
Proven ability to mentor and set standards in fast-moving environments (for senior-level candidates)
Company
People In AI
At People in AI, we specialize in staffing solutions for the rapidly expanding AI sector.
Founded in 2023
2-10 employees
Funding
Current Stage
Early StageCompany data provided by crunchbase