Scale AI · 2 weeks ago
Tech Lead/Manager, Machine Learning Research Scientist- LLM Evals
San Francisco Bay Area
Full-time
Onsite
Senior Level
$280K/yr - $380K/yr
5+ years expScale AI is the leading data and evaluation partner for frontier AI companies, dedicated to advancing the evaluation and benchmarking of large language models. As the Tech Lead Manager of the LLM Evals Research team, you will lead a team focused on developing novel evaluation methodologies and metrics to assess the capabilities of cutting-edge LLMs.
AI InfrastructureArtificial Intelligence (AI)Data Collection and LabelingGenerative AIImage RecognitionMachine Learning
Responsibilities
Lead a team of highly effective research scientists and research engineers on LLM evals
Conduct research on the effectiveness and limitations of existing LLM evaluation techniques
Design and develop novel evaluation benchmarks for large language models, covering areas such as instruction following, factuality, robustness, and fairness
Communicate, collaborate, and build relationships with clients and peer teams to facilitate cross-functional projects
Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols
Implement scalable and reproducible evaluation pipelines using modern ML frameworks
Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives
Remain up-to-date on ongoing research in the team, help work through technical challenges, and be involved in design decisions
Remain deeply involved in the research community, both understanding trends, and setting them
Thrive in a high-energy, fast-paced startup environment and are ready to dedicate the time and effort needed to drive impactful results
Qualification
Required
5+ years of hands-on experience in large language model, NLP, and Transformer modeling, in the setting of both research and engineering development
Experience and track of recording in landing major research impacts in a fast-paced environment
Experience supporting and leading a team of research scientists and research engineers
Excellent written and verbal communication skills
Published research in areas of machine learning at major conferences (NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR, etc.) and/or journals
Previous experience in a customer facing role
Benefits
Comprehensive health, dental and vision coverage
Retirement benefits
A learning and development stipend
Generous PTO
Commuter stipend
Company
Scale AI
Scale’s mission is to develop reliable AI systems for the world’s most important decisions.
501-1000 employees
H1B Sponsorship
Scale AI has a track record of offering H1B sponsorships. Please note that this does not
guarantee sponsorship for this specific role. Below presents additional info for your
reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (82)
2024 (54)
2023 (29)
2022 (17)
2021 (10)
2020 (10)
Funding
Current Stage
Late StageTotal Funding
$15.9BKey Investors
MetaAccelTiger Global Management
2025-06-10Corporate Round· $14.3B
2025-06-04Series Unknown
2024-05-21Series F· $1B
Leadership Team
Jason Droege
Interim Chief Executive Officer
Dennis Cinelli
Chief Financial Officer
Recent News
CB Insights
2026-01-09
Crunchbase News
2026-01-07
Company data provided by crunchbase