Senior Software Engineer, AI Eval jobs in United States
cer-icon
Apply on Employer Site
company-logo

Sentry · 16 hours ago

Senior Software Engineer, AI Eval

positionSan Francisco, CA
timeFull-time
remoteHybrid
senioritySenior Level
money$240K/yr - $280K/yr
date5+ years exp

Sentry is on a mission to help developers write better software faster by building performance and error monitoring tools. The Senior Software Engineer on the AI/ML team will be responsible for developing evaluation infrastructure that measures the accuracy and reliability of AI systems, ensuring their correct and predictable behavior as they scale.

Application Performance ManagementDeveloper ToolsReal TimeWeb Apps
check
Growth Opportunities
check
H1B Sponsor Likelynote

Responsibilities

Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

Qualification

AI/ML experiencePythonTypeScriptData infrastructureEvaluation techniquesCross-functional collaborationAttention to detail

Required

Minimum 5+ years of professional experience with a Bachelor's degree in computer science, machine learning, or a related field
Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
Comfort writing production-quality code (we use Python and TypeScript)
Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)

Preferred

Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools

Benefits

Incentive compensation
Equity grants
Paid time off
Group health insurance coverage

Company

Sentry

twittertwittertwitter
Glassdoor4.3
company-logo
Sentry is a developer of an application monitoring platform that helps developers optimize the performance of their code.
dateFounded in 2012
location
San Francisco, California, USA
people-size201-500 employees
web-link
https://www.sentry.io

H1B Sponsorship

Sentry has a track record of offering H1B sponsorships. Please note that this does not guarantee sponsorship for this specific role. Below presents additional info for your reference. (Data Powered by US Department of Labor)
Distribution of Different Job Fields Receiving Sponsorship
Represents job field similar to this job
Trends of Total Sponsorships
2025 (4)
2024 (4)
2023 (3)
2022 (12)
2021 (6)
2020 (4)

Funding

Current Stage
Late Stage
Total Funding
$217M
Key Investors
AccelNew Enterprise Associates
2022-05-04Series E· $90M
2021-02-18Series D· $60M
2019-09-24Series C· $40M

Leadership Team

leader-logo
Milin Desai
Chief Executive Officer
linkedin
leader-logo
Chris Jennings
Co-founder, Chief Creative Officer
linkedin

Recent News

Accel
Accel Portfolio - Sentry.
2026-01-06
The New Stack
Agent Skills: Anthropic’s Next Bid to Define AI Standards
2025-12-19
SiliconANGLE
From observability to reasoning: How AI agents are catching bugs before they ship
2025-12-06
Company data provided by crunchbase