Modicus Prime · 1 day ago
AI Evaluation Engineer
Austin, TX
Full-time
Hybrid
Mid Level
3+ years expModicus Prime is improving the quality of pharmaceutical drug manufacturing and saving patient lives by leveraging the power of compliant AI and advanced analytics. They are seeking an AI Evaluation Engineer responsible for evaluating the technical suitability, risks, and limitations of AI models and AI-enabled products in regulated environments, focusing on automating compliance.
Artificial Intelligence (AI)Biopharma
No H1B
Responsibilities
Evaluate and document AI model origin, derivation, dependencies, versioning, and change behavior for externally supplied AI models and AI-enabled products
Familiar with software release practices and modern tools such as Github releases, server containerization, python software packaging, and Jira reporting
Assess high-level training approaches and data characteristics, including disclosed limitations, biases, and data governance practices, without requiring access to proprietary datasets
Evaluate model behavior, known failure modes, performance in the intended context of use, reproducibility, and limitations, including stratified performance where applicable
Assess explainability, human oversight models, and human factors risks relevant to appropriate use and reliance
Evaluate AI security and abuse-resistance controls, including access management, logging, monitoring, and susceptibility to AI-specific threats
Assess integration and operational fit, including preprocessing and post-processing controls, scalability, performance, and cost predictability
Evaluate readiness for monitoring, drift detection, traceability, artifact retention, and model retirement to support audits and investigations
Document technical assessment conclusions and communicate identified risks and limitations to VP of GxP Quality and Compliance for inclusion in overall risk assessments
Maintain current knowledge of AI technical risk areas, development practices, and industry best practices relevant to regulated use
Other duties as assigned as needed
Qualification
Required
Evaluate and document AI model origin, derivation, dependencies, versioning, and change behavior for externally supplied AI models and AI-enabled products
Familiar with software release practices and modern tools such as Github releases, server containerization, python software packaging, and Jira reporting
Assess high-level training approaches and data characteristics, including disclosed limitations, biases, and data governance practices, without requiring access to proprietary datasets
Evaluate model behavior, known failure modes, performance in the intended context of use, reproducibility, and limitations, including stratified performance where applicable
Assess explainability, human oversight models, and human factors risks relevant to appropriate use and reliance
Evaluate AI security and abuse-resistance controls, including access management, logging, monitoring, and susceptibility to AI-specific threats
Assess integration and operational fit, including preprocessing and post-processing controls, scalability, performance, and cost predictability
Evaluate readiness for monitoring, drift detection, traceability, artifact retention, and model retirement to support audits and investigations
Document technical assessment conclusions and communicate identified risks and limitations to VP of GxP Quality and Compliance for inclusion in overall risk assessments
Maintain current knowledge of AI technical risk areas, development practices, and industry best practices relevant to regulated use
Bachelor's degree in Computational Sciences, Engineering, Information Systems, or a related technical field, or equivalent combination of education and relevant professional experience
Minimum of three (3) years of experience assessing, developing, testing, or supporting AI-enabled or software systems in regulated, safety-critical, or high-risk environments, with a demonstrated ability to evaluate technical risk, document findings, and support audit-ready decision-making
Familiarity with version control and release management systems, such as Github, Github Actions or equivalent
Experience with project management tracking and reporting software, such as Jira
Knowledge of text-based utilities such as bash
Familiarity with MLflow, Weights and Biases, and OpenTelemetry to ensure reliable model evaluation with appropriate metrics
Hands-on experience contributing to software, AI, or data-driven systems through coding, testing, or technical problem-solving
Understanding of AI model lifecycle concepts, including training approaches, inference, versioning, monitoring, drift, and retirement
Ability to assess supplier and internal technical documentation without access to proprietary algorithms or training data
Working knowledge of key AI risk domains, including bias, robustness, explainability, security, and human oversight
Strong analytical, documentation, and communication skills, with the ability to produce clear, audit-defensible technical assessments
Ability to work independently, manage multiple priorities, and contribute effectively across assurance and development activities
Preferred
Location: Austin preferred
Benefits
Comprehensive medical, dental, and vision coverage
401(k) retirement plan
Company
Modicus Prime
Modicus Prime enables pharmaceutical manufacturers to deploy AI systems that are auditable and compliant by design.
11-50 employees
Funding
Current Stage
Early StageTotal Funding
$4.7MKey Investors
Silverton PartnersGranatus VenturesSmartGateVC
2024-07-15Seed· $3.5M
2024-02-07Seed
2022-12-08Pre Seed· $1.2M
Leadership Team
Taylor Chartier
Chief Executive Officer
Recent News
2025-05-06
Triple S Ventures
2025-05-06
2025-02-10
Company data provided by crunchbase