CereCore · 22 hours ago
Conversational AI Business Quality Analyst
Nashville, TN
Full-time
Onsite
Mid Level
3+ years expCereCore is seeking a Conversational AI Business Quality Analyst who plays a critical role in ensuring the quality, performance, and customer experience of Virtual Agent and IVR solutions. This position involves business analysis, quality assurance, and collaboration with various teams to support the full lifecycle of conversational solutions.
ConsultingInformation ServicesInformation Technology
Responsibilities
Review and evaluate all Virtual Agent (VA) interactions using standardized scoring criteria and audit rubrics
Ensure 100% interaction coverage, validating that each experience meets approved quality, design, and compliance standards
Identify and document intent misclassifications, Natural Language Understanding (NLU) gaps, Automatic Speech Recognition (ASR) failures
Assess conversational user experience (UX), including, turn efficiency and clarity, prompt effectiveness, customer effort and friction points
Validate accurate task completion and proper backend logic execution across all supported journeys
Confirm correct handling of fallback, recovery, and escalation scenarios, including transfers to live agents where applicable
Monitor and validate authentication workflows to ensure compliance with business and regulatory requirements
Identify, document, and escalate potential compliance risks or incidents discovered during interaction reviews
Partner with IT, operations, and compliance stakeholders to support timely investigation and mitigation of identified risks
Document functional and experience defects with clear expected vs. actual behavior, supporting screenshots, transcripts, or call references as needed
Categorize defects using established classifications (e.g., NLU gap, conversation design flaw, logic error, compliance issue)
Create and manage Azure DevOps (ADO) stories for each identified issue, assigning priority using the approved evaluation and prioritization matrix
Collaborate with product, IT, and design teams to clarify defects and recommended remediation approaches
Perform regression testing to validate defect resolution and ensure no unintended impacts to existing functionality
Track and analyze Virtual Agent performance trends
Leverage audit findings and interaction data to surface systemic issues and improvement opportunities
Partner closely with Conversation Designers and Computational Linguists to refine prompts, intents, decision logic, and conversational flows
Produce weekly and monthly operational summary reports highlighting quality trends, high-risk issues, experience breakdowns, recommended actions and prioritization
Support special projects related to Virtual Agent and IVR initiatives as assigned
Assist with training, documentation, and knowledge - sharing efforts as needed
Participate in ad - hoc analysis, initiatives, or process reviews outside primary quality responsibilities
Perform other duties as assigned
Qualification
Required
Bachelor's degree in computer science, Analytics, Information Systems, Business Administration, Engineering, or related field
3+ years in business analysis or product analysis roles required
Contact Center experience is required
Business Analysis & QA
Contact Center & Virtual Agent Expertise
Solid understanding of contact center environments, IVR flows, call routing, and customer experience metrics
Hands on experience supporting or QA testing Virtual Agents, Voicebots, IVR, or CCaaS platforms
Ability to conduct call audits and interaction reviews across multiple platforms to validate performance and experience outcomes
Experience identifying trends in call volume, FSS, containment, transfer rates, and other KPIs
Strong SQL proficiency for validation, querying, and issue investigation
Advanced Microsoft Excel skills for analysis, reporting, and audit tracking
Ability to synthesize quantitative and qualitative data into executive level summaries and insights
Excellent written and verbal communication skills, including the ability to produce clear QA findings and executive summaries
Strong organizational skills with the ability to manage multiple priorities simultaneously
Demonstrated ability to work independently with minimal oversight
High attention to detail and commitment to quality
Customer focused mindset with a drive to exceed expectations and deliver 'wow' experiences
Preferred
2+ years in QA roles within contact center technologies preferred
Experience working in Agile work environments preferred
Experience with data visualization tools like Tableau, Power BI, etc. preferred
Experience in related industries a plus (e.g., healthcare, collections, financial, data analytics and processing)
Experience with virtual agent and IVR systems such as Google CCAI, Dialogflow, Avaya, Noble, or CCaaS platforms preferred
Familiarity with natural language processing (NLP) concepts and conversational design principles
Ability to collaborate effectively with conversation designers, linguists, and product owners
Experience applying design thinking to improve customer journeys
Basic scripting or text manipulation skills (e.g., Python) for data review or QA support
Knowledge of relational database concepts and data processing workflows
Strong stakeholder management and influencing skills, including comfort presenting insights to senior leadership
Initiative to proactively identify issues, own investigations, and drive improvements through resolution
Experience with Google Conversational AI / Dialogflow, Genesys, or similar CCaaS platforms
Familiarity with cloud-based contact center technologies (CCaaS, Voicebots, Virtual Assistants)
Experience using BigQuery, SSMS, or similar data environments
Familiarity with data visualization tools (Power BI, Tableau)
Exposure to test automation tools (e.g., Selenium, QTP, or equivalent)
Experience performing system or performance testing in contact center or digital channels
Prior experience supporting healthcare contact center use cases
Understanding of regulatory or compliance considerations impacting healthcare customer interactions
Company
CereCore
CereCore has implemented EHR systems in more than 300 facilities and offers staffing and remote support services for major EHR acute.
501-1000 employees
Funding
Current Stage
Late StageLeadership Team
Curtis Watkins
President & CEO
Recent News
PR Newswire
2025-11-07
2025-06-07
2025-05-17
Company data provided by crunchbase