AI Product Analyst - AI Receptionist Team
About The Position
At MEDvidi, we are scaling AI-powered systems across critical parts of our product and operations.
As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge.
We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale.
This role is focused on analyzing real AI system behavior in production environments:
- understanding product and operational impact,
- identifying failure patterns on generational tasks,
- measuring quality and reliability of outputs,
- helping teams make accurate decisions under ambiguity.
Responsibilities
- Analyze production behavior of AI agents, automation systems, and LLM-powered workflows,
- Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys,
- Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights,
- Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability,
- Build monitoring systems and dashboards for AI performance visibility,
- Identify operational and product risks introduced by automation,
- Measure the impact of AI systems on business metrics, operational efficiency, and user experience,
- Detect data quality issues, shifts, and anomalies affecting AI system performance,
- Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives,
- Support rapid decision-making in fast-changing and ambiguous environments.
Requirements
- 3+ years of experience in analytical roles related to AI systems, ML products, automation platforms, or complex production systems,
- Strong Python skills for deep analytical investigations and working with large-scale noisy text data,
- Advanced SQL skills,
- Understanding of LLM limitations,hallucinations, evaluation challenges, prompt sensitivity, model trade-offs, and AI failure modes,
- Experience building monitoring systems, analytical frameworks, or operational dashboards for production systems,
- Ability to work with incomplete, ambiguous, and rapidly changing data environments,
- Strong ownership mindset and ability to independently drive investigations and analytical conclusions,
- Ability to connect AI system quality with business and operational impact,
- Fluent Russian; English level B2 or higher,
- Ability to work CET-overlapping hours.
Nice to Have
- Experience with conversational AI or AI agents,
- Experience evaluating LLM outputs or annotation quality,
- Experience with AI quality metrics and evaluation methodologies,
- Experience working with Langchain infrastructure,
- Experience with ML data pipelines or synthetic data generation,
- Background in anomaly detection.
What we offer
- Work on a modern AI product where quality is a strategic function, not a formality
- Real influence on product direction and engineering practices
- High-impact work in a product-driven medical tech company
- Startup environment: fast iteration, high ownership, minimal bureaucracy
- Competitive compensation and growth opportunities
Benefits
- Health insurance after probation period
- Sports & wellness compensation
- Fully remote, long-term B2B collaboration
- 19 working days of vacation per year
- 3 additional wellness days per year
- Paid sick leave for the first 5 working days
- Gifts to employees for important events
- Career Development
- Personalized English lessons via Preply
Ready to make an impact? Send us your profile - let’s build something meaningful together.
Apply for this position
Ready to join?
- Explore open roles and apply
- Talk to our HR manager
- Talk to our Head of MedOps
- Get an offer!