About this opportunity
24-Mag Llc lists this Remote | Medicine Physician (MD/DO/Doctoral study/PhD) opportunity in workfromhome, New York. Review the employer’s description below for duties, qualifications and application requirements.
Job description
We are sharing a specialised part-time opportunity for licensed physicians in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario development, structured assessment, and expert review of model performance.
Selected physicians will work with research teams to evaluate how AI systems approach realistic medical problems, identify gaps in clinical knowledge and reasoning, and help develop evaluation methods that reflect the complexity and nuance of evidence-based clinical practice.
Key Responsibilities
Clinical Reasoning Evaluation
Evaluate AI performance on realistic medical problems
Assess the quality of clinical reasoning and decision-making
Identify gaps in medical knowledge or clinical judgement
Review model conclusions against evidence-based practice
Distinguish significant clinical errors from minor limitations
Clinical Scenario Development
Create realistic scenarios for medical AI evaluation
Design cases requiring nuanced clinical judgement
Incorporate diagnostic and management decision points
Develop scenarios reflecting real-world clinical complexity
Ensure cases meaningfully test model reasoning capabilities
Evaluation Framework Design
Design systematic frameworks for medical AI assessment
Define criteria for evaluating clinical reasoning quality
Develop methods that capture nuance in clinical practice
Support consistent assessment across repeated evaluation tasks
Refine evaluation methods based on observed model performance
Medical Knowledge & Decision Analysis
Assess model understanding of clinical concepts
Evaluate diagnostic and treatment-related reasoning
Identify unsupported assumptions or missing considerations
Apply specialty-specific expertise where relevant
Provide structured rationale for evaluation decisions
Research Collaboration & Feedback
Collaborate with research teams on medical AI evaluation
Communicate clinical findings clearly and concisely
Provide actionable feedback on model strengths and limitations
Contribute to strategies for improving medical model performance
Support development of clinically meaningful benchmarks
Ideal Profile
Licensed physician currently engaged in active clinical practice
MD, DO, or equivalent medical qualification
Physicians from any medical specialty may be considered
Strong experience with clinical decision-making
Solid understanding of evidence-based medicine
Ability to assess complex medical reasoning objectively
Strong analytical and critical-thinking capabilities
Excellent written and verbal communication skills
Comfortable explaining nuanced clinical decisions clearly
Ability to identify gaps in medical knowledge or reasoning
Interest in how AI can support clinical practice
Comfortable contributing to structured model-evaluation workflows
Strong problem-solving abilities
Able to collaborate effectively with research teams
Comfortable working independently in a remote environment
Able to integrate project work around existing clinical commitments
Engagement Details
Part-time remote engagement
Flexible scheduling
Commitment of up to 30 hours per week
Initial project duration of approximately 1 month
Extension may be available depending on performance and project fit
Work will focus on clinical reasoning evaluation, scenario development, and medical benchmark design
Physicians from any clinical specialty may be considered
Scheduling is intended to accommodate ongoing clinical commitments
Compensation is not specified in the source materials
Work must be completed without using confidential, proprietary, patient-identifiable, protected health, unpublished clinical, or otherwise restricted information belonging to any patient, employer, healthcare organisation, research institution, client, or other third party
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
#J-18808-Ljbffr
Worksite address
workfromhome, NY, 10261, US
Who can apply
Review the original listing for work authorization, qualifications and employer requirements.