Skip to content

Support MedMCQA and MedBullets benchmark #2054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

mar-cry
Copy link

@mar-cry mar-cry commented Apr 26, 2025

Motivation

The motivation for this PR is to enrich the evaluation capabilities of existing LLMs in the medical domain. By adding support for two new medical benchmarks, MedMCQA and MedBullets, this PR aims to provide a broader and more comprehensive assessment of LLMs' performance on domain-specific tasks, particularly in the healthcare and clinical fields.

Modification

This PR introduces two new benchmark configuration files for MedMCQA and MedBullets. These additions allow users to easily evaluate LLMs on these datasets by leveraging the existing evaluation framework.

BC-breaking (Optional)

This PR does not introduce any backward compatibility breaking changes. Existing workflows and configurations remain fully functional without any required modifications.

Use cases (Optional)

With the addition of MedMCQA and MedBullets benchmarks:

  • Researchers can benchmark LLMs specifically for medical QA and clinical knowledge tasks.
  • Developers can better understand and improve their models' performance in healthcare-related applications.
  • Facilitates more targeted fine-tuning and domain-specific evaluations.

Checklist

Before PR:

  • ✅ Pre-commit or other linting tools have been used to fix potential lint issues.
  • ✅ Bug fixes are fully covered by unit tests.
  • ✅ The modifications are covered by complete unit tests.
  • ✅ Documentation has been updated accordingly.

After PR:

  • ✅ Potential downstream or related projects have been considered for testing.
  • ✅ CLA has been signed and all committers have signed the CLA for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants