Skip to content

[Dataset]Add GAIA Datasets #2051

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Conversation

domonic18
Copy link

Motivation

Add GAIA dataset support for OpenCompass
为 OpenCompass 添加 GAIA 数据集支持

Modification

中文说明

  • 新增opencompass/datasets/gaia.py 文件,包含从Huggingface或者从本地加载数据集的过程
  • 新增opencompass/configs/datasets/GAIA/gaia_gen.py,包含gaia_reader_cfg、gaia_infer_cfg、gaia_eval_cfg的配置
  • 更新opencompass/datasets/init.py,增加from .gaia import *
  • 更新opencompass/utils/datasets_info.py,增加# GAIA Datasets
  • 更新dataset-index.yml,增加GAIA数据集信息

英文说明
Added opencompass/datasets/gaia.py to implement dataset loading from Huggingface or local storage
Added opencompass/configs/datasets/GAIA/gaia_gen.py containing configurations for gaia_reader_cfg, gaia_infer_cfg, and gaia_eval_cfg
Updated opencompass/datasets/init.py with from .gaia import *
Added # GAIA Datasets section in opencompass/utils/datasets_info.py
Updated dataset-index.yml with GAIA dataset metadata

BC-breaking (Optional)

影响:无向后兼容性破坏
理由:本次修改为纯新增功能,未改动现有接口和配置文件结构

Impact: No backward compatibility breaks
Reason: This is a pure feature addition without modifying existing interfaces or config structures

Use cases (Optional)

1、通过命令下载数据集到COMPASS_DATA_CACHE

modelscope download --dataset AI-ModelScope/GAIA --local_dir ./opencompass/data/GAIA

2、通过命令测试API方式调用gaia_gen数据集的过程

opencompass --models custom_api --datasets gaia_gen --debug

1、Download dataset to COMPASS_DATA_CACHE

modelscope download --dataset AI-ModelScope/GAIA --local_dir ./opencompass/data/GAIA

2、Test API-based dataset loading for gaia_gen

opencompass --models custom_api --datasets gaia_gen --debug

Checklist

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues.
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
  • The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • CLA has been signed and all committers have signed the CLA in this PR.

deadwalks and others added 8 commits April 10, 2025 11:04
* 'main' of https://github.com/domonic18/opencompass:
  [Refactor] Refactorize openicl eval task (open-compass#1990)
  [ci] update baseline for kernal change of vllm and lmdeploy (open-compass#2011)
  [Feature] Make dump-eval-details default behavior (open-compass#1999)
  [Fix] OpenICL Math Evaluator Config (open-compass#2007)
  [Feature] Add CascadeEvaluator (open-compass#1992)
  [Dataset] Add MedXpertQA (open-compass#2002)
  [Dataset] Update dingo 1.5.0 (open-compass#2008)
  [CI] fix baseline score (open-compass#2000)
  [Doc] Fix links between zh & en (open-compass#2001)
* 'main' of https://github.com/domonic18/opencompass:
  [Feature] LLM Judge sanity check (open-compass#2012)
  [Fix] fix order bug Update arena_hard.py (open-compass#2015)
  [Fix] Fix compare error when k is list in base_evaluator (open-compass#2010)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants