AnaFig: A Human-Aligned Dataset for Scientific Figure Analysis

Introduction

Scientific Figure Analysis (SFA) is a challenging task requiring the extraction of analytical insights from figures by integrating visual and textual inputs, surpassing surface-level descriptions in conventional tasks like captioning. It demands visual recognition, scientific knowledge integration, contextual reasoning, and multimodal comprehension . While multimodal large language models (MLLMs) have advanced in image-to-text tasks, their ability to perform high-level scientific analysis remains unclear. Existing datasets focus on descriptive accuracy rather than analytical depth, lacking scientific context for evaluating reasoning about complex data .

To address this, we introduce AnaFig, a dataset designed to assess three key MLLM capabilities: adherence to complex scientific instructions, multimodal perception, and analytical summarization. Comprising 1,000 samples from eight physics subfields, each sample pairs figures with descriptive text from research papers, requiring analytical summaries synthesizing key insights . A five-dimensional scoring framework (faithfulness, completeness, conciseness, logicality, depth of analysis) ensures rigorous evaluation, with human-expert summaries and 5,000 score labels establishing a benchmark. Testing five MLLMs reveals performance gaps in analytical depth compared to human experts, highlighting the need for improved scientific reasoning in MLLMs .

Importance of descriptive contextual information on the quality of analytical summaries. Different color fonts represent the corresponding different qualities of the generated content.

AnaFig-Dataset Overview

The AnaFig dataset is designed to evaluate multimodal large language models (MLLMs) in Scientific Figure Analysis (SFA), focusing on three core capabilities: following complex instructions, multimodal perception, and analytical summarization.

• 1,000 high-quality samples from 8 physics subfields
• Multimodal input: figure + caption + contextual text
• 5-dimensional evaluation framework (Faithfulness, Completeness, Conciseness, Logicality, Depth of Analysis)
• 5,000 expert-assigned score labels

Statistics of figure application domains, an example input of AnaFig.

Examples of figure types.

Annotation and Scoring Process of AnaFig Dataset

The annotation process of AnaFig involves ten physics experts, with five acting as initial annotators to generate summaries based on established criteria and the other five as checkers to independently evaluate and score all 1,000 data samples, where summaries scoring 3 or lower on any of the five dimensions (faithfulness, completeness, conciseness, logicality, depth of analysis) are iteratively revised until all dimensions achieve a score of 4 or higher; the five-dimensional evaluation criteria include faithfulness (strictly adhering to the information in figures and descriptions), completeness (encompassing all key information and trends in figures), conciseness (avoiding redundant information), logicality (being logically coherent and consistent with expert knowledge), and depth of analysis (providing insightful and thorough data understanding), with checkers assigning scores from 1 to 5 for each criterion to ensure high-quality, human-aligned summaries and evaluations.

AnaFig dataset annotation and scoring process.

Detailed evaluation criteria.

Benchmark Results

Results of various evaluation methods in summarization-level. MET.=METEOR, R1=ROUGE1, R2=ROUGE2, RL=ROUGEL:

MLLM Score in ffve-dimensional evaluation：

Get Started

We are making the code and dataset open source on github

Quick Start

                        # Clone the Git environment and download the dataset images.
                        git clone https://github.com/yuetanbupt/AnaFig.git
                        cd images
                        wget https://drive.usercontent.google.com/download?id=1szWDGkZXbw67u9WGy_qrs8GNjRaiFFPg&export=download&confirm=t&uuid=8222368e-2802-4aea-8f99-3a30237bfc8a
                        unzip AnaFig-image.zip && rm AnaFig-image.zip

                        # Install dependencies
                        pip install -r requirements.txt

                        # Call the API to run a closed - source model for summary generation
                        python model/API_gen.py \
                            --api_link $api_link \
                            --model_name $model_name \
                            --api_key $openai_key
                        
                        # Call the API to run a closed - source model to score the previously generated summarization
                        python model/API_eval.py \
                            --file_name $file_name \
                            --api_link $api_link \
                            --model_name $model_name \
                            --api_key $openai_key