What it is
EduFig-IC is an image–text consistency benchmark built over STEM figure-based exam questions.
Each sample is a question (stem + figure), and the task is to decide whether the figure is consistent with the stem — exactly the fine-grained ability where today's multimodal LLMs (LMM/VLM) most often fail and which is hardest to evaluate: models "describe the picture fluently yet ignore the one anchor that actually matters."
The defining feature is graded difficulty: inconsistency is not binary but constructed in three systematic levels — wrong-figure → parameter mismatch → logically unsolvable — so a model's image–text reconciliation ability can be measured level by level.
How it was built
Every erroneous sample in EduFig-IC was produced automatically by
mm_dataset_factory,
a multimodal data-production platform I built end-to-end: structuring Word exam papers → VLM error planning at three levels → AI image editing → a planner–judge quality-gate loop.
Full technical docs:
documentation repo.
Datasheet
- Task: given (stem text, figure), output whether they are consistent; inconsistent samples are further labelled with a difficulty level and a concrete, explainable reason.
- Pos / neg: 669 positive (inconsistent) vs. 304 negative (the original consistent controls) — supports both binary and graded evaluation out of the box.
- Subjects: materials/crystallography, automatic control, organic chemistry, physics (optics), fluid mechanics — STEM figure-based subjective questions.
- Fields: stem, figure, error level,
err_reason, correction, generation model, generation metadata — fully traceable per sample.
Difficulty levels
L0 · consistent original matching control
L1 · wrong figure right question, figure swapped from another
L2 · parameter mismatch one anchor in the figure conflicts with the stem
L3 · logically unsolvable figure edited to violate the problem's physics/math
L1 tests global semantic mismatch (relatively easy); L2 collapses the error to one identifiable local anchor (a label, a number), forcing literal cross-checking; L3 injects a contradiction at the structural / logical level, requiring the model to understand what the figure means rather than do surface OCR. Difficulty increases accordingly.
Sample viewer
Five representative samples below (click any image to enlarge). Each shows the original (consistent) figure alongside progressively harder L2 / L3 inconsistent versions, with the exact cause of every error.
Sample AMaterials · Crystal structure
StemThe sphalerite (ZnS) structure is shown. Its Bravais lattice is ( ) A. simple cubic B. face-centered cubic C. hexagonal close-packed
Original L0 consistent
Legend reads Zn, S, consistent with "sphalerite (ZnS)".
L2 parameter mismatch local anchor
Legend changed from Zn to Fe, contradicting the ZnS composition; layout unchanged, so only a literal check of the legend reveals it.
L3 logically unsolvable structural
A non-nearest-neighbour long-range bond is forced between body-center S and corner Zn, breaking sphalerite's tetrahedral coordination — the lattice can no longer be determined from the figure.
✔ Answer: B. face-centered cubic
Sample BAutomatic control · Circuit element
StemThe circuit of a control element is shown; from its transfer function it is a typical ( ) element. A. proportional B. inertial C. first-order differential D. dead-time
Original L0 consistent
R-C series circuit, capacitor labelled C, giving the inertial transfer function 1/(RCs+1).
L2 parameter mismatch local anchor
Capacitor label changed from C to L, conflicting with the RC implied by the transfer function.
L3 logically unsolvable structural
Topology changed to RC parallel; the transfer function degenerates to 1, structurally contradicting the given inertial formula 1/(RCs+1).
✔ Answer: B. inertial element
Sample COrganic chemistry · Reaction mechanism
StemFor the following electrophilic substitution, which substituent makes the reaction fastest? ( ) A. Br B. CH₃ C. OCH₃ D. COCH₃
Original L0 consistent
Friedel–Crafts acylation of a ring bearing a generic substituent R, consistent with "compare substituents".
L2 parameter mismatch local anchor
Substituent R replaced by a specific NO₂, which no longer maps onto the options (Br/CH₃/OCH₃/COCH₃).
L3 logically unsolvable structural
The bond between R and the ring is broken; the reactant is incomplete and has no electrophilic site, so the premise collapses.
✔ Answer: C. OCH₃ (strong activator, ortho/para director — fastest)
Sample DPhysics · Wave optics
StemNatural light of intensity I₀ passes through polarizers P₁ then P₂ whose transmission axes are 30° apart. The transmitted intensity I = ( ) A. I₀/4 B. √3·I₀/4 C. √3·I₀/2 D. 3I₀/8
L2 parameter mismatch local anchor
The figure labels the angle as 45°, inconsistent with the stem's 30°, directly affecting the Malus-law result.
L3 logically unsolvable structural
P₁, P₂ are redrawn orthogonal (90°) and concentric while still labelled 30° — geometrically self-contradictory.
✔ Answer: D. 3I₀/8 (natural light halved by P₁, then Malus cos²30°)
Sample EFluid mechanics · Jet impact
StemA jet of speed v strikes an inclined plate at angle α = 60° (no head loss, no friction, momentum factor 1). Prove that the split-flow ratio Q₁/Q₂ = 3.
L2 parameter mismatch local anchor
The figure states the flow ratio as 2, contradicting the result of 3 that the problem asks to prove.
L3 logically unsolvable structural
The ratio is changed to ∞, implying infinite split flow — violating mass conservation in jet impact, physically impossible.
✔ Result: Q₁/Q₂ = (1+cos α)/(1−cos α) = 3
Data availability
On the full dataset
The source papers and full set of image–text samples in EduFig-IC are subject to the data owner's rights and are
not publicly released. This page shows five representative samples to illustrate how the benchmark is constructed and graded. For the evaluation protocol, field definitions, or a collaborative evaluation, please reach out via
GitHub.
Cite & related
