π Multimodal Dataset Factory
mm_dataset_factory β Word/DOCX β multimodal error-question image dataset in one click; 4-stage pipeline + SSE real-time streaming + Skill-as-contract delivery architecture
Overview
Targeting multimodal model (LMM/VLM) training data production, the system automatically extracts
image-bearing exam questions from Word/DOCX documents and produces training-ready error-question
image datasets through a 4-stage pipeline:
JSON / JSONL / XLSX annotation files + image directory (zip archive).
Decoupled front/back-end architecture: FastAPI provides async REST API + SSE real-time streaming,
React/Vite provides the UI β launched with a single python run.py.
Tech Stack
4-Stage Production Pipeline
Architecture
Technical Deep-Dive
β Stage 1: Document Structuring (paper-structuring)
Reads Word/DOCX documents and extracts content at the granularity of individual sub-questions (one Sample per sub-question).
Rule: only image-bearing questions are retained; text-only questions are skipped.
Images are re-numbered per question (img_01, img_02 β¦),
and question text retains placeholders in the form <img_01>
to track image-text correspondence in downstream stages.
Images are saved to tasks/{task_id}/images/ as q0001_img_01.png.
Debug artifacts are also written: extracted text, imageMap JSON, raw image backup.
Key Sample fields: sample_no Β· source_file Β· question_no Β· subject Β· question_type Β· question_text Β· image_count Β· image_map
β‘ Stage 2: VLM Error Planning (vlm-error-planning) β L1/L2/L3 Error Taxonomy
Calls the VLM API to analyze each image-bearing question and outputs exactly 3 planning rows per sample (L1 / L2 / L3):
unsolvable_reason. Also enters the evaluation loop; reaches machine_reject after max retries.
Text enrichment mechanism: When an L2/L3 plan involves "modifying a numeric value in the image"
but the question text lacks the corresponding numeric anchor, a controlled text enrichment pass is executed
first and its output is merged into the exported question_text before evaluation.
The enrichment must not be deferred to the image-generation stage β data consistency is enforced at planning time.
Planning artifacts: planning/plan_eval_report.jsonl (severity / decision / judge_trace) + questions_prompt_plan.xlsx/.jsonl
β’ Stage 3: Image Generation (text2image-generation)
Uses questions_prompt_plan.jsonl as input. L2/L3 rows invoke the Text-to-Image API
to generate replacement images; L1 rows directly swap images from the task image library without calling
the generation API. Generated images are saved to tasks/{task_id}/generated/,
indexed by sample_no + error_level.
β£ SSE In-Process Event Bus (asyncio EventBus)
Each pipeline stage runner broadcasts progress events via EventBus.publish(Event).
The SSE route /api/events maintains an independent asyncio.Queue per subscriber,
achieving zero-latency in-process broadcast. The frontend subscribes via a useTaskEvents() Hook
with auto-reconnect and 15s keepalive, showing per-question processing status and progress counts in real time.
β€ Data Model (SQLAlchemy async ORM)
Stage progress is aggregated on the frontend by counting SampleState.status values
(total / done / failed / machine_reject) β no separate progress table needed.
All ORM operations use async with session for non-blocking execution.
β₯ SKILL.md as Functional Delivery Contract
Each feature module ships a SKILL.md that formally defines:
supported actions (create_task / run_task / list_samples / export_dataset),
input field constraints, output schema (success / data / errors / meta),
and stage artifact paths and formats. This lets AI Coding Agents validate features
against the contract rather than guessing API behavior from code.
β¦ Export Formats
After the quality gate passes, multiple export formats are supported:
- JSON / JSONL β standard multimodal training data format
- XLSX β human review and editing
- images.zip β all generated images packaged
- StaticFiles allowlist ensures only known-safe paths are accessible via
/static/*
API Endpoints
Screenshots
Prompt Engineering & LLM Architecture
The planning stage is the heart of the prompt engineering. No LangChain β
a thin hand-rolled wrapper (httpx + OpenAI-compatible Chat Completions + .prompt
templates + Pydantic) for full control over prompts, JSON tolerance and multi-vendor adaptation.
- Form: a deterministic workflow skeleton with an embedded plannerβjudge (LLM-as-Judge) self-correction loop (pass / repair / regenerate / machine_reject).
- Prompt skeleton: role β input slots β objective β tiered param strategy β level hard-boundary (l2 = param layer / l3 = structural) β 3-element writing constraints (anchor + concrete drawing + visual acceptance) β positive/negative examples β JSON field allowlist.
- Control: system-forced JSON, temperature tiers (0.0 extraction/judging, 0.2 planner), judge emits alignment_table before the verdict, thinking disabled.
- Image prompt: fixed "local edit" preamble + stem + image params + planner instruction; original image uploaded; L1 / text skip generation.
- Fallbacks: 3-tier JSON tolerance, planner regex guard, judge exception β regenerate, best-of on exhaustion, generation retry + backoff, provider auto-switch to DashScope.
- Few-shot: l2 / l3 / judge prompts embed positive + negative examples ("style only, do not copy").
Key Highlights
- 4-stage pipeline covers the full chain from document parsing to dataset export, with per-stage retry support
- L1/L2/L3 three-tier error planning + post-generation evaluation loop ensures data quality
- asyncio EventBus + SSE streaming β frontend shows per-question progress with zero polling
- SKILL.md contract-driven development β AI agents can validate features against the formal spec
- SQLAlchemy async ORM + SQLite β fully async, non-blocking
- TypeScript strict bans all any; Python fully typed