户型图转3D建模管线

2D Floorplan to 3D Blender Pipeline

An experimental pipeline that converts 2D architectural floorplan images into 3D wall geometry in Blender, using AI vision (Google Gemini) for floorplan interpretation and the Blender MCP addon for programmatic scene control.

Status: Research/prototype. Functional pipeline with validated results on residential floorplans. Multiple extraction methods tested and ranked.

Table of Contents

• How It Works
• Architecture
• Pipeline Stages
• Two Extraction Approaches
• Key Findings
• Prompt Engineering Results
• Wall Method Comparison
• Setup
• Usage
• Project Structure
• Schema Reference
• Known Limitations
• Handover Notes for Dev Teams

How It Works

┌──────────────┐     ┌──────────────┐     ┌───────────────┐     ┌──────────────┐
│  2D Floorplan │────>│  Gemini      │────>│  Wall         │────>│  Blender     │
│  Image        │     │  Vision API  │     │  Processing   │     │  3D Scene    │
│  (JPG/PNG)    │     │  (JSON out)  │     │  Pipeline     │     │  (via MCP)   │
└──────────────┘     └──────────────┘     └───────────────┘     └──────────────┘

1. Feed a 2D floorplan image to Google Gemini with a structured extraction prompt
2. Gemini returns JSON with wall segments, room data, openings, and scale metadata
3. Python pipeline validates, adapts, and converts the data to Blender Python code
4. Code is executed in Blender via the MCP (Model Context Protocol) addon
5. Optional: AI-driven visual validation compares the 3D result against the original floorplan

Architecture

pipeline/
├── gemini_client.py        # Gemini API wrapper (image → JSON extraction)
├── compute_walls.py        # Room rectangles → wall segments (legacy approach)
├── adapt_gemini_output.py  # Gemini wall-direct format → Blender format adapter
├── validate_walls.py       # Recursive visual validation orchestrator
├── generate_blender.py     # Wall segments → Blender Python code generator
└── pixel_walls.py          # Red-pixel detection → direct mesh approach

prompts/
├── gemini_floorplan_prompt.md          # Production extraction prompt (scored 80/100)
└── claude_code_wall_validation_prompt.md # Visual validation system prompt

schemas/
├── floorplan_schema.json       # Wall-direct JSON schema (current)
└── floorplan_schema_legacy.json # Room-rectangle schema (deprecated)

tests/
├── prompt_variants.py       # 10 prompt variant definitions + scoring framework
├── run_single_variant.py    # Parallel variant test runner
├── run_production_prompt.py # Production prompt tester
├── test_compute_walls.py    # Unit tests for wall computation
├── test_validate_walls.py   # Unit tests for validation helpers
├── test_adapt_gemini.py     # Unit tests for format adapter
└── test_pixel_walls.py      # Unit tests for pixel wall pipeline

addon.py  # Blender MCP addon (third-party, by Siddharth Ahuja)

Pipeline Stages

Stage 1: Floorplan Analysis (Gemini Vision)

File: pipeline/gemini_client.py

Sends a floorplan image to Gemini with a two-phase structured prompt:

• Phase 1 (Dimensions & Layout): Calibrate scale from the image's scale bar, detect open-plan areas (e.g., KITCHEN/DINING with no dividing wall = one room), build complete room inventory
• Phase 2 (Wall Geometry): Trace building perimeter clockwise, extract every wall segment as (x1, y1, x2, y2) with type, thickness, room references, and openings

The prompt (prompts/gemini_floorplan_prompt.md) was optimized through systematic testing of 10 variants — see Prompt Engineering Results.

Output: JSON conforming to schemas/floorplan_schema.json containing:

• metadata: scale, floor area, building dimensions
• building_perimeter: closed polygon of corner points
• walls[]: each with endpoints, type (external/internal), thickness, room references, openings
• rooms[]: name, center coordinates, exterior/separate flags

Stage 2: Format Adaptation

File: pipeline/adapt_gemini_output.py

Bridges Gemini's wall-direct output to the format expected by the Blender code generator:

• Maps id → name, thickness_m → thickness, merges rooms_left/rooms_right → rooms
• Validates required fields, checks for zero-length walls, duplicate IDs
• Computes building bounds and per-room bounding boxes from wall endpoints

Stage 3: Blender Code Generation

File: pipeline/generate_blender.py

Produces a self-contained Python script for Blender that:

1. Clears the scene
2. Creates wall materials (red for internal, darker red for external)
3. Generates each wall as a scaled/rotated cube primitive
4. Optionally overlays the floorplan image as a semi-transparent texture at z=0.001
5. Sets up an orthographic top-down camera and sun light

Supports calibrated image overlay using pixel-per-metre values and origin offsets for precise alignment between 3D walls and the 2D source image.

Stage 4: Blender Execution (MCP)

File: addon.py

The Blender MCP addon (by Siddharth Ahuja) runs a socket server inside Blender that accepts Python code execution commands. The generated code from Stage 3 is sent to Blender via this protocol.

MCP commands used:

• execute_blender_code: Run Python in Blender's context
• get_viewport_screenshot: Capture the 3D viewport as an image
• get_object_info: Inspect object properties
• get_scene_info: List all scene objects

Stage 5: Visual Validation (Optional)

File: pipeline/validate_walls.py

A recursive validation loop that uses AI vision to verify the 3D output:

1. Full scene check: Take orthographic screenshot, compare wall overlay against floorplan
2. Per-room zoom: Position camera over each room, check enclosure and alignment
3. Correction loop: Up to 5 iterations — diagnose issues, apply fixes, re-validate
4. Convergence tracking: Stops if issue count isn't decreasing after 2 iterations

Generates Blender camera code, analysis prompts, and wall modification code. The AI agent (Claude) drives the loop by interpreting screenshots and calling the helpers.

Two Extraction Approaches

The project explored two fundamentally different approaches to wall extraction:

Approach A: AI Interpretation (Recommended)

Floorplan Image → Gemini Vision → Structured JSON → Wall Segments → Blender

Gemini interprets the floorplan semantically — it understands room labels, dimensions, wall types, and openings. The production prompt scored 80/100 on a comprehensive rubric.

Strengths: Understands architectural semantics, handles open-plan detection, produces labeled walls with room references, works with any floorplan style.

Weaknesses: Right wall position errors (Gemini detected deck boundaries instead of building walls in some cases), inconsistent with small partitions (robes, linen closets).

Approach B: Pixel-Direct Mesh

Floorplan Image → Gemini Red-Wall Render → Red Pixel Detection → Mesh Quads → Blender

File: pipeline/pixel_walls.py

1. Use Gemini's image generation to render the floorplan with walls highlighted in red
2. Detect red pixels via RGB thresholds (R≥180, G≤80, B≤80)
3. Downsample (default 0.25x) for manageable vertex counts
4. Convert each red pixel to a world-coordinate quad
5. Build a unified mesh, merge-by-distance, apply solidify modifier for wall height

Strengths: No AI interpretation errors — the image IS the geometry. Works regardless of floorplan notation style.

Weaknesses: Requires a clean red-wall render (depends on Gemini image generation quality), produces rough mesh geometry, no semantic data (room names, openings), scale detection still requires AI.

Key Findings

1. Two-Phase Prompting is Critical

The single most impactful improvement was splitting the extraction into two phases:

• Phase 1: Extract dimensions and layout FIRST (scale calibration, open-plan detection, room inventory)
• Phase 2: Extract geometry SECOND (wall endpoints, using Phase 1 data as constraints)

Without this split, Gemini frequently miscounted rooms, merged rooms that should be separate, or produced walls that contradicted labeled dimensions.

2. Right Wall Detection is the Hardest Problem

Across 8 extraction methods, the most differentiating factor was the right external wall X position:

• x=7.1: Correct (confirmed by pixel analysis + dimension annotations + visual alignment)
• x=8.0: All Gemini-vision methods got this wrong — they detected the DECK/ENTERTAINING area boundary instead of the building wall
• x=6.5: LSD line detection found an internal wall, not the external

Takeaway: AI vision models struggle to distinguish between building walls and adjacent outdoor structure boundaries. Explicit perimeter-tracing instructions help but don't fully solve this.

3. Open-Plan Detection Needs Explicit Instructions

Without explicit instructions, Gemini created false walls between KITCHEN and DINING areas that share an open-plan space. The fix was adding a mandatory pre-check: "If two labeled spaces have NO wall between them, treat as ONE room."

4. Scale Calibration is Reliable

Gemini consistently extracts accurate pixels-per-metre values from scale bars and dimension annotations. Cross-validation against multiple labeled dimensions typically agrees within 5%.

5. Image Overlay Calibration

For visual validation to work, the floorplan overlay in Blender must be precisely calibrated. The key parameters:

• ppm_x, ppm_y: Pixels per metre (can differ for non-square aspect ratios)
• origin_col, origin_row: Pixel coordinates of world origin (0,0)
• Image dimensions for computing overlay size and position

Prompt Engineering Results

10 prompt variants were tested against Gemini 3 Pro Preview using a 100-point scoring rubric:

Results

The winning prompt combines v08's two-phase approach with v02's explicit wall-per-perimeter-segment tracing instructions.

Test harness: tests/prompt_variants.py contains all 10 variant definitions and the scoring framework. tests/run_single_variant.py runs them in parallel.

Wall Method Comparison

8 different wall extraction methods were tested in Blender (see output/wall_method_comparison_report.md):

Setup

Prerequisites

• Python 3.10+
• Blender 3.0+ (with the MCP addon installed)
• Google Gemini API key (get one here)

Install

# Clone the repo
git clone https://github.com/jake-jiffi/2d_3d_blender_mcp.git
cd 2d_3d_blender_mcp

# Install Python dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env and add your GEMINI_API_KEY

Blender MCP Addon

1. Open Blender → Edit → Preferences → Add-ons → Install
2. Select addon.py from this repo
3. Enable "Blender MCP" in the addon list
4. In the 3D Viewport sidebar (N panel), find the BlenderMCP tab
5. Click "Start Server" — the MCP socket server runs on localhost:9876

Usage

Quick Start: AI Interpretation Pipeline

from pipeline.gemini_client import analyze_floorplan
from pipeline.adapt_gemini_output import adapt_gemini_output, validate_gemini_json
from pipeline.generate_blender import generate_code

# 1. Extract floorplan data via Gemini
data = analyze_floorplan("path/to/floorplan.jpg")

# 2. Validate the extraction
issues = validate_gemini_json(data)
if issues:
    print("Validation issues:", issues)

# 3. Adapt to Blender format
walls, bounds, room_bounds = adapt_gemini_output(data)

# 4. Generate Blender code
code = generate_code(
    walls,
    bounds,
    wall_height=2.7,
    floorplan_image_path="path/to/floorplan.jpg",
    overlay_config={
        "ppm_x": 78.75,
        "ppm_y": 79.42,
        "origin_col": 374,
        "origin_row": 1923,
        "image_width_px": 1822,
        "image_height_px": 2061,
    },
)

# 5. Send to Blender via MCP (or save to file and run manually)
print(code)

Quick Start: Pixel-Direct Pipeline

from pipeline.pixel_walls import generate_pixel_walls_code

# Auto-detects scale via Gemini, processes red pixels, generates Blender code
code = generate_pixel_walls_code(
    image_path="path/to/floorplan_red_walls.png",
    wall_height=2.7,
    scale_factor=0.25,  # Downsample for performance
)

# Or provide scale manually
code = generate_pixel_walls_code(
    image_path="path/to/floorplan_red_walls.png",
    pixels_per_metre=96.0,
    wall_height=2.7,
)

Running Tests

# Unit tests
pytest tests/

# Test a single prompt variant against Gemini
python tests/run_single_variant.py

# Test the production prompt
python tests/run_production_prompt.py

Running the Visual Validation Loop

The validation system is designed to be driven by an AI agent (Claude Code with Blender MCP access). See prompts/claude_code_wall_validation_prompt.md for the full system prompt that orchestrates:

1. JSON integrity pre-check
2. Wall generation and overlay setup
3. Full-scene top-down validation via screenshot analysis
4. Per-room close-up validation
5. Recursive correction (up to 5 iterations)

Project Structure

2d_3d_blender_mcp/
├── README.md                 # This file
├── CLAUDE.md                 # AI agent operating instructions
├── addon.py                  # Blender MCP addon (third-party)
├── requirements.txt          # Python dependencies
├── .env.example              # Environment variable template
├── .gitignore
│
├── pipeline/                 # Core pipeline modules
│   ├── __init__.py
│   ├── gemini_client.py      # Gemini API wrapper
│   ├── compute_walls.py      # Room rectangles → wall segments
│   ├── adapt_gemini_output.py # Format adapter
│   ├── validate_walls.py     # Visual validation orchestrator
│   ├── generate_blender.py   # Blender code generator
│   └── pixel_walls.py        # Red-pixel mesh pipeline
│
├── prompts/                  # AI prompts
│   ├── gemini_floorplan_prompt.md          # Production extraction prompt
│   └── claude_code_wall_validation_prompt.md # Validation system prompt
│
├── schemas/                  # JSON schemas
│   ├── floorplan_schema.json       # Current wall-direct format
│   └── floorplan_schema_legacy.json # Deprecated room-rectangle format
│
├── tests/                    # Tests and prompt experiment harness
│   ├── prompt_variants.py    # 10 prompt variants + scoring
│   ├── run_single_variant.py # Variant test runner
│   ├── run_production_prompt.py # Production prompt test
│   ├── test_compute_walls.py
│   ├── test_validate_walls.py
│   ├── test_adapt_gemini.py
│   └── test_pixel_walls.py
│
├── output/                   # Generated artifacts (mostly gitignored)
│   ├── wall_method_comparison_report.md  # 8-method comparison
│   ├── variant_*.json        # Prompt variant test results
│   └── ...                   # Generated Blender scripts, images
│
└── docs/                     # Working documentation
    └── discovery/            # Research and reference materials

Schema Reference

Wall-Direct Format (schemas/floorplan_schema.json)

Each wall is defined as a line segment with metadata:

{
  "id": "wall_ext_south",
  "x1": 0.0,
  "y1": 0.0,
  "x2": 7.1,
  "y2": 0.0,
  "wall_type": "external",
  "thickness_m": 0.15,
  "rooms_left": null,
  "rooms_right": "GARAGE",
  "openings": [
    {
      "type": "single",
      "position_along_wall_m": 3.5,
      "width_m": 0.9
    }
  ]
}

Coordinate system:

• Origin (0, 0) at bottom-left of main building footprint
• X increases right, Y increases up
• All measurements in metres
• All coordinates must be >= 0

Wall types:

• external: Building perimeter (typically 0.15-0.20m thick)
• internal: Room dividers (typically 0.10-0.15m thick)

Opening types: single, double, sliding, bifold, window

Known Limitations

1. Right wall detection: Gemini vision methods consistently detect outdoor structure boundaries (decks, patios) instead of building walls. Manual verification needed for buildings with adjacent outdoor structures.

2. Small partition walls: Robes, linen closets, ensuite subdivisions are frequently missed. The AI sees them but doesn't always include them in the structured output.

3. Single floorplan tested: Production prompt optimized against one residential floorplan. Robustness across different styles (apartment, commercial, multi-story) is unvalidated.

4. No opening cutouts: Current pipeline builds all walls as solid. Door/window openings are recorded in the JSON but not cut from the 3D geometry. This is intentional for the validation phase but would need implementation for a production model.

5. Overlay calibration is manual: The overlay_config parameters (ppm, origin offsets) must be determined per-image. Auto-calibration from Gemini's scale metadata is partially implemented but not fully reliable.

6. Pixel-direct approach is resolution-dependent: The red-pixel mesh quality depends heavily on input image resolution and the quality of Gemini's red-wall render.

Handover Notes for Dev Teams

What Works

• Gemini extraction pipeline: End-to-end from image to Blender scene. The production prompt reliably extracts 80%+ of wall geometry from residential floorplans.
• Format adapter: Robust validation and conversion between Gemini output and Blender input.
• Blender code generation: Correctly produces walls, overlay, camera, lighting. Calibrated overlay alignment confirmed working.
• Prompt scoring framework: Reusable harness for testing prompt variants against real floorplans with quantitative scoring.
• Visual validation system: Architecture in place for AI-driven recursive correction.

What Needs Work

1. Multi-image testing: Test against 3-5 different floorplan styles. The prompt was optimized for one image.

2. Right wall disambiguation: Add explicit instructions to distinguish building walls from adjacent outdoor structure boundaries. Possible approaches:

   • Ask Gemini to identify the "building envelope" as a separate step
   • Use dimension annotations as constraints (if LIVING is 3.7m wide and starts at x=3.4, right wall must be at x=7.1)
   • Cross-validate with a second Gemini call at different temperature

3. Missing partition recovery: After initial extraction, run a second pass specifically asking "Are there any small partitions (robes, closets, linen cupboards) that are missing?"

4. Opening cutouts: Implement door/window cutouts in generate_blender.py using boolean operations or by splitting wall meshes at opening positions.

5. Auto-calibration: Use Gemini's extracted pixels_per_metre and building dimensions to auto-compute overlay_config, eliminating manual calibration.

6. Pixel-direct improvements: The pixel mesh approach produces rough geometry. Consider:

   • Contour detection instead of per-pixel quads
   • Wall centerline extraction from the red mask
   • Hybrid: use pixel mesh for perimeter, AI interpretation for semantics

Architecture Decisions & Rationale

Key Constants

EXTERNAL_WALL_THICKNESS = 0.15  # metres
INTERNAL_WALL_THICKNESS = 0.10  # metres
WALL_HEIGHT = 2.7               # metres (standard residential)
TOLERANCE = 0.05                # metres (shared edge matching)
PIXEL_SCALE_FACTOR = 0.25       # downsample for pixel pipeline
RED_THRESHOLDS = (R≥180, G≤80, B≤80)  # red pixel detection

Environment

• Python 3.10+
• Blender 3.0+ with MCP addon on localhost:9876
• Google Gemini API (primary: gemini-2.5-pro, fallback: gemini-2.5-flash-lite)
• Dependencies: google-genai, python-dotenv, Pillow, pytest, numpy (for pixel pipeline)

Credits

• Blender MCP Addon: Siddharth Ahuja
• Pipeline Development: Jake Shelley
• AI Models: Google Gemini (floorplan extraction), Anthropic Claude (validation agent)

License

MIT