空间感知服务

"where is the soldering iron?"
→ /tmp/world_state.json → last seen at [2.4, 0.8, 0.9] → 4 minutes ago → confidence 0.91

LLM: "Is there anyone in the room?"
→ query_world() → people: [{position: [1.2, 0.3, 0.0], identity: "unknown", emotion: "neutral"}]
→ "Yes, one person, about 1.2 metres ahead, appears neutral."

LLM: "Go find the 3D printer"
→ navigate_to("3D printer")
→ not in world_state → visual search mode
→ LLM generates description → robot sweeps room → YOLO+CLIP locks on → pathplans precisely
→ saves named memory → next time: direct navigation in <15 seconds

LLM: "Remember where I left my keys"
→ remember_object("keys")
→ YOLO watches for keys class → locks 3D position on next detection
→ survives reboots → queryable forever

┌─────────────────────────────────────────────────────────┐
│  TIER 1 — always on                                     │
│  ORB-SLAM3 · YOLO11 · Event checkpointer               │
│  ~1GB VRAM · 3-4 CPU cores · never killed              │
├─────────────────────────────────────────────────────────┤
│  TIER 2 — event-driven                                  │
│  SAM2 · InsightFace · FER · Async frame analyser       │
│  spawned on trigger · killed when idle                  │
├─────────────────────────────────────────────────────────┤
│  TIER 3 — world synthesis                               │
│  World synthesiser · Named memory store                 │
│  merges all perception → world_state.json every 3s     │
├─────────────────────────────────────────────────────────┤
│  TIER 4 — persistence                                   │
│  /tmp/stm/checkpoints · world_state.json               │
│  ~/howyouseeme_persistent/named_memories.json           │
├─────────────────────────────────────────────────────────┤
│  TIER 5 — interface                                     │
│  MCP server :8090 · RViz · Rerun · Ally Robot Mode     │
└─────────────────────────────────────────────────────────┘

Natural language goal
        ↓
Semantic goal resolver
  ├─ named memory?    → direct 3D coordinate → RRT* → DWA → /cmd_vel
  ├─ seen recently?   → last known position  → RRT* → DWA → /cmd_vel
  ├─ room name?       → SLAM map centroid    → RRT* → DWA → /cmd_vel
  └─ never seen?      → visual search mode
                            ↓
                      LLM generates description
                      + optional web reference image
                            ↓
                      room sweep: rotate → expand spiral
                      score: YOLO conf × CLIP similarity
                            ↓
                      detection → back-project → save memory → pathplan

Robot active  →  keyframes saved to disk continuously
Robot sleeps  →  OpenSplat runs on accumulated frames (15-25 min on RTX 3050)
Robot wakes   →  photorealistic .ply splat loaded as /splat/background
                 YOLO semantic labels float on top of photorealistic map
                 map quality improves every night automatically

"What can you see right now?"
→ query_world → objects with 3D positions → natural language description

"Go find the 3D printer"
→ navigate_to("3D printer") → visual search → pathplan → arrive → confirm

"Remember where the apple is"
→ remember_object → YOLO watches → pins on next detection → survives reboot

"Show me what you see"
→ get_camera_frame → base64 JPEG → vision LLM describes the scene

# Dependencies (system Python 3.12, ROS2 Jazzy)
sudo apt install ros-jazzy-desktop ros-jazzy-cv-bridge
pip install open3d ultralytics rerun-sdk --user

# Build
cd ros2_ws && colcon build && source install/setup.bash

# Launch everything
./scripts/run_complete_slam_system.sh

# Visualisation (separate terminal)
rerun   # Rerun opens automatically via rerun_bridge_node
# or RViz:
rviz2

./scripts/cv_pipeline_menu.sh

CV Pipeline — Model Selection

  1) SAM2          segment anything
  2) FastSAM       fast segmentation with text prompts
  3) YOLO11        detection, pose, segmentation, OBB
  4) InsightFace   face recognition + liveness
  5) FER           emotion detection (7 emotions)

  8) List available models
  9) Stop active streaming

HowYouSeeMe/
├── ros2_ws/src/
│   ├── kinect2_ros2_cuda/            Kinect v2 bridge — do not modify
│   ├── bluelily_bridge/              IMU serial bridge
│   │   └── src/bluelily_imu_node.cpp
│   ├── cv_pipeline/                  AI model nodes (C++ pipeline + Python workers)
│   │   ├── src/cv_pipeline_node.cpp
│   │   └── python/
│   │       ├── cv_model_manager.py
│   │       ├── sam2_server_v2.py
│   │       ├── sam2_worker.py
│   │       └── insightface_worker.py
│   └── kinect2_slam/                 SLAM, memory, MCP, visualisation
│       ├── kinect2_slam/
│       │   ├── tsdf_integrator_node.py
│       │   ├── event_checkpointer_node.py
│       │   ├── async_analyser_node.py
│       │   ├── live_enrichment_node.py
│       │   ├── world_synthesiser_node.py
│       │   ├── named_memory_node.py
│       │   ├── rerun_bridge_node.py
│       │   └── mcp_server.py         ← MCP on :8090
│       └── launch/
│           └── howyouseeme_memory.launch.py
├── scripts/
│   ├── run_complete_slam_system.sh   ← main launch script
│   ├── run_phase2_3.sh               ORB-SLAM3 + TSDF
│   └── cv_pipeline_menu.sh           interactive model selector
├── integration/
│   └── Ally/glass-pip-chat/          Ally desktop overlay + Robot Mode
├── data/faces/                       InsightFace database
├── orb_slam3_configs/                ORB-SLAM3 YAML configs
├── kalibr_configs/                   IMU-camera calibration
└── docs/                             guides and references