桌面自动化控制服务

lazy-desktop-mcp

Local-first desktop control MCP server (computer use): a Rust host process plus an npm-distributed Node launcher for safe app/window/input/screenshot automation with an operator Presence UI.

Works with Codex, Claude Code, and other MCP clients over stdio. Defaults are locked down until you configure a host policy.

Keywords: MCP server, desktop automation, computer use, macOS, Windows, Rust, Codex, policy-gated control, Presence UI.

Platforms: macOS (full backend + Presence UI) and Windows (Win32 window/input/screenshot backend; see docs/windows.md).

Clients: Tool names use underscores (app_list, not app.list) so Grok, Codex, Claude Code, OpenAI-style hosts, and other strict MCP clients all register tools correctly. See docs/client-compatibility.md.

What Ships

• desktop-core: shared types, policy evaluation, audit payload handling, and host wire protocol
• desktop-host: local privileged host with audit storage, session handling, screenshot capture, and platform adapters
• desktop-mcp: MCP stdio server that proxies tool calls to desktop-host
• lazy-desktop-mcp: Node launcher published to npm so Codex can start the MCP server with npx or a global install

Presence UI (AI is controlling…)

While the host handles desktop actions it publishes operator-facing status to:

{artifact_dir}/presence/current.json
{artifact_dir}/presence/events.jsonl
{artifact_dir}/presence/STOP     # operator halt (host denies gated actions)
{artifact_dir}/presence/PAUSE    # operator pause (host waits until cleared)

desktop.runtime includes presence_state_path, presence_events_path, presence_stop_path, presence_pause_path, plus presence_ui_app_path / presence_ui_running.
See docs/presence-ui.md for menu bar / HUD design and STOP/PAUSE semantics.

Presence UI lifecycle (macOS)

End-user guide: docs/FOR_USERS.md

desktop-host opens ComputerUsePresence.app (HUD, edge glow, AI cursor) when a session opens or gated desktop control runs — not on idle host/MCP startup.

When the last automation session is closed (session.close) — or the host process exits — the host quits Presence UI so the menu bar/glow does not keep looking like AI is still controlling the Mac.

# One-time: build + install the UI next to host data dir
npm run install:presence-ui

# Then start MCP as usual (Codex / npx lazy-desktop-mcp)
# Host logs on control: "Launched Presence UI: …"
# Host logs after last session.close: "Quit Presence UI (…)"

desktop.runtime reports presence_ui_auto_launch, presence_ui_auto_quit, and presence_ui_running.
Agents can also call presence.ui.quit anytime to force-quit the Presence HUD (always available).

Installed location:

~/Library/Application Support/dev.lazy.desktop-mcp/PresenceUI/ComputerUsePresence.app

Lifecycle (defaults): control starts → launch Presence → last session.close or host exit → host quits Presence. The app also has an optional ~180s idle auto-quit; do not rely on it alone. Closing the HUD does not stop the agent unless you enable “Write STOP when HUD is closed”.

Security Defaults

The public package is intentionally locked down until the operator configures a host policy file.

• desktop.capabilities, desktop.permissions, desktop.runtime, presence.ui.quit, session.open, and session.close are always available
• standalone capabilities such as app.list and observe.capture are disabled until allowed by host policy
• session capabilities such as app.launch are disabled until allowed by host policy
• raw coordinate input is disabled unless explicitly enabled by host policy
• window-scoped screenshots require capture_scope other than primary
• full OCR text requires ocr_allow_full; screenshots stay on disk and are never returned as MCP image bytes
• MCP tool results are compact by default (detail=full for diagnostics)
• on macOS and Windows, out-of-policy app, window, and session-scope requests can trigger a local user approval dialog that persists a target-only allowlist overlay (expires after overlay_max_age_seconds)
• desktop-mcp refuses to start if it cannot find the expected desktop-host binary

See SECURITY.md and docs/security-model.md before enabling desktop control features.

Installation

The npm package builds native binaries during postinstall, so the target machine needs:

• Node.js 20+
• Rust and Cargo
• On Windows: the MSVC toolchain (Visual Studio Build Tools with C++ / VCTools)

Install globally:

npm install -g lazy-desktop-mcp

Or run without a global install:

npx -y lazy-desktop-mcp

The published package was smoke-tested from the npm registry with npx -y lazy-desktop-mcp on macOS, including a real MCP initialize handshake. Windows host builds are verified with cargo test and local MCP tool smoke tests (see docs/windows.md).

If you want to skip the install-time build for CI or packaging experiments:

LAZY_DESKTOP_SKIP_POSTINSTALL=1 npm install
npm run build:native

Host Policy

The host reads a JSON policy file from LAZY_DESKTOP_POLICY_PATH or its local application data directory. Start from the shipped example:

cp config/policy.example.json /path/to/policy.json
export LAZY_DESKTOP_POLICY_PATH=/path/to/policy.json

Example policy:

{
  "allowed_standalone_capabilities": ["app_list", "observe_capture", "ocr_read"],
  "allowed_session_capabilities": ["app_launch"],
  "allowed_apps": ["TextEdit"],
  "allowed_windows": [],
  "allowed_screens": ["primary"],
  "allow_raw_input": false,
  "max_actions_per_minute": 30,
  "capture_scope": "primary",
  "capture_max_long_edge": 1280,
  "capture_format": "jpeg",
  "capture_retain_seconds": 300,
  "ocr_max_chars": 500,
  "ocr_allow_full": false,
  "lists_max_items": 30,
  "overlay_max_age_seconds": 86400
}

For repeatable local development, this repository also ships:

• config/client-config.json: canonical development config source for client wiring
• config/policy.dev.json: generated development policy for interactive desktop workflows

Use the sync script to regenerate policy.dev.json and upsert the matching client entries:

npm run sync:clients

Preview the rendered policy and client config without writing files:

npm run sync:clients:dry

The sync flow defaults to both Codex and OpenCode. Override the target set with LAZY_DESKTOP_CLIENTS=codex, LAZY_DESKTOP_CLIENTS=opencode, or explicit config paths via CODEX_CONFIG_PATH and OPENCODE_CONFIG_PATH.

Runtime Approval Overlay

When the host policy enables a capability class but the requested app, window, or screen target is outside the configured allowlist, the macOS system backend can ask the logged-in user for approval.

• the dialog is local to the target machine and uses the native macOS dialog UI
• Allow persists only the requested target into a local policy-overlay.json
• overlay grants expire after overlay_max_age_seconds (default 24 hours)
• Deny, closing the dialog, or timeout keeps the request blocked
• runtime approval never enables a new capability class and never enables raw coordinate input

The overlay file is stored under the host application data directory and merged with the base policy at startup. Delete that overlay file if you need to clear previously approved targets.

Client Setup

The published package can still be registered manually with Codex:

codex mcp add lazy-desktop \
  -- npx --prefix ~/.codex/mcp-cache/lazy-desktop-mcp -y lazy-desktop-mcp

The isolated --prefix keeps npm's execution context stable even when Codex is launched from a repository that has the same package name as the published MCP package.

If you need an explicit config entry:

[mcp_servers.lazy-desktop]
command = "npx"
args = ["--prefix", "/absolute/path/to/.codex/mcp-cache/lazy-desktop-mcp", "-y", "lazy-desktop-mcp"]

[mcp_servers.lazy-desktop.env]
LAZY_DESKTOP_POLICY_PATH = "/absolute/path/to/policy.json"

If you prefer a fully deterministic local install, npm install -g lazy-desktop-mcp and pointing Codex at the global lazy-desktop-mcp binary also works.

For local repository development, prefer npm run sync:clients; it wires both Codex and OpenCode to the checked-out target/release binaries and the repo-managed development policy.

Desktop App Development

lazy-desktop-mcp is meant to sit above framework-specific desktop stacks.

• Tauri: use it for launch, window targeting, input orchestration, screenshot capture, OCR, and local operator approval loops around a real packaged or dev-run desktop shell
• PyQt: use it for native widget smoke tests, focused regression checks, and screenshot-driven debugging when browser tooling is not available
• keep framework-native tests for deterministic unit/component coverage; use the MCP for end-to-end operator flows and local exploratory validation

The standard local development workflow is:

1. Build the native binaries with npm run build:native
2. Sync the repo-managed client config with npm run sync:clients
3. Grant OS permissions when needed (macOS: Accessibility / Automation / Screen Recording; Windows: see docs/windows.md)
4. Start the target Tauri or PyQt app
5. Verify live availability with desktop.capabilities, desktop.permissions, and desktop.runtime
6. Open a scoped session, then run app/window/input/capture/OCR or vision steps as needed

For interactive flows, prefer the higher-level tools first (low token cost):

• app_activate when you want to bring an app to the front without depending on an exact window title
• selector-based window_focus using window_id, exact title, partial title_contains, or app
• input_click_target for OCR-matched text or window-relative clicks before falling back to raw coordinates
• window_list / app_list with a query instead of dumping the full desktop
• observe_capture only when selectors fail; it returns an artifact id, not pixels. If it reports unchanged, skip OCR/vision. Do not loop full-screen capture plus vision_describe.

See docs/desktop-app-development.md for a more detailed workflow and troubleshooting notes.

Runtime Availability

The development policy enables app launch, window control, screenshot capture, OCR, and interactive input by default. Vision remains optional and only becomes available when a local vision command is configured. Actual runtime availability still depends on:

• the current backend implementation on the active platform (docs/windows.md for the Windows matrix)
• local OS permissions such as Accessibility, Automation, and Screen Recording (macOS) or interactive-desktop / elevation constraints (Windows)
• optional dependencies such as tesseract
• optional vision command wiring

Use desktop.capabilities, desktop.permissions, and desktop.runtime as the source of truth for the current machine instead of assuming a static capability matrix.

If a capability shows Disabled by the host security policy, inspect desktop.runtime first. The default MCP response is compact (policy basename and counts). Pass detail=full (or verbose=true) to see full paths and policy snapshots. A common local-development mistake is wiring the client to config/policy.example.json instead of the repo-managed config/policy.dev.json; rerun npm run sync:clients and restart the client after rebuilding when that happens.

Local Development

Build native binaries:

npm run build:native
npm run sync:clients

Run the full verification stack:

npm run security
npm run verify
npm run pack:dry

Prepare an npm release without publishing yet:

npm run release:prep
npm run release:notes
npm run release:check

The verification flow runs:

• JavaScript wrapper tests
• cargo fmt --check
• cargo clippy -D warnings
• cargo test
• cargo audit
• npm pack --dry-run

Publishing

Before npm publish, make sure:

• the version in package.json matches the Rust workspace version in Cargo.toml
• the canonical client config in config/client-config.json still produces the intended development policy
• the policy example still matches the shipped host behavior
• the README and security docs reflect the actual supported capabilities
• npm run release:prep passes with a clean worktree and a version that is newer than the latest git tag
• npm run release:notes produces the release note draft you intend to ship

See docs/publishing.md for the release checklist.

Related

• lazy-media-mcp — image/video prep for AI vision
• lazy_mobile_mcp — Android/iOS mobile automation MCP
• session-collab-mcp — multi-session collaboration MCP

License

MIT © leaf76