Architecture

Peekaboo Architecture Overview

Home GitHub Edit page

Peekaboo Architecture Overview

This document provides a high-level overview of how Tachikoma and PeekabooCore work together to provide AI-powered macOS automation capabilities.

#System Architecture

#Core Components

┌─────────────────┐
│   Tachikoma     │  AI models + streaming
└────────┬────────┘
         │
┌────────▼────────┐      ┌────────────────────┐      ┌────────────────────┐
│ PeekabooAutomation│◄───►│ PeekabooAgentRuntime │◄───►│  PeekabooVisualizer  │
│ UI/system services│      │ Agent + MCP runtime │      │ Visual feedback stack │
└────────┬────────┘      └──────────┬──────────┘      └──────────┬──────────┘
         │                           │                           │
         └───────────────┬───────────┴───────────┬───────────────┘
                         ▼                       ▼
                  ┌─────────────┐        ┌──────────────┐
                  │  PeekabooCore│        │   Apps / CLI │
                  │ (umbrella)   │        │  consumers   │
                  └─────────────┘        └──────────────┘
  • PeekabooFoundation – shared errors and low-level value types used across packages.
  • PeekabooProtocols – cross-module protocols and transport-safe model contracts.
  • PeekabooExternalDependencies – central package boundary for AXorcist, Swift Algorithms, Swift Log, Swift System, Commander, and Collections.
  • PeekabooAutomationKit – owns capture, observation, input, application/menu/window services, snapshot management, and typed automation models.
  • PeekabooAutomation – re-exports PeekabooAutomationKit and adds Peekaboo configuration, Tachikoma model resolution, and visualizer feedback adapters.
  • PeekabooBridge – request/response transport and host/client plumbing for permission-bearing runtimes.
  • PeekabooVisualizer – standalone visual feedback layer (VisualizationClient, event store, presets) used by automation and apps.
  • PeekabooUICore – shared SwiftUI inspector and overlay components used by app surfaces.
  • PeekabooAgentRuntime – MCP tools, ToolRegistry/formatters, and the agent service itself. Depends on PeekabooAutomation for services/data models and on PeekabooVisualizer for status tokens.
  • PeekabooCore – thin umbrella (_exported imports + PeekabooServices convenience container). Apps/CLI keep importing PeekabooCore, but large features can now link the more focused products directly. Whoever instantiates PeekabooServices is responsible for calling installAgentRuntimeDefaults() so MCP tools and the ToolRegistry share that instance.
  • Tachikoma – still the AI provider surface that the runtime modules call through. See

    • providers.md for the current provider and model catalog.

#Runtime hosting

Permission-bound automation can execute in three runtime shapes:

RuntimeState and permissionsTransport
Reusable daemonWarm snapshots, tracking, browser MCP state; daemon process TCCdaemon.sock Bridge protocol
Peekaboo.appGUI-held TCC grants and app lifecyclebridge.sock Bridge protocol
MCP serverProcess-local services owned by the MCP clientstdio; no Bridge listener

CLI automation resolves a healthy daemon first, then a capable Peekaboo.app host, then auto-starts the reusable daemon. Operations that permit local fallback can run in the CLI process when no host is usable. Explicit socket and --no-remote flags override this selection.

The daemon and GUI app never share a socket. Each Bridge listener holds an exclusive lease, publishes its socket atomically, and removes only the filesystem object it owns. See daemon.md and bridge-host.md for lifecycle, migration, security, and TCC troubleshooting.

#Dependency Flow

Tachikoma (AI Model Management)

  • Provides LanguageModel, the ModelProvider protocol, provider parsers, and Tachikoma.generateText.
  • TachikomaConfiguration supplies provider credentials and endpoint configuration.
  • PeekabooAIService resolves configured model strings and calls Tachikoma with the selected LanguageModel.

PeekabooAutomationKit

  • Exposes automation protocols and concrete implementations such as ScreenCaptureService, UIAutomationService, MenuService, and ProcessService.
  • Owns capture, observation, input, window filtering, snapshot persistence, and their typed models.

PeekabooAutomation

  • Re-exports PeekabooAutomationKit and adds ConfigurationManager, PeekabooAIService, and VisualizerAutomationFeedbackClient.
  • Depends on Tachikoma for provider/model execution and on PeekabooVisualizer for optional UI feedback.

PeekabooAgentRuntime

  • Imports PeekabooAutomation for services/models and hosts MCP/agent tooling (PeekabooAgentService, MCPToolContext, ToolRegistry, CLI/MCP formatters).
  • Provides a clean PeekabooServiceProviding protocol so higher layers (CLI, macOS app, and the MCP server entrypoints) can swap concrete service collections without touching globals.

PeekabooVisualizer

  • Stays decoupled from automation; only consumes PeekabooProtocols data (DetectedElement, LogLevel) so it can be embedded in other contexts later.
  • VisualizationClient is still accessed via PeekabooAutomation convenience wrappers, but the module boundary keeps visual dependencies out of headless hosts.

#Tachikoma: AI Model Management

#Architecture Pattern: Explicit Model Selection

Peekaboo resolves configured provider strings into Tachikoma LanguageModel values before executing requests:

let ai = PeekabooAIService(configuration: .shared)
let model = ai.resolveConfiguredModel("openai/gpt-5.6")
let text = try await ai.generateText(prompt: "Describe this workflow", model: model)

#Key Components

#LanguageModel

  • Role: Typed model selection with provider identity and capability metadata.
  • Resolution: ProviderParser, AIProviderParser, and LanguageModel.parse(from:) turn configuration strings into model values.

#ModelProvider

  • Role: Provider execution protocol used by Tachikoma and Peekaboo's custom compatible-provider adapter.
  • Supported Providers: See providers.md for the current provider reference.

#TachikomaConfiguration

  • Role: Provider keys, endpoints, and request configuration for Tachikoma calls.
  • Sources: Peekaboo's ConfigurationManager loads ~/.peekaboo/config.json, ~/.peekaboo/credentials, and environment variables, then applies those values to Tachikoma.

#PeekabooCore: Automation Engine

#Architecture Pattern: Service Orchestration

PeekabooCore uses a service locator pattern with specialized service delegation:

let services = PeekabooServices()
let automation = services.automation  // UIAutomationService
let screenCapture = services.screenCapture  // ScreenCaptureService
let applications = services.applications  // ApplicationService

#Service Hierarchy

#PeekabooServices (Service Locator)

  • Role: Central registry for all automation services
  • Pattern: Service locator with dependency injection support
  • Lifecycle: Manages service initialization and coordination

##### Installing a services instance PeekabooServices no longer registers itself globally. Whoever constructs an instance (CLI runtime, macOS app, integration test, etc.) must call services.installAgentRuntimeDefaults() immediately after initialization. This wires the container into MCPToolContext and ToolRegistry so downstream tooling (MCP server, CLI peekaboo tools, agent service) can resolve the exact same services without touching singletons. Skipping the install step will cause MCP and ToolRegistry code to fatal because no default factory is configured.

#UIAutomationService (Orchestrator)

  • Role: Primary automation interface delegating to specialized services
  • Delegation: Routes operations to appropriate specialized services
  • Snapshot Management: Maintains state across automation workflows

#Specialized Services

Each service handles a specific aspect of automation:

  • ClickService: Mouse interaction and element targeting
  • TypeService: Keyboard input and text manipulation
  • ScreenCaptureService: Display and window capture
  • ApplicationService: Application discovery and management
  • WindowManagementService: Window positioning and state control
  • MenuService: Menu bar navigation and interaction
  • SnapshotManager: State persistence and element caching

#Threading Model

Main Thread Requirement: All UI automation operations run on MainActor due to macOS requirements:

@MainActor
public final class UIAutomationService: UIAutomationServiceProtocol {
    // All operations are main-thread bound
}

#Integration Points

#AI Integration

PeekabooCore integrates with Tachikoma through PeekabooAgentService:

let services = PeekabooServices()
services.installAgentRuntimeDefaults()
let ai = PeekabooAIService(configuration: services.configuration)
let model = ai.resolveConfiguredModel("anthropic/claude-opus-5") ?? .anthropic(.opus5)
let agent = try PeekabooAgentService(services: services, defaultModel: model)

#Visual Feedback Integration

Services automatically connect to PeekabooVisualizer when available:

// Automatic visualizer integration
let visualizerClient = VisualizationClient.shared
_ = await visualizerClient.showClickFeedback(at: clickPoint, type: clickType)

Behind the scenes the client serializes a VisualizerEvent into ~/Library/Application Support/PeekabooShared/VisualizerEvents/<uuid>.json and posts boo.peekaboo.visualizer.event via NSDistributedNotificationCenter. When Peekaboo.app is alive its VisualizerEventReceiver loads the payload and hands it to VisualizerCoordinator; otherwise the event is silently dropped and execution continues.

#Data Flow Architecture

#Automation Workflow

  1. Input: Natural language task or direct API call
  2. AI Processing: PeekabooAgentService uses Tachikoma models
  3. Service Orchestration: UIAutomationService delegates to specialized services
  4. Platform Integration: Services use macOS APIs (Accessibility, ScreenCaptureKit)
  5. Visual Feedback: Operations trigger visualizer animations
  6. Snapshot Management: State cached for subsequent operations

#Example Flow: "Click the Submit button"

User Input ("Click Submit")
    ↓
PeekabooAgentService (AI interpretation)
    ↓
UIAutomationService.detectElements() → ElementDetectionService
    ↓
UIAutomationService.click() → ClickService
    ↓
macOS Accessibility APIs
    ↓
VisualizationClient (click animation)

#Performance Characteristics

#Service Performance Ranges

  • Element Detection: 200-800ms (AI analysis + accessibility correlation)
  • Click Operations: 10-50ms (accessibility API optimization)
  • Screen Capture: 20-100ms (ScreenCaptureKit acceleration)
  • Application Discovery: 20-200ms (depending on system load)
  • Window Management: 10-200ms (depending on operation complexity)

#Optimization Strategies

  • Snapshot Caching: Element detection results cached per snapshot
  • Accessibility Timeouts: Reduced from 6s to 2s to prevent hangs
  • Dual APIs: Modern ScreenCaptureKit with CGWindowList fallback
  • Visual Feedback: Async animations don't block automation operations

#Error Handling Strategy

#Layered Error Handling

  1. Service Level: Individual services handle API-specific errors
  2. Orchestration Level: UIAutomationService provides unified error handling
  3. Agent Level: AI agent handles retry logic and error recovery
  4. Client Level: Applications receive structured error information

#Defensive Programming

  • Permission Validation: Automatic checks for Screen Recording and Accessibility permissions
  • Timeout Protection: Configurable timeouts prevent system hangs
  • Graceful Degradation: Fallback strategies for problematic applications
  • State Validation: Element existence and accessibility verification

#Configuration Management

#Multi-Source Configuration

  1. Environment Variables: PEEKABOO_AI_PROVIDERS, OPENAI_API_KEY, etc.
  2. Credential Files: ~/.peekaboo/config.json, ~/.tachikoma/credentials
  3. Runtime Parameters: Method-level configuration overrides
  4. Feature Flags: PEEKABOO_USE_MODERN_CAPTURE, etc.

#Configuration Precedence

CLI Arguments > Environment Variables > Credential Files > Config Files > Defaults

#Future Architecture Considerations

#Scalability

  • Service architecture supports horizontal scaling through additional specialized services
  • AI model provider supports multiple concurrent model instances
  • Snapshot management designed for multi-user and multi-process scenarios

#Extensibility

  • Plugin architecture possible through service locator pattern
  • AI model provider supports custom model implementations
  • Visual feedback system can be extended with additional visualization types

#Cross-Platform Potential

  • Service interfaces abstract platform-specific implementations
  • Threading model adaptable to other platforms
  • AI integration remains platform-agnostic

This architecture has been designed to be "really easy for other people to understand" while providing the performance and reliability needed for production automation workflows.