Open source · Swift 6 · SwiftUI · no Electron

The Mac is the cockpit. Every other screen is a control surface.

blau is four native Apple apps built as one system. Pilot runs your terminals, browser, and iPhone mirror in a single macOS window — and your iPhone, iPad, and Watch connect to it peer-to-peer to steer, annotate, and answer.

View on GitHub Read the architecture
  • 4 apps, 1 project spec
  • Real Ghostty terminals
  • Peer-to-peer on your LAN
  • Voice transcribed on-device
Pilot for macOS — a single window with terminal, browser, and iPhone mirror panes arranged side by side.
Pilot — terminal, browser, and device panes in one workspace.
The fleet

Four apps. One system.

Each app does one job and does it natively — same Swift 6 codebase, same shared Sources tree, one generated Xcode project.

Pilot

macOS

The cockpit. Ghostty terminal splits, a dev-server-aware browser, a USB iPhone mirror, git and GitHub inspectors, markdown notes — and the hub every other device connects to.

Pilot on macOS: terminal splits, dev browser, and inspectors in a single window

Copilot

iOS

The pocket remote. Mirrors Pilot's workspaces and tabs, drives the Mac cursor from a touch trackpad, and turns the volume buttons into push-to-talk.

Copilot on iPhone mirroring Pilot's workspaces

Plotter

iPadOS

The second screen. Receives Pilot's window as live hardware-decoded HEVC and layers an Apple Pencil canvas whose strokes land back on the Mac.

Plotter on iPad mirroring the Mac as live HEVC

Wingman

 watchOS

The Enter key. Double-pinch on your wrist and Enter lands in the Mac's active terminal, relayed through Copilot. One gesture, one job.

Wingman on Apple Watch, armed to send Enter
Pilot · macOS

Everything a working session needs, in one window.

Three pane kinds — terminal, browser, device — in resizable splits, with git inspectors, notes, and an ink overlay around them.

  • Real Ghostty terminals

    Every terminal pane is the actual Ghostty engine: libghostty rendering on a native Metal surface, with your existing Ghostty config loaded. Panes attach to named tmux sessions, so shells and long-running agents survive a relaunch when tmux is installed — without it, your working directory still comes back.

    GhosttyKit.xcframework · OSC 7 cwd tracking

  • A browser that knows your dev servers

    WKWebView with the Web Inspector enabled, and a scanner that reads package.json and .env to offer one-click localhost links — Next, CRA, Nuxt, and Remix on 3000, Vite on 5173, Astro on 4321, with explicit --port flags and PORT values parsed. It even probes which servers are actually running.

    LocalServerScanner.swift

  • Your iPhone, in a pane

    Tether over USB and the phone appears QuickTime-style via CoreMediaIO. Record a timestamped .mov, or grab a PNG screenshot — both land on the Desktop, and screenshots hit the clipboard at the same time.

    CoreMediaIO · AVFoundation

  • Git and GitHub, ambient

    Commits, Actions runs, and open issues polled through git and gh. Click an issue and Pilot pastes an “Implement issue #N” prompt into the active terminal. A background sweeper badges any workspace whose Action completes while you're elsewhere.

    gh run list · 30s poll · 60s sweep

  • Notes that behave like a tool

    A GitHub-flavored markdown editor with multi-cursor split-into-lines, clickable checkboxes that auto-sort completed tasks down, per-block copy buttons, table auto-format, and clickable color swatches. .env-style values can be visually masked against shoulder-surfing, but Notes stores plaintext locally and is not a secret manager.

    ⇧⌘Lsplit into lines

  • Opens your IDE, too

    One command opens the workspace in Cursor, Codex, or VS Code — whichever is installed, in that order.

    IDELauncher.swift

Plotter on iPad receiving a live hardware-decoded mirror of Pilot's macOS window
Plotter receiving Pilot's window — 60fps target, ~2s max keyframe interval.
Plotter · iPadOS

The Sidecar playbook, tuned for one window.

ScreenCaptureKit captures Pilot's window straight into GPU-backed buffers. VideoToolbox encodes HEVC on the Mac's media engine — real-time mode, no frame reordering — and the iPad hardware-decodes into an AVSampleBufferDisplayLayer. Frames travel device-to-device over Bonjour and peer-to-peer Wi-Fi. No router required. No cloud involved.

  1. ScreenCaptureKit
  2. VideoToolbox HEVC
  3. _blau-frames._tcp · Bonjour/AWDL
  4. VideoToolbox decode
  5. PencilKit overlay

Built for text, not movies

The stream negotiates up to 4:4:4 chroma by capability handshake, so terminal output and code stay crisp where ordinary 4:2:0 video smears fine colored text.

Adaptive by feedback

Plotter reports link quality back; Pilot's encoder retunes its bitrate and resends keyframes on loss. Capture runs only while a Plotter client is connected.

Ink that round-trips

Pencil strokes are normalized to the video frame, sent over a dedicated annotation channel, and rendered in an AppKit overlay on the Mac — with per-stroke undo kept in sync on both sides.

Present beyond the app

A Home Screen and Lock Screen widget holds the latest received frame, and a Live Activity puts mirror-session status in the Dynamic Island.

Copilot · iOS — Wingman · watchOS

Speak it. Point it. Pinch it.

  • Push-to-talk, transcribed on the phone

    Hold volume-down and talk. WhisperKit transcribes on-device — Whisper compiled to Core ML, scheduled across the Neural Engine — and only the finished text crosses the wire for Pilot to paste into the held workspace's terminal. Audio never leaves the phone.

    WhisperKit · openai_whisper-base · Core ML

  • A trackpad that posts real events

    The touch trackpad streams relative deltas and clicks over the encrypted session; Pilot posts genuine CGEvents at the HID tap, gated by the macOS Accessibility permission.

    CGEvent · AXIsProcessTrusted

  • Hardware buttons as navigation

    A KVO observer on the system output volume turns the physical buttons into list navigation — volume HUD suppressed by a hidden MPVolumeView. Hold volume-up to send Enter.

    AVAudioSession.outputVolume KVO

  • Workspaces in your pocket

    Pilot's workspaces and pane tabs mirror live to the phone as a switcher — select a tab on the phone, it changes on the Mac.

    TabSummary · SelectTab

  • Enter, from the wrist

    Wingman binds the system Double Tap gesture: double-pinch and Enter lands in the Mac's active terminal, relayed through Copilot over WatchConnectivity, confirmed with a haptic.

    .handGestureShortcut(.primaryAction)

Copilot on iPhone: push-to-talk, touch trackpad, and live workspace switcher driving Pilot on the Mac
Wingman on Apple Watch: double-pinch sends Enter to the Mac's active terminal

Architecture

Serverless on your LAN. Zero-knowledge beyond it.

Every channel is peer-to-peer, and each one is chosen for its job — ordered control state, latency-sensitive media, and an end-to-end-encrypted lane for messages.

  1. Control

    MultipeerConnectivity

    Workspace state, tab selection, trackpad deltas, voice text, Enter. An MCSession with encryption required, service type blau-sync.

  2. Media

    Network.framework TCP

    HEVC frames, keyframe requests, link stats. _blau-frames._tcp on Bonjour, including peer-to-peer Wi-Fi (AWDL), so no shared router is required.

  3. Annotation

    Network.framework TCP

    Pencil strokes, undo, clear — both directions. Its own channel: _blau-annotate._tcp.

  4. Watch

    WatchConnectivity

    Double-pinch → Enter, with guaranteed background transfer as the fallback when the phone is unreachable.

  5. Secure Messaging

    Noise IK over UDP

    End-to-end encrypted messaging between Pilot and Copilot across networks: X25519 + ChaChaPoly via CryptoKit, NAT hole punching, ACK'd delivery with replay protection. Keys exchange automatically over the Multipeer channel.

FOOTNOTE — One server exists. A Cloudflare rendezvous worker brokers {public key, IP, port} so two devices on different networks can hole-punch — it never sees plaintext or session keys, and it's contacted only for cross-network messaging. Everything else never leaves your local network.

Built on

Credit where it’s due.

  • Ghostty (libghostty)

    Terminal emulation, font handling, and Metal rendering — the same core as the standalone Ghostty app. MIT-licensed, written in Zig, by Mitchell Hashimoto.

  • WhisperKit

    Argmax’s open-source Whisper runtime for Apple Silicon: models compiled to Core ML, inference on the Neural Engine, MIT-licensed.

  • Network.framework + Bonjour

    Apple’s modern user-space transport stack, with discovery over the IETF-standard mDNS/DNS-SD — no directory server in the path.

  • ScreenCaptureKit + VideoToolbox

    GPU-backed window capture and fixed-function HEVC encode/decode — mirroring doesn’t compete with your build for CPU.

  • Swift 6 + SwiftUI

    Strict-concurrency language mode across all four targets. Data races are compile errors here, not crash reports.

  • CryptoKit

    X25519, ChaChaPoly, and HKDF — the primitives under the Noise IK secure channel.

  • XcodeGen

    One YAML spec generates all four targets — the project definition is plain text, so build configuration gets code-reviewed like everything else.

  • tmux

    The long-standing standard for session persistence — each terminal pane attaches to its own named session.