ctrl

Appearance

@involvex/ctrl – Universal Agent Plan ​

Lokale Agent-Plattform mit Expo (Android/Web), Electron (Desktop), Bun-Gateway (@involvex/ctrl-a) und späterem Rust-Backend.

1. Leitentscheidungen ​

  • Gateway als eigener Prozess: Bun + TypeScript in packages/ctrl-a, kompiliert mit bun build --compile. Eingebettet, lokaler Daemon oder LAN-Server.
  • Desktop-Shell: Electron. Startet und überwacht das Gateway; Tray, Autostart, Updates, Secrets.
  • Mobile und Web: Separate Expo-Apps (apps/mobile, apps/web) mit Expo Router.
  • Local-first identity: Persona and transcript live on each client. Generators are interchangeable (on-device LiteRT, cloud via host, peer PC gateway). Opening chat does not require the gateway.
  • Gateway is the home node: Tools, MCP, channels, backups, and append-only sync when reachable — not a requirement to chat.
  • Clients speak the gateway protocol when paired (@involvex/ctrl-s / ctrl-i); on-device LiteRT is a local generator plug inside the same chat UI.
  • Rust später: Backend agentd ersetzt das Gateway schrittweise, abgesichert durch Contract-Tests (ctrl-t).
  • Nicht im Scope: Flutter, Ionic als Hauptarchitektur, alles in Rust.
  • Naming & Publish: Siehe Namespace_and_Design.md und ADR-0002. Nur ctrl-f5 ist publishable.
  • Quality: Bun-native only — CodeQuality.md, ADR-0003.

2. Architektur ​

text
Electron Desktop | Expo Android | Expo Web/PWA | CLI (ctrl-f5)
        \            |              |           /
         ------- AgentClient (HTTP / WebSocket) -------
                          |
                 Gateway @involvex/ctrl-a (Bun + TS)
   Sessions | Workspaces | Agents | Jobs | Events | Auth
   Agent Loop | Tool Router | Context Builder | Permissions
   Provider Registry | MCP | Skills | Knowledge | Memory
        |                    |                  |
  Provider Adapter     Runtime Adapter    Channel Adapter
  OpenAI-kompatibel    llama-server       Webhook, Telegram
  Custom HTTP, CLI     LiteRT-LM          Discord, Slack

Regel: Cloud and PC providers go through the paired gateway. On-device LiteRT runs locally as a generator plug. Identity/transcript stay on the client; sync is append-only when the home node is up.

3. Monorepo ​

text
ctrl/
├─ apps/
│  ├─ desktop/        # Electron: main, preload, renderer (Vite + React)
│  ├─ mobile/         # Expo Android
│  ├─ web/            # Expo Router Web/PWA
│  └─ cli/            # @involvex/ctrl-f5 (publishable)
├─ packages/
│  ├─ ctrl-a/         # Gateway (Bun Binary)
│  ├─ ctrl-s/         # Protocol: Zod, Events, JSON Schema / OpenAPI
│  ├─ ctrl-r/         # Runtimes & Providers
│  ├─ ctrl-z/         # Memory, SQLite, Storage
│  ├─ ctrl-space/     # Shared Matrix UI tokens / components
│  ├─ ctrl-c/         # Channels
│  ├─ ctrl-m/         # MCP Client
│  ├─ ctrl-t/         # Contract tests
│  ├─ ctrl-k/         # Knowledge
│  ├─ ctrl-i/         # Typed Agent Client
│  ├─ ctrl-x/         # Agent Core (loop, tools, context)
│  ├─ ctrl-l/         # Logger
│  ├─ ctrl-comma/     # Config
│  └─ ctrl-slash/     # Skills
├─ crates/            # later: agentd
├─ native/android-litert/
└─ docs/

Scaffold vs. later packages ​

Jetzt (Phase 0)Später
Root workspaces, bunfig.toml—
packages/ctrl-s, packages/ctrl-actrl-r, ctrl-z, ctrl-space, ctrl-c, ctrl-m, ctrl-t, ctrl-k, ctrl-i, ctrl-x, ctrl-l, ctrl-comma, ctrl-slash
apps/desktop, mobile, web, cliElectron lifecycle, LAN pairing, packaging
Docs + ADRscrates/, native/android-litert/, threat-model

UI-Sharing: Types, Hooks, API-Client, State Machines, Markdown-Renderer, Theme-Tokens und Form-Schemas über ctrl-space / ctrl-i / ctrl-s. Das Desktop-UI (Split-Panes, Shortcuts, Tray) bleibt Vite + React, nicht React Native Web als Primary.

Theme-Akzent: #00ff66 (siehe Namespace).

4. Desktop-Host mit Electron ​

Prozessmodell ​

text
Electron Main
 ├─ Single-Instance-Lock
 ├─ Gateway-Child-Process (ctrl-a Binary, 127.0.0.1)
 ├─ Tray, Autostart, Updater, safeStorage
 └─ BrowserWindow (Renderer, Vite + React)
       └─ Preload: schmale, typisierte IPC-API

Regeln ​

  • Der Renderer nutzt HTTP/WebSocket zum Gateway. IPC nur für Native: Dateidialoge, Tray, Notifications, Autostart, Secrets.
  • contextIsolation: true, nodeIntegration: false, sandbox: true. Jede IPC-Nachricht mit Zod (ctrl-s) validieren.
  • CSP setzen, Navigation und window.open blockieren.
  • SQLite liegt im Gateway (bun:sqlite via ctrl-z). Keine Native-Module-Rebuilds für Electron.
  • Gateway-Token in safeStorage; Übergabe per Env oder Stdin.
  • Sidecars (llama-server) verwaltet der RuntimeManager im Gateway (ctrl-r), nicht Electron.

Gateway-Lifecycle ​

  1. Electron startet die Binary mit zufälligem oder konfiguriertem Port.
  2. Wartet auf /healthz.
  3. Bei Absturz: Restart mit exponentiellem Backoff.
  4. Beim Beenden: SIGTERM, Timeout, dann Kill.
  5. Optional: Daemon-Modus nach UI-Close.

Betriebsmodi ​

ModusBindungZweck
embeddedProzess-intern oder LoopbackAndroid, einfache Desktop-Nutzung
local-daemon127.0.0.1:7331Desktop, CLI, IDE-Plugins
lan-server0.0.0.0:7331nur explizit aktivierbar

LAN: Pairing-Code, Token pro Gerät, Widerruf, Origin/IP-Checks, Rate Limiting, Audit Log, optional TLS.

5. Provider- und Runtime-Abstraktion ​

Intern gilt ein kanonisches Format (in ctrl-s), nicht das OpenAI-Schema.

ts
export interface ModelProvider {
  id: string;
  capabilities(): Promise<ModelCapabilities>;
  listModels(): Promise<ModelDescriptor[]>;
  generate(
    req: GenerateRequest,
    ctx: ProviderContext
  ): AsyncIterable<AgentEvent>;
  cancel?(runId: string): Promise<void>;
}

export type AgentEvent =
  | { type: 'run.started'; runId: string }
  | { type: 'message.delta'; text: string }
  | { type: 'reasoning.delta'; text: string }
  | { type: 'tool.requested'; call: ToolCall }
  | { type: 'tool.started'; callId: string }
  | { type: 'tool.completed'; callId: string; result: unknown }
  | { type: 'usage.updated'; usage: TokenUsage }
  | { type: 'run.completed'; finishReason: string }
  | { type: 'run.failed'; error: SerializedError };

Adapter v1 (ctrl-r): OpenAICompatibleProvider, CliProvider, MockProvider, LlamaCppProvider.

Lokale Runtimes ​

  • Desktop: Gateway verwaltet llama-server als Sidecar.
  • Android: Zuerst externer Endpoint (Desktop-Gateway im LAN); später LiteRT-LM Native Module.

6. Datenmodell und Speicher ​

text
Workspace
├─ Agents (Instructions, Modell, Tools, MCP, Skills, Memory- und Permission-Policy)
├─ Conversations (Messages, Runs, Tool Calls, Attachments, Branches)
├─ Knowledge Sources, Skills, MCP Servers, Provider Profiles, Policies
text
data/  app.db | blobs/ | workspaces/ | skills/ | knowledge/ | models/ | logs/
  • Secrets nicht in app.db: Desktop safeStorage, Android Keystore.
  • Context, Knowledge, Memory, Skills getrennt.
  • Skills ohne automatische Systemrechte — Permission Policy.
  • Versionierte Migrationen; genau ein Backend besitzt die DB.

7. Channels ​

Channels (ctrl-c) sind dünne Transportadapter (start, stop, sendMessage, onMessage).

Reihenfolge: Webhook, Telegram, Discord, Slack, WhatsApp zuletzt.

Identitäten: ExternalUser → ChannelIdentity → LocalUser → Workspace-Rechte.

8. API ​

Interne API unter /api/v1: Workspaces, Conversations, Runs, Providers, Models, Skills, MCP, Pairing.

Kompatibilitätsschicht: /v1/models, /v1/chat/completions, /v1/responses, /v1/embeddings — Adapter auf den Core, nicht das Domain-Modell.

9. Phasenplan ​

Phase 0 – Architecture Spike: Bun Workspace, Expo Android/Web, Electron-Shell, ctrl-s, WebSocket-Streaming, Electron startet Gateway-Binary, Android verbindet sich im LAN. Abbruchkriterium: Dieselbe Conversation auf Desktop und Android, Streaming auf beiden, keine Provider-Logik in der UI.

Phase 1 – Chat MVP: Workspace-/Conversation-CRUD, Streaming, OpenAI-kompatibler Provider, Markdown/Code, Stop/Retry, SQLite (ctrl-z), Export/Import, Mock Provider.

Phase 2 – Agent Runtime: Tool Registry, Approval-UI, Shell-/FS-Tool mit Sandbox, Agent Profiles, Cancellation, Timeouts, Audit Log, MCP (ctrl-m).

Phase 3 – Desktop Host: Electron-Lifecycle, Tray, Autostart, Updater, safeStorage, llama-server-Sidecar, lokaler OpenAI-Endpoint, LAN-Pairing, Packaging.

Phase 4 – Personal Knowledge: Attachments, Extraktion, Chunking, FTS, Memory-Review-UI, Context Inspector (ctrl-k / ctrl-z). Vector/hybrid search follows in Phase 11 (ADR-0004).

Phase 5 – Channels: Telegram, Discord, Slack, Identity Mapping, Queue/Retry, Rate Limits (ctrl-c).

Phase 6 – On-Device-Inferenz (Android): Native Module, LiteRT-LM, Capability Detection, Thermal/Battery-Policies.

Phase 7 – Sync (host-centric): Gateway owns the DB. Clients never sync SQLite files peer-to-peer. MVP = on-host backup snapshots (create / list / restore) of workspace exports under data/backups/; desktop UI + contract tests. Later = passphrase-encrypted backups. Incremental pull/push between devices is out of MVP (see §10).

Phase 8 – Featured CLI (ctrl-f5): Publishable ops tool after Sync. Full local ops: config file (CTRL_* / ~/.config/ctrl/config.json), gateway start/stop/restart/status (spawn ctrl-a), pairing/token, workspaces + interactive/one-shot chat (streamChat), thin wrappers for knowledge/channels. Shell tab-completions for bash and PowerShell (ctrl-f5 completion bash|powershell). Speaks only HTTP/WS protocol via ctrl-i (prep for Rust agentd).

Phase 9 – Rust-Backend (optional): Siehe Abschnitt 11 (was Phase 8).

10. Nicht ins MVP ​

WhatsApp, Cross-Device-Konfliktlösung, autonomes Long-Term-Memory, eigener Vector-DB-Server, Model-Downloader, eingebettetes Android-LLM, Plugin-Marketplace, Multi-Agent-Orchestrierung, Internetzugriff von außen, automatische Skill-Generierung, volle OpenAI-Kompatibilität.

11. Weg zum Rust-Backend (agentd) ​

Voraussetzungen ​

  1. ctrl-s ist Quelle der Wahrheit (JSON Schema / OpenAPI).
  2. Contract-Tests (ctrl-t) gegen jedes Backend über HTTP/WebSocket.
  3. Ein Prozess besitzt die DB; Schema versioniert.
  4. CLI spricht nur das Protokoll.

Migrationsreihenfolge ​

  1. RuntimeManager / Process-Manager als Rust-Sidecar.
  2. Pairing, Auth, LAN-Server.
  3. Storage-Layer bei Bedarf (Schema vorher einfrieren).
  4. Agent Loop nur bei klarem Grund.

Typen für Rust aus dem Schema (typify, progenitor).

12. Contract-Test-Strategie ​

  • Ort: packages/ctrl-t, unabhängig vom Backend; Ziel über BASE_URL.
  • Fixtures: Request/Response- und Event-Sequenzen.
  • Mock Provider: deterministische Events.
  • Prüfungen: JSON Schema, Event-Reihenfolge, Auth/Pairing, Cancel/Timeouts, Migrationen.
  • CI: Gegen Bun-Gateway bei jedem Commit; später parallel gegen agentd.

13. Technische Leitplanken ​

  • Zod an jeder Prozessgrenze (ctrl-s), auch IPC.
  • Strukturierte Events und AbortSignal überall.
  • Provider Capability Detection.
  • Migrationsfähiges SQLite; getrennte Secrets.
  • Permission Policy und Audit Log.
  • Standardbindung 127.0.0.1.
  • Threat Model in docs/threat-model vor Phase 2 und vor LAN.

14. ADR-0001: Electron als Desktop-Runtime ​

Status: Akzeptiert

Kontext: Desktop soll lokales Gateway starten, Tray, Autostart, Updates und Secrets bieten und TypeScript mit dem Monorepo teilen.

Entscheidung: Electron als Shell. Gateway bleibt eigener Bun-Prozess (ctrl-a). Renderer ist normaler Gateway-Client; IPC nur für Native.

Konsequenzen: Höherer RAM/Bundle akzeptiert; CI-Matrix früh; Security-Checkliste pflegen; Tauri später möglich, weil Renderer reines Web ist.

Alternativen: Tauri 2, Ionic/Capacitor, Flutter.

15. Nächste Schritte ​

  1. Monorepo mit Bun Workspaces anlegen; ctrl-s und ctrl-a (/healthz) — done.
  2. Electron-Shell mit Gateway-Lifecycle — done.
  3. Expo-Clients im LAN + WebSocket-Demo-Stream (/ws/v1/demo, @involvex/ctrl-i) — done.
  4. Contract-Tests (ctrl-t) und CI-Matrix — done.
  5. Dependabot, SECURITY.md, tag→GitHub Release workflow — done (see ci-and-releases.md).
  6. Phase 1 Chat MVP — done: ctrl-z SQLite, workspace/conversation/message API, ctrl-r providers (mock + OpenAI-compatible), Markdown chat UI, export/import, contract tests.
  7. Phase 2 Agent Runtime — done: ctrl-x tool registry + sandbox shell/fs, approval gate + desktop UI, cancel/timeouts, audit log, agent profiles, ctrl-m MCP stdio JSON-RPC, LLM-native tool calling (OpenAI-compatible + mock !call).
  8. Phase 3 Desktop Host — done: Tray + close-to-tray, autostart, electron-updater scaffold, safeStorage (gateway token + OpenAI key), gateway bearer auth / LAN pairing, ctrl-r llama-server sidecar (CTRL_LLAMA_BIN), packaging via electron-builder.
  9. Phase 4 Personal Knowledge — done: ctrl-k extract/chunk/FTS, attachments + blobs in ctrl-z (schema v3), memory review + context inspector APIs/UI.
  10. Phase 5 Channels — done: ctrl-c adapters (webhook + mock telegram/discord/slack), identity mapping, outbox queue/retry/rate limits, gateway hooks + desktop panel.
  11. Phase 6 On-Device Inference (Android) — done: LiteRT-LM Expo plugin (react-native-litert-lm), capability probe (ABI/RAM/battery), thermal/battery policy in ctrl-s (re-exported from ctrl-r), mobile /on-device UI with download retries + filesystem .litertlm import (CDN mid-stream failures), expo-dev-client, release ProGuard consumer-rules plugin, native/android-litert docs.
  12. Phase 7 Sync — done (MVP + 7b): host backups under data/backups/ (BackupStore), /api/v1/backups create/list/get/restore, optional passphrase AES-256-GCM encryption, desktop panel + ctrl-f5 backup, contracts. No device↔device SQLite sync.
  13. Phase 8 Featured CLI (ctrl-f5) — done: LAN pair QR; gateway 0.0.0.0; Metro host auto-fill; start/stop/status/chat/backup; completion bash|powershell.
  14. Phase 9 Rust agentd — optional. Provider selection persists in data/provider.json (non-secret); desktop/mobile/web provider + knowledge + tools + channels lockstep. Desktop nav panels are exclusive (one open at a time; Escape closes). Telegram Bot API send adapter (real); Discord/Slack still mock.
  15. Phase 10 — Local-first chat (done this slice):
    • One chat UI with generator picker litert | gateway | zen; /on-device is model setup only.
    • Local brain (ctrl.mobile.brain.v1) + append-only sync when gateway health succeeds.
    • Empty Android ABI counts as arm64 for LiteRT policy; policy disables LiteRT only, not chat.
    • Standalone android:release = Gradle assembleRelease (embedded JS; no Metro :8081/:8082).
    • Out of this pass: Rust agentd, Discord/Slack real SDKs, vector search.
  16. Phase 11 — Personal Knowledge vector & hybrid search (done this slice):
    • Schema v5 knowledge_vectors (raw float32 BLOBs, cascade with chunks); cosine scoring in-process inside ctrl-k (see ADR-0004).
    • Embedder contract in ctrl-s; ctrl-r embedders hash (deterministic default, offline), openai/llama/litert via OpenAI-compatible /embeddings, off.
    • searchKnowledge gains mode: fts | vector | hybrid; hybrid fuses bm25 + cosine with reciprocal rank fusion (k = 60); hits gain score + source.
    • Embed on ingest (failure tolerated), POST /api/v1/workspaces/:id/knowledge/reindex, GET /api/v1/embeddings status; desktop knowledge panel gets mode picker, status line, reindex button; unit + contract tests.
    • Out of this pass: ANN index (sqlite-vec/HNSW), cross-workspace search, re-embedding on model switch (use reindex --force), semantic reranking.
  17. Ship v0 (standalone daily-driver) — see standalone-local.md, soak-v0.md, RELEASE-v0.md.
    • Parked until soak is boring: in-app binary downloaders, HF catalog, Discord/Slack real SDKs, incremental device sync, Rust agentd, monetization.

Local agent platform under @involvex

Sponsors → /funding · Security → /security