Jimmy
Always-on ambient assistant for Windows 11
Remembers what you saw and heard, answers when you ask, and otherwise stays quiet.
5/5
Build stages shipped
59
Assert-based checks
4/hr
Card ceiling
25.8MB
Storage per hour
01The problem
You saw it, you heard it, and now you can't find it — the deadline on a tab you closed on Tuesday, the thing someone said on a call. Continuous capture is a commodity. The hard part is an assistant that knows when not to interrupt.
02The approach
Screen text and nearby speech become one local, searchable history with a blurred screenshot for every moment. Say “Jimmy, …” out loud and the answer arrives beside the exact moments it came from. Built for six good interruptions an evening, not for throughput.
03Architecture — the camera follows the data
How it works
> Capture never leaves the laptop — only the evidence shown with an answer does.
Step 1: Capture
DXGI frames · UI text · mic
Step 2: Blur
faces blurred, exclusions first
Where the work happens
Step 3: Store
SQLite FTS5 · bge-m3 vectors
Step 4: Gate / route
rules, then qwen2.5:3b
Step 5: Overlay
answer beside its evidence
Where the work happens
//Launch film
04Perks — why it's built this way
The edge
Privacy by construction
Faces are blurred before anything is written, and a test re-runs face detection on the saved JPEG and requires zero hits. There is no faces table — the absence is the design.
Hybrid recall
SQLite FTS5 keyword matches fused with local bge-m3 meaning search, so “consulting application” finds a page titled “McKinsey Forward”.
The model never writes a card
It only answers yes/no questions. Code writes the card from words that exist in the evidence and real timestamps, so a card can't contain anything invented.
Knows when to shut up
At most 4 cards an hour, never two within 10 minutes. Dismiss one and it stays quiet for 30. Silence is the default.
Asks back
“What's this?” could mean the screen now or one from last week — so it asks, then waits for a spoken answer. No wake word needed for the reply.
05Tech — and what each part does
The stack
- Python 3.12
- Capture loop, store, trigger gate
- DXGI + UI Automation
- Screen frames and on-screen text
- Whisper large-v3-turbo
- Local speech-to-text
- SQLite + FTS5
- History and keyword search
- bge-m3 (Ollama)
- Local meaning search
- qwen2.5:3b (Ollama)
- Yes/no card decisions
- NVIDIA Nemotron
- Optional spoken answers
- Electron + React
- Click-through overlay
06Evidence
The proof
07Known limits — stated plainly
Where it doesn't win
- The model reads text, not pixels — canvases, video and some apps' main panes aren't described yet.
- No retention policy yet. Everything stays on your own disk: about 6 GB a month at 8 hours a day with 640 px thumbnails.
- A face embedding is a biometric template under India's DPDP Act even if never persisted. The design shrinks that exposure substantially; it doesn't take it to zero.