Trackshot • In Development 2026
Sub-Millisecond Semantic Search for Local Screenshots
Trackshot is a local-first desktop engine that turns unorganized screenshot folders into an instantly searchable visual knowledge base using quantized vision-language models, Apple Silicon neural acceleration, and USearch SIMD vector indexing.
Role
Systems & Desktop Software Engineer
Timeline
December 2025 – January 2026
Team
Solo
Skills
Multimodal Embeddings • USearch Vector Indexing • CoreML / ANE • Swift • TypeScript • Electron
Overview
No screenshot goes to waste.
Software engineers, designers, and researchers capture dozens of screenshots every day: UI inspiration mockups, compiler stack traces, architecture diagrams, benchmark graphs, and quick chat snippets.
Almost immediately, these files disappear into ~/Desktop or ~/Downloads under default filenames like Screenshot 2026-01-15 at 11.42.08.png.
Operating system search tools like macOS Spotlight rely primarily on exact filename matches or basic OCR text strings. When a user tries to find "that minimalist dark mode dashboard with a circular progress gauge" or "the Docker out-of-memory error from last Tuesday," traditional search fails completely.
Trackshot eliminates this visual blind spot. It watches screenshot directories in real time, generates dense multimodal vector embeddings on the Apple Neural Engine (ANE), extracts text via hardware OCR, and delivers natural language visual search in under 10 milliseconds—entirely offline with zero cloud dependency.
Problem
Screenshots are the fastest way to capture context, but the slowest to retrieve.
Existing desktop search and organization tools suffer from three fundamental architectural flaws:
1. The Semantic Understanding Gap
Standard optical character recognition (OCR) only detects verbatim textual strings. It cannot comprehend layout, aesthetics, UI component hierarchies, color schemes, or visual diagrams.
2. Severe Privacy and Compliance Risks
Sending personal screenshots to external vision APIs exposes proprietary codebases, AWS console screens, customer data, and authentication tokens to third-party cloud servers.
3. Background Indexing Overhead
Traditional Python-heavy vision pipelines consume gigabytes of RAM and induce high CPU spikes, causing laptop fans to spin and battery life to degrade during routine desktop work.
Solution
Dual-stream hybrid indexing with native hardware acceleration.
Trackshot bridges the gap between vision-language models and native desktop performance by combining a lightweight background daemon with an ultra-responsive Spotlight-style search overlay.
Natural Language Visual Queries
Search for images using descriptive concepts ("dark mode mobile checkout", "flame graph showing CPU bottleneck", "Figma component variant").
Dual-Stream Hybrid Retrieval
Combines dense multimodal vector similarity with sparse BM25 OCR token search via Reciprocal Rank Fusion (RRF).
Sub-Millisecond USearch Engine
Leverages memory-mapped USearch vector indexes with SIMD hardware acceleration (ARM NEON and AVX-512) for instant query execution.
100% Private, Zero-Cloud Execution
All neural inference, vector indexing, and image processing run locally on-device with zero network telemetry.
Core Architecture
The dual-stream indexing pipeline: Dense vectors meet sparse OCR tokens.
When a new screenshot is saved to the filesystem, Trackshot processes the image through an asynchronous, two-stage extraction pipeline:
┌─────────────────────────────────────────────────────────────┐
│ macOS Filesystem (FSEvents) │
│ ~/Desktop/Screenshot-2026-01-15-at-14.22.png │
└──────────────────────────────┬──────────────────────────────┘
│ Non-blocking Event
┌──────────────────────────────▼──────────────────────────────┐
│ Trackshot Ingestion Daemon │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Perceptual Hash (pHash) Deduplication Check │ │
│ └──────────────────────────┬──────────────────────────┘ │
│ │ │
│ ┌────────────────────┴────────────────────┐ │
│ │ │ │
│ ▼ ▼ │
│ [ Stream A: Dense Vision ] [ Stream B: Sparse Text ]│
│ - Quantized UForm / SigLIP - Apple Vision OCR Engine│
│ - 512-dim Float16 Vector - Full Text Tokens & BBox│
│ - Apple Neural Engine (ANE) - Inverted Keyword Index │
│ │ │ │
│ └────────────────────┬────────────────────┘ │
│ │ │
│ ┌──────────────────────────▼──────────────────────────┐ │
│ │ Hybrid Index Storage & Memory Map │ │
│ │ - USearch Native Vector Index (.usearch) │ │
│ │ - SQLite FTS5 Metadata Store │ │
│ └─────────────────────────────────────────────────────┘ │
└──────────────────────────────┬──────────────────────────────┘
│
┌──────────────────────────────▼──────────────────────────────┐
│ Search & Retrieval Flow │
│ User Query: "kubernetes pod out of memory trace" │
│ → Dense Cosine Sim + Sparse BM25 → Reciprocal Rank Fusion │
│ → Result displayed with bounding box in <10ms │
└─────────────────────────────────────────────────────────────┘
Technical Deep Dives
1. Dual-Stream Hybrid Retrieval Engine
Visual search over screenshots requires understanding both conceptual scene aesthetics and exact textual matches (such as specific error codes, port numbers, or variable names).
Trackshot implements a hybrid scoring function that combines normalized cosine similarity from dense vision vectors with BM25 keyword rankings from Apple Vision OCR tokens:
// Hybrid ranking calculation using Reciprocal Rank Fusion (RRF)
func computeHybridRank(denseRank: Int, sparseRank: Int, k: Double = 60.0) -> Double {
let denseScore = 1.0 / (k + Double(denseRank))
let sparseScore = 1.0 / (k + Double(sparseRank))
return (0.65 * denseScore) + (0.35 * sparseScore)
}
This hybrid approach ensures that a search for E0502 compiler error anchors immediately on exact OCR tokens, while a query for clean glassmorphism card layout retrieves relevant designs even when no matching text is present.
2. Ultra-Fast Vector Search with USearch
Traditional vector databases (like Chroma or Pinecone) introduce heavy runtime dependencies, high memory overhead, and significant query latency when integrated into client desktop apps.
Trackshot integrates USearch, a header-only C++ vector search library with native Swift bindings. USearch builds Hierarchical Navigable Small World (HNSW) graphs directly on disk and memory-maps the index file:
- Zero Memory Overhead: Indexes are mapped directly from the filesystem (
.usearch), avoiding duplicate heap allocations. - Hardware SIMD Acceleration: Distance calculations utilize ARM NEON instructions on Apple Silicon and AVX-512 on x86_64 chips.
- Sub-10 Microsecond Retrieval: Cosine similarity searches across 50,000 screenshot embeddings resolve in under 0.05ms.
3. Real-Time Ingestion Without System Slowdowns
To ensure background indexing never interferes with active developer workflows, Trackshot implements a multi-stage throttle:
- FSEvents File Monitoring: Subscribes to low-overhead macOS kernel filesystem notifications instead of polling directory trees.
- Perceptual Hashing (pHash): Fast 64-bit perceptual hashing instantly discards duplicate captures and near-identical window frames before triggering neural inference.
- Dedicated Neural Engine Dispatch: Quantized INT8 weights execute strictly on the Apple Neural Engine (ANE), keeping CPU and GPU cores completely idle for the user's IDE and compilers.
Indexing Pipeline Performance:
├── Image Load & Downsample: 1.8ms
├── Perceptual Hash Check: 0.2ms
├── Vision Model Inference: 14.2ms (ANE)
├── OCR Text Extraction: 6.1ms
└── USearch Index Insertion: 0.3ms
───────────────────────────────────
Total End-to-End Latency: 22.6ms per screenshot
4. The Zero-Cloud Privacy Guarantee
Trackshot operates under a strict offline-first security contract:
- No Remote Network Calls: The desktop binary makes zero outgoing network connections.
- Local Key & Secret Shielding: Screenshots containing detected SSH keys or private certs are marked with encrypted metadata tags.
- Portable Data Store: All indexes and thumbnails live in
~/.trackshot/, allowing instant backup, encryption, or deletion.
Outcomes & Benchmarks
Verified performance on real-world screenshot datasets.
Tested across a benchmark dataset of 25,000 developer screenshots, design mockups, and technical documents:
8.4ms Average Query Latency
From pressing Return in the search overlay to rendering visual image results on screen.
42 Images / Second Indexing Throughput
Continuous batch indexing rate during initial folder onboarding on an Apple M3 chip.
Sub-85MB Steady-State RAM
Memory footprint maintained under normal background monitoring mode.
Reflection
What I learned building on-device multimodal search.
Pure vector search is incomplete without lexical precision.
While multimodal embeddings excel at understanding broad visual themes, they struggle with tiny, specific strings like UUIDs, error numbers, or hex color codes. Unifying dense neural vectors with sparse OCR tokens via Reciprocal Rank Fusion was essential to creating a reliable search experience.
On-device AI outperforms cloud APIs for personal productivity.
Running quantized models locally with memory-mapped vector search is not just a privacy win; it is an order of magnitude faster than sending requests over the public internet. Sub-10ms latency transforms search from an intentional chore into an instinctive reflex.