on-devicearchitectureprivacy

Building a Private, On‑Device Browser Agent (like Puma): Architecture for Mobile Semantic Search

UUnknown

2026-02-19

10 min read

Design a privacy‑first on‑device browser agent: run embeddings, FAISS, and quantized LLMs locally with secure sync and production patterns for mobile vector search.

Hook: Why building a private, on‑device browser agent is urgent for 2026

Developers and DevOps teams shipping search and assistant features face the same harsh tradeoffs in 2026: user privacy, latency, cost, and relevance. The Puma browser story — a lightweight browser that runs Local AI on iPhone and Android — crystallizes a practical alternative: run embeddings, a vector DB and a small LLM on device so users keep data private while getting near‑server quality answers. This article lays out a production architecture inspired by Puma/local browser work, with actionable patterns for mobile vector search, on‑device FAISS, index updates, model selection, quantization and sync strategies that scale.

Executive summary — most important points first

Architecture pattern: capture → embed → index (FAISS) → retrieve → rerank with a small local LLM.
Privacy-first: default to on‑device embeddings and encrypted sync; server helps only for optional global features.
Performance: combine HNSW/IVF + PQ and aggressive quantization (4–8 bit) to fit large corpora on mobile.
Model ops: maintain a small family of quantized encoder and LLM artifacts for different device classes (high‑end, midrange, low‑end).
Sync: differential sync with content hashing, Bloom filters, and CRDT‑style merges for concurrent edits.

Why Puma's approach matters for DevOps and product teams

Puma and similar local browser projects demonstrated a shift in 2024–2026: users prefer devices that process sensitive content locally (tabs, history, bookmarks, form data) and use small LLMs for context‑aware actions. For engineering teams, this means rethinking server-centric retrieval architectures. You need a reproducible, scalable pattern to run a private vector DB and LLM on a user’s phone without sacrificing relevance or battery life.

Topline tradeoffs

Privacy vs model capacity: smaller models fit on‑device but sometimes lower quality. Use reranking with a few local tokens or server fallback for heavy tasks.
Storage vs recall: quantized indexes reduce size but affect neighbor accuracy. Tune PQ bits and HNSW ef parameters.
Latency vs battery: batching and background indexing reduce perceived cost but must respect system sleep policies.

Architecture overview: components and flow

Below is a pragmatic architecture for a mobile browser agent similar to Puma that supports private semantic search and local assistants.

Core components

Data capture layer — browser/webview hooks, OS share sheets, and file listeners collect content (pages, notes, PDFs).
Preprocessing pipeline — tokenization, chunking, canonicalization, language detection.
Embedding model — on‑device encoder (quantized) that produces fixed‑length vectors.
Vector store (on‑device FAISS) — local FAISS index for nearest neighbor search (HNSW/IVF + PQ variants).
Rerank LLM — compact instruction/tiny LLM for context and answer synthesis (GGML/TFLite/ONNX).
Sync/Cloud service (optional) — encrypted metadata store and optional ensemble index for cross‑device search.
Telemetry & monitoring — local logs, opt‑in performance telemetry, server health endpoints for sync service.

Choosing the right embedding and LLM models in 2026

By 2026 there are mature tiny encoders and condensed LLMs that trade far less compute for good semantic quality. Your selection strategy should be:

Pick a family: high‑end (6–12B quantized), midrange (1–3B), low‑end (<1B).
Use specialized encoders for semantic search (sentence models), not generation models repurposed as encoders.
Provide dynamic selection by device class and battery/thermal state.

Examples of on‑device artifacts common in 2025–2026: quantized Llama‑family guests (4‑bit/8‑bit), Mistral tiny variants, and compact sentence encoders converted to TFLite/ONNX/ggml. The production pattern is to ship encoder + LLM models as versioned artifacts and to enable remote patching for security or quality updates.

On‑device FAISS: index formats and strategies

FAISS is battle‑tested for ANN. For mobile, you must tailor index configuration to memory and CPU constraints.

Index choices and when to use them

Flat — exact search, small corpora (<10k vectors). Use for high precision but not scalable on phones.
HNSW — great recall/latency tradeoff, incremental updates, good for dynamic personal corpora (recommended default).
IVF + PQ — best when you must fit millions of vectors; combine IVF (coarse clustering) with PQ (product quantization) for space savings.
Hybrid — HNSW over PQ centroids or HNSW + OPQ (optimized PQ) for larger sets with faster updates.

Quantization and accuracy

Quantization is your primary lever to reduce footprint. In 2026, mobile pipelines commonly use:

8-bit float/int for safe quality and simple compatibility.
4-bit (GEMM-wrap / GPTQ) for LLM weights and sometimes PQ codebooks; expect small recall drops but dramatic size wins.
OPQ + PQ to bring down vector store size while retaining high recall.

Practical FAISS example (pseudocode)

Below is a simplified workflow to create an HNSW index with PQ on device (C++/NDK or via WASM). Translate to your runtime as needed.

// Pseudocode: build & persist an HNSW + PQ index
int dim = 768;
IndexHNSWFlat hnsw(dim);
// optional: fine‑tune HNSW parameters
hnsw.hnsw.efConstruction = 200;

// Add vectors incrementally
for (vector in new_chunks) {
  vec = embed(vector.text);
  hnsw.add(vec);
}

// Serialize index to local protected storage
hnsw.save("/data/user/0/app/files/faiss_index.bin");

Index updates and incremental workflows

Real users constantly add pages, bookmarks, screenshots and notes. Your architecture must support streaming updates without costly rebuilds.

Incremental add/delete patterns

HNSW native adds — HNSW supports efficient single‑vector insertion. Use periodic rebalancing (rebuild with higher efConstruction overnight) for optimal connectivity.
Soft deletes — mark vectors as deleted in metadata; compact periodically.
Chunked rebuilds — when PQ/IVF needs re‑training, perform a background compact on device and swap atomically.

Safe persistence

Persist indexes atomically and encrypt at rest. On iOS, use the keychain to derive an encryption key stored in Secure Enclave. On Android, use Android Keystore + StrongBox TEE where available.

Sync strategies — balancing privacy and cross‑device features

Privacy‑first apps often offer optional cross‑device search. Use these patterns:

Differential, encrypted sync: compute SHA‑256 hashes of content chunks and embeddings. Upload only new hashes and encrypted payloads when the user opts in.
Bloom filters for membership checks: server computes approximate set difference to avoid uploading full indexes.
CRDT or version vectors: for concurrent edits (notes/bookmarks) use CRDTs so clients can merge offline and then reconcile deterministically.
Zero‑knowledge server mode: server stores only encrypted blobs and metadata; server cannot decode user vectors or content.

Sync flow example

Client computes embeddings and chunk metadata, stores locally.
Client sends content hashes + FNV64/sha signatures to the sync service.
Service replies with missing hashes. Client uploads encrypted blobs for missing items.
Server stores encrypted blobs; optional server index stores obfuscated embeddings (e.g., permuted or homomorphically transformed) for cross‑device search only if user opts in.

Design rule: treat the server as a backup and coordination plane, not a plaintext search engine when privacy is required.

Retrieval pipeline: from vectors to high‑quality answers

A typical runtime retrieval flow in the browser agent:

Index lookup: nearest neighbors from FAISS (k = 10–100 depending on corpus size).
Coarse‑filter: remove outdated or low‑confidence matches based on metadata and timestamp.
Rerank & fuse: use a compact LLM or cross‑encoder to score top candidates by contextual relevance.
Answer synthesis: local LLM composes the final answer, optionally invoking server for heavy compute or web‑referenced facts.

Latency and accuracy knobs

Increase efSearch in HNSW for recall at cost of CPU.
Use asymmetric quantizers or hybrid exact search for top‑k rerank to recover quality.
Cache embeddings for repeated pages and warm indexes at app start.

Model ops: rollout, compatibility and A/B testing

Key operational patterns for 2026:

Versioned model bundles: embedder + LLM + index metadata shipped as a single artifact with semantic versioning.
Canary updates: roll new model bundles to a small cohort; measure recall@k and latency plus user satisfaction signals before full rollout.
Telemetry hygiene: collect only anonymized performance metrics and opt‑in relevance labels to improve models.

DevOps and CI/CD for on‑device AI

Operationalizing on‑device AI is not the same as deploying a server. Consider:

Artifact signing: sign model bundles and index migration scripts to prevent tampering.
Binary size limits: use delta updates for model patches and differential downloads (Brotli/bsdiff) to reduce user data usage.
Automated benchmarks: for each model candidate run battery, latency and recall tests on a matrix of device profiles (low‑end CPU, midrange, flagship with NPU) in CI.

Monitoring and success metrics

Track a compact set of KPIs:

Recall@10, MRR: measured via offline evaluation sets and user‑labeled data when available.
Latency P50/P95: end‑to‑end query to answer times on device.
Battery impact & memory footprint: measured on representative devices; set SLAs per user tier.
Sync reliability: success rate of differential sync and encrypted upload throughput.

Security and privacy hardening

Implement these protections by default:

Encrypted model & index at rest: AES‑GCM with keys in OS keystore or Secure Enclave.
Policy sandboxing: run model inference in isolated processes to limit attack surface.
Opt‑in telemetry: require explicit user consent for any performance or usage logs.
Serverless fallback: for server fallback, use ephemeral keys and short‑lived tokens; never persist PII on servers unencrypted.

Operational case study: mapping the Puma/browser experience to production patterns

Puma demonstrated a minimalist approach: let users choose local LLMs and run operations locally. If you map Puma to a production architecture, you get:

Onboarding: user chooses privacy mode; device checks available model bundles and downloads the appropriate quantized encoder + LLM.
Capture: browser intercepts page text and creates chunked HTML snapshots with metadata.
Embed & index: chunks are embedded via a TFLite/ggml encoder and inserted into a local HNSW index, saved encrypted.
Local assistant: queries run entirely on device; server used only for optional cross‑device sync with encrypted blobs.

Key implementation lessons from Puma style deployments:

Keep default UX private and offline; make cloud features opt‑in.
Bundle a small but expressive model set for offline quality; allow advanced users to drop in a custom model artifact.
Automate index compaction during idle times to minimize user impact.

Future trends and 2026 predictions

Looking ahead from early 2026, expect these trends to solidify:

Hardware NPU ubiquity: by late 2026 more midrange devices will include NPUs accelerating 4‑bit matrix ops, pushing much larger quantized models on device.
Standardized on‑device runtime APIs: WebNN/WebGPU and unified ONNX/TFLite backends will make cross‑platform model packaging easier.
Privacy primitives: secure enclaves and hardware attestation will enable stronger guarantees for on‑device ML artifacts and verified model provenance.
Federated evaluation: federated fine‑tuning and private relevance feedback loops will improve quality without revealing raw data.

Actionable checklist to implement this architecture

Inventory device classes you must support and choose three model tiers (high/mid/low).
Prototype an on‑device encoder + FAISS HNSW pipeline on one Android and one iOS device; measure P95 latency & memory.
Implement encrypted persistence and atomic index swaps for safe updates.
Build a differential sync service that accepts encrypted blobs and returns membership via Bloom filters.
Set up CI jobs that run quantized model benchmarks and run canary rollouts to a subset of users.

Conclusion — the operational payoff

Shipping a private on‑device browser agent like Puma means rethinking where retrieval and generation happen. With the right combination of quantized encoders, FAISS‑based indexes configured for mobile, and careful sync/ops patterns, you can offer fast, private, and high‑quality semantic search on phones. The payoff is clear: lower server costs, better privacy guarantees, and reduced latency — all of which improve user trust and retention.

Next steps and call to action

If you’re evaluating a migration from server‑heavy semantic search to a privacy‑first on‑device architecture, start with a small pilot: select a representative corpus (5k–50k items), embed it on device using a quantized encoder, and run an HNSW index with efSearch and efConstruction tuning. Measure recall, latency and battery. If you want a reproducible template, download our starter kit (model selection matrix, FAISS config presets and sync server reference) and run the 7‑day pilot. Get in touch with our engineering team to customize the pipeline for your product and device fleet.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.