Self-Hosted AI Code Completion Engines in Production: Comparing Tabby, Continue, Refact.ai, and Twinny Architecture, Fill-in-the-Middle Pipelines, AST Context Retrieval, and Serving Latency

Deploying AI-assisted code completion within air-gapped environments, regulated enterprises, and privacy-conscious engineering teams requires decoupling developer tooling from third-party cloud APIs. While agentic coding systems focus on multi-turn planning and repository-wide refactoring with relaxed latency budgets, inline code completion operates under strict real-time constraints. Developers expect ghost-text suggestions to appear within 100 to 200 milliseconds of pausing keystrokes.

Delivering accurate inline suggestions without cloud telemetry requires specialized model architectures trained on Fill-in-the-Middle (FIM) objectives, combined with local context assembly and low-latency inference runtimes. Four open-source and self-hosted tools have emerged to solve this problem across different deployment tiers: Tabby, Continue, Refact.ai, and Twinny.

Fill-in-the-Middle (FIM) Mechanics and Prompt Framing

Standard autoregressive language models generate tokens strictly from left to right, conditioned only on preceding tokens. In real-world code editing, the cursor is rarely at the end of a file; code exists both before the cursor (the prefix) and after the cursor (the suffix). Left-to-right generation fails in this context because the model cannot condition its output on existing closing brackets, subsequent method calls, or type declarations below the cursor.

Fill-in-the-Middle (FIM) pre-training resolves this by transforming documents during training. A document is partitioned into three segments: Prefix ($P$), Middle ($M$), and Suffix ($S$). The model is then trained autoregressively on rearranged sequences:

• Prefix-Suffix-Middle (PSM): Format is [PREFIX] P [SUFFIX] S [MIDDLE] M
• Suffix-Prefix-Middle (SPM): Format is [SUFFIX] S [PREFIX] P [MIDDLE] M

During inference, the completion engine formats the code surrounding the cursor into the prompt and prompts the model to generate the middle segment until an end-of-sequence token or indentation boundary is encountered.

Special token conventions vary across open-weight model families:

• Qwen2.5-Coder: Uses <|fim_prefix|>, <|fim_suffix|>, and <|fim_middle|>, alongside <|repo_name|> and <|file_sep|> for multi-file contexts.
• StarCoder2: Employs <fim_prefix>, <fim_suffix>, and <fim_middle>.
• DeepSeek-Coder: Uses <｜fim begin｜>, <｜fim hole｜>, and <｜fim end｜>.
• CodeLlama: Uses <PRE>, <SUF>, and <MID>.

Tabby: Monolithic Rust Daemon and Centralized Indexing

Tabby implements a centralized, client-server architecture designed for team-wide self-hosting. Written in Rust, Tabby runs as a single standalone daemon or container that handles git synchronization, static analysis, full-text indexing, and model inference.

Architectural Components

1. Git Synchronization and Repository Ingestion: Tabby periodically clones and pulls configured team repositories via Git, maintaining local working trees on the server filesystem.
2. Abstract Syntax Tree (AST) Parsing: Using Tree-sitter, Tabby extracts symbol definitions, class signatures, and function interfaces across every file in the indexed repositories.
3. Hybrid Search Index: Extracted symbols and chunked source files are indexed into an embedded Tantivy full-text engine alongside vector embeddings for lexical and semantic retrieval.
4. Context Construction and Inference: When an IDE client (supporting VS Code, JetBrains, Vim, and Neovim) fires a completion request, Tabby extracts relevant symbol declarations from Tantivy, prefixes them into the prompt, and dispatches the job to its built-in inference runtime or an external OpenAI-compatible engine.
5. Team Governance: Tabby provides an administrative web dashboard with role-based access control, token usage analytics, and completion acceptance rate tracking.

Because indexing and inference occur centrally, individual developer machines do not bear the compute overhead of indexing large repositories.

Continue: Client-Centric IDE Middleware and Embedded Vector Search

Continue operates on an entirely client-side, local-first paradigm. It is packaged as an IDE extension for VS Code and JetBrains, acting as a modular orchestrator between the editor buffer and external inference backends such as Ollama, vLLM, or LM Studio.

Architectural Components

1. Local-First Storage: Continue maintains configuration and data in the developer's home directory (~/.continue/). It utilizes SQLite for transactional history and LanceDB as an embedded vector database.
2. In-IDE Context Aggregation: Continue crawls the local workspace, splitting code into semantic chunks with Tree-sitter and embedding them into LanceDB locally.
3. Decoupled Model Configuration: The extension separates configuration for chat models from completion models (tabAutocompleteModel). This allows developers to route fast FIM completions to a local 1.5B or 3B model while dispatching chat and refactoring tasks to larger hosted or remote endpoints.
4. Zero Server Footprint: Continue requires no central infrastructure. Team policies can be distributed by committing a standardized config.yaml or config.json into repository dotfiles.

The primary limitation of Continue's client-centric design is that each developer must index codebases locally, and team-wide usage metrics cannot be captured centrally without additional telemetry proxies.

Refact.ai: On-Premises Cluster and Continuous Model Fine-Tuning

Refact.ai targets enterprise deployments requiring customized models trained directly on internal codebases. It combines an on-premises model server, IDE plugins, and an automated fine-tuning pipeline.

Architectural Components

1. Integrated Fine-Tuning Engine: Unlike general-purpose completion proxies, Refact includes automated LoRA (Low-Rank Adaptation) training pipelines that fine-tune base models (such as Refact-1.6B or StarCoder2) on private repositories on a scheduled cadence.
2. Style and Idiom Adaptation: By fine-tuning weights directly on internal code patterns, Refact reduces the reliance on large prompt context injection, as company-specific APIs, abstractions, and naming conventions are internalized into model parameters.
3. Multi-Model Orchestration: The Refact server hosts both completion and chat endpoints, managing GPU memory allocation, dynamic batching, and model switching within a unified container cluster.
4. Telemetry and Analytics: Refact tracks acceptance rates, keystroke metrics, and generated line counts across engineering teams, reporting these via an administrative dashboard.

The operational overhead of Refact is higher than pure inference gateways, as training nodes require sustained GPU compute for periodic model updates.

Twinny: Minimalist Zero-Telemetry Autocomplete Extension

Twinny is a lightweight, zero-dependency VS Code extension engineered exclusively for low-latency completions and privacy.

Architectural Components

1. Zero Central Dependencies: Twinny eliminates analytics, telemetry, and background sync services.
2. Template-Driven FIM Engine: Developers can define raw prompt templates directly in the settings UI, allowing instant compatibility with custom prompt formats across llama.cpp, Ollama, or vLLM backends.
3. Optional Gateway Daemon (twinny-server): For small teams sharing a local workstation GPU, Twinny provides a lightweight Node.js gateway that forwards requests without storing state.

Twinny lacks full-repository vector indexing and semantic AST symbol resolution, relying primarily on adjacent open editor tabs and cursor proximity for context. In return, it introduces near-zero editor overhead and minimal CPU usage.

Serving Latency, Hardware Requirements, and GPU Economics

Inline code completion has much tighter latency limits than interactive chat. User experience studies indicate that completion ghost text must begin rendering within 150 to 200 milliseconds of keystroke inactivity to avoid disrupting typing rhythm.

Latency Budget Breakdown

1. Debounce Interval (100–150ms): The editor waits for typing to pause before issuing an HTTP request, preventing unnecessary inference calls on mid-word keystrokes.
2. Context Extraction and Prompt Assembly (10–30ms): Tree-sitter AST queries, index lookups, and token framing must execute with minimal latency.
3. Time-To-First-Token / TTFT (30–80ms): The inference engine must process prefill tokens and emit the first completion token.
4. Token Generation (20–40ms): Generating 16 to 32 tokens at 100+ tokens per second ensures multi-line completions stream without perceptible lag.
5. Client Rendering (<10ms): Ghost text is rendered in the editor buffer.

Hardware Sizing for FIM Models

• 1.3B – 1.5B Parameters (e.g. Qwen2.5-Coder-1.5B): Requires 2 GB to 4 GB VRAM (FP16 or INT8). Runs on modern laptop hardware (Apple Silicon or CPU AVX-512) with 30–60 ms TTFT. Ideal for solo local development and ultra-low latency autocomplete.
• 3B Parameters (e.g. StarCoder2-3B, Qwen2.5-Coder-3B): Requires 4 GB to 8 GB VRAM (FP16 or INT4). Runs on entry-level GPUs (RTX 4060 or Apple M-series) with 50–90 ms TTFT. Delivers balanced local desktop completions.
• 7B Parameters (e.g. Qwen2.5-Coder-7B, DeepSeek-Coder-6.7B): Requires 8 GB to 12 GB VRAM (FP8 or AWQ). Runs on mid-tier GPUs (RTX 4080, RTX 4090, or A10G) with 80–140 ms TTFT. Suitable for high-accuracy shared team completion servers.
• 14B Parameters (e.g. Qwen2.5-Coder-14B): Requires 16 GB to 24 GB VRAM (FP8 or INT4). Runs on enterprise server GPUs (A100 or L40S) with 120–200 ms TTFT. Provides deep multi-file and cross-module reasoning.

Request Cancellation and Abort Controllers

Because developers frequently resume typing before a completion finishes generating, client extensions must immediately cancel in-flight requests using standard HTTP AbortController signals. Runtimes supporting chunked prefill and request preemption (such as vLLM and Tabby's native inference queue) immediately discard canceled generation tasks, freeing GPU compute cycles for subsequent keystrokes.

Architectural Trade-Offs and Selection Framework

Choosing among self-hosted code completion engines depends on team size, compliance constraints, and infrastructure maturity:

• Choose Continue when the priority is client-side flexibility, allowing individual developers to switch between local backends (Ollama/vLLM) and various chat models without deploying central server infrastructure.
• Choose Tabby when deploying a standardized, team-wide code completion service with centralized Git repository indexing, role-based access control, and usage analytics.
• Choose Refact.ai when organization-specific coding patterns, internal APIs, and proprietary libraries require regular on-premises model fine-tuning.
• Choose Twinny when running on resource-constrained single-developer environments where minimal memory footprint and zero telemetry take precedence over repository-wide search.

Sources

• Efficient Training of Language Models to Fill in the Middle (arXiv:2207.14255)
• Qwen2.5-Coder Technical Report (arXiv:2409.12186)
• DeepSeek-Coder: When the Large Language Model Meets Programming (arXiv:2401.14196)
• CodeLlama: Open Foundation Models for Code (Meta AI)
• StarCoder2 and The Stack v2 (Hugging Face)
• Tree-sitter Code Navigation Documentation
• TabbyML GitHub Repository and Architecture
• Continue Documentation and LanceDB Architecture
• Refact.ai Self-Hosted Fine-Tuning Architecture
• Twinny Open-Source AI Coding Assistant (GitHub)