Memo 001
Domain-Specific SLM Fine-Tuning vs. Frontier RAG
This memo reports one evaluation. Assessments stay vendor neutral.
1. Executive Summary & Verdict
The Industry Hypothesis: Fine-tuning a 4B/8B parameter model lets enterprises bypass expensive frontier APIs for domain knowledge.
Finding 1. Closed-book fine-tuning fails at scale
Small models do not reliably memorize dense procedural specifications. The best SLM closed-book scored 0.344 compared to Claude Sonnet at 0.454.
Finding 2. Retrieval is the true lever
Moving from keyword to hybrid retrieval lifted accuracy across every model significantly, dwarfing the gains from repeated LoRA tuning.
Finding 3. The Production Tradeoff
Untuned Base 4B + Hybrid RAG delivered 84.1% accuracy at $0.0001/query, while Sonnet delivered 95.8% accuracy at $0.019/query. Fine-tuning did not improve raw RAG accuracy, but it cut latency by 75% and reduced verbosity.
2. The Macro Tradeoff Matrix (FinOps & Unit Economics)
| Architecture | Eval Accuracy (RAG) | Single-Request Latency | Est. Cost / Query | Enterprise Fit |
|---|---|---|---|---|
| Claude Sonnet API | 0.958 | 7.0 s | $0.0190 | Mission-critical, zero-fault tolerance |
| Untuned Base 4B (Self-Hosted) | 0.841 | 4.9 s | $0.0001 | Broad internal search, budget cap |
| Fine-Tuned 4B (LoRA) | 0.788 | 1.2 s | $0.00008 | High-throughput agents, strict conciseness |
3. Deep Dive: What the Data Actually Proved
The Limits of Closed-Book Adaptation
Parameterized memory collapses on exact specs, torque ratings, and error codes.
The Base Model Paradox
The untuned base model achieved higher broad recall (0.841) than the fine-tuned 8B (0.794). The fine-tuned 4B in the tradeoff matrix is 0.788.
What Fine-Tuning Actually Bought
Collapsed output latency (from 4.9 s to 1.2 s), slashed unsupported detail flags on answerable items (from 0.65 down to 0.12), and restored proper abstention behavior.
4. Qualitative Failure Analysis
Where Sonnet dropped points
Real edge-case manual exclusions (e.g., claiming data wasn't in the passage when it was).
Where SLMs dropped points
Misattributed settings, invented thresholds, or confident wrong-number hallucinations.
5. Evaluation Setup & Reproducibility
- Domain specification
- 2024 Automotive Technical Manual (dense technical, safety-critical procedures).
- Dataset and queries
- 357 structured queries (323 answerable, 34 unanswerable edge cases).
- Retrieval stack
- Hybrid BM25 + dense embedding (BGE-base) with Reciprocal Rank Fusion (RRF) achieving 0.975 gold-page recall.
- Compute and training
- Single A100 (80GB), LoRA rank 64 / alpha 128.
- Methodological caveat
- Automated evaluation using LLM-as-a-judge (Claude Haiku).
6. Architectural Decision Tree for CTOs
01
Claude Sonnet
Deploy Claude Sonnet if domain error cost outweighs GPU compute cost, or query volumes are under 100k/month.
02
Untuned base SLM
Deploy Untuned Base SLM + Hybrid RAG in air-gapped/on-prem environments where high answer completeness is required without model training overhead.
03
Fine-tuned SLM
Fine-tune an SLM if there is a strict SLA on latency (under 1.5 s), high query concurrency, and rigid response-formatting constraints.