Who This Is For
Intermediate.
Think of it like a librarian who's actually read every book on the shelf. Ask a question and they point straight to the right paragraph, instead of you skimming every book yourself.
What You'll Build
A local AI assistant answers a question from your own document set and identifies the source material.
Prerequisites
- Ollama and Open WebUI already running (see the Local AI article)
- A small test document set
Architecture
Retrieval-Augmented Generation (RAG), in plain terms: documents get processed and split into chunks, each chunk is turned into an embedding, embeddings live in a vector store, a question retrieves the most relevant chunks, and the local model answers using only that retrieved context.
Use Cases
- Homelab documentation
- Network diagrams and notes
- PDF manuals
- Personal research notes
Select an Embedding Model
Open WebUI supports several embedding models out of the box, a smaller embedding model is usually fine even if your chat model is large.
Upload a Test Document Set
Start small, five to ten representative documents, before importing an entire library.
Create Focused Collections
- Network docs
- Server notes
- Vendor manuals
- Personal research
Ask Answerable Questions
Start with questions you already know the answer to, so you can verify the retrieval is actually working correctly before trusting it.
Verify Citations
Always check the source document Open WebUI cites, treat an uncited answer with more suspicion.
Improve Results
- Use clear, descriptive file names
- Keep documents clean and well-formatted
- Use smaller, more focused collections rather than one giant dump
- Write clear, specific prompts
Security & Backup Notes
- "Local" does not automatically mean "secure", protect the Open WebUI interface itself, and back up the knowledge base like any other data
Troubleshooting
- Retrieval finds irrelevant chunks, collection is too broad; split into smaller, topic-focused collections
- PDF text extraction fails, scanned PDFs need OCR first (see Paperless-ngx) before they're searchable
- Context window too small, reduce the number of retrieved chunks or use a model with a larger context length
- Model invents answers, a smaller/weaker model or an overly broad collection increases hallucination risk; verify citations
- Document upload exposes sensitive info to other users, scope collections per-user if multiple people share the instance
Lab Finish Line
A local AI assistant answers a question from a user-supplied document and identifies the source material.