AI Configuration
Customize the RAG pipeline, system prompts, search settings, and UI
AI Configuration
This page covers how to customize the AI behavior in VectraDocs. The AI is built on LangChain and uses Orama for client-side vector search.
Which Version Are You Using?
This page is for the main VectraDocs Next.js starter.
If you're using VitePress, Docusaurus, or Scalar plugins with an external backend, see Backend Setup instead.
How It Works
VectraDocs uses Retrieval-Augmented Generation (RAG):
- User asks a question → The frontend sends the message to
/api/chat - Search relevant docs → Orama searches the documentation index
- Build context → The top results are combined into a context string
- Generate response → LangChain sends the question + context to the LLM
- Stream response → The AI's answer is streamed back to the user
1. Customizing the System Prompt
The AI's "personality" is defined in app/api/chat/route.ts.
Find the ChatPromptTemplate:
const prompt = ChatPromptTemplate.fromMessages([
["system", `You are a helpful documentation assistant for "VectraDocs".
Answer questions based ONLY on the provided documentation context.
Context:
{context}
IMPORTANT RULES:
1. Only answer based on the provided context
2. If the answer is not in the context, apologize and say you don't have information
3. Be concise but helpful
4. Include code examples when relevant
5. At the END of your response, ALWAYS include a "📚 References" section
6. Use the EXACT URLs from the context (relative paths like /docs/installation)
7. Format references as markdown links: [Page Title](/docs/page-path)
Example reference section:
📚 **References:**
- [Installation Guide](/docs/installation)
- [Backend Setup](/docs/backend-setup)`],
new MessagesPlaceholder("history"),
["human", "{input}"],
]);Automatic References
The AI automatically includes documentation references at the end of each response. URLs are relative paths (like /docs/installation) so they work on any domain.
Customization Ideas
| Goal | Prompt Addition |
|---|---|
| Brand voice | "You represent the ExampleCo brand. Be friendly and professional." |
| Code language | "Always provide code examples in TypeScript." |
| Restrict topics | "Only answer questions about our product. Decline unrelated questions." |
| Response length | "Keep responses under 200 words unless code is needed." |
| Include sources | "At the end, list the documentation pages you referenced." |
2. Adjusting Search Settings
The RAG system uses Orama to find relevant documentation. You can tune this in app/api/chat/route.ts:
// Search for relevant documents
const searchResult = await search(db, {
term: lastMessage,
limit: 3 // Number of results to include
});Settings
| Parameter | Default | Description |
|---|---|---|
limit | 3 | Number of doc sections to include in context |
tolerance | 0.5 | Fuzzy matching tolerance (0-1) |
threshold | 0 | Minimum score to include a result |
Recommendations
- Simple docs:
limit: 3is usually enough - Complex docs: Increase to
limit: 5-10 - Token budget concerns: Keep
limitlow, use summaries - Accuracy issues: Increase
limitand addthreshold: 0.3
3. Switching LLM Providers
VectraDocs uses environment variables to configure the LLM:
LLM_BASE_URL="https://api.openai.com/v1"
LLM_API_KEY="sk-..."
LLM_MODEL="gpt-4o"Supported Providers
| Provider | Base URL | Models |
|---|---|---|
| OpenAI | https://api.openai.com/v1 | gpt-4o, gpt-4o-mini, gpt-3.5-turbo |
| Anthropic | https://api.anthropic.com/v1 | claude-3-sonnet, claude-3-opus |
| Groq | https://api.groq.com/openai/v1 | llama-3.1-70b-versatile |
| Together AI | https://api.together.xyz/v1 | meta-llama/Llama-3-70b-chat |
| Ollama | http://localhost:11434/v1 | llama3, mistral, etc. |
Using Anthropic (Claude)
For deeper Claude integration:
npm install @langchain/anthropicUpdate app/api/chat/route.ts:
import { ChatAnthropic } from "@langchain/anthropic";
const model = new ChatAnthropic({
model: "claude-3-sonnet-20240229",
apiKey: process.env.ANTHROPIC_API_KEY,
});4. UI Customization
The chat UI lives in components/ai-chat.tsx.
Accent Color
Change the primary color throughout the component:
// Find and replace occurrences of:
border-orange-500 → border-purple-500
bg-orange-500 → bg-purple-500
text-orange-500 → text-purple-500Floating Bar Position
Modify the position in the floating bar styles:
// Bottom center (default)
className="fixed bottom-8 left-1/2 -translate-x-1/2"
// Bottom right
className="fixed bottom-8 right-8"
// Bottom left
className="fixed bottom-8 left-8"Markdown Rendering
Customize how AI responses render:
<ReactMarkdown
components={{
h1: ({ children }) => <h1 className="text-2xl font-bold mt-4">{children}</h1>,
h2: ({ children }) => <h2 className="text-xl font-semibold mt-3">{children}</h2>,
code: ({ children }) => (
<code className="bg-zinc-800 px-1.5 py-0.5 rounded text-orange-400">
{children}
</code>
),
a: ({ href, children }) => (
<a href={href} className="text-orange-500 hover:underline">
{children}
</a>
),
}}
>
{message.content}
</ReactMarkdown>5. Adding Hidden AI Context
You can add hidden metadata in your MDX files that the AI can search but users won't see:
---
title: My Page
description: A page about something
---
# My Page
{/* AI Context: This page covers X, Y, and Z. Related topics include A and B. Common questions: How do I...? What is...? */}
Visible content here...This is powered by MDX comments ({/* ... */}) which are:
- ✅ Indexed by the search system
- ✅ Available to the AI as context
- ❌ Not rendered to users
6. Using Vector Databases
For larger documentation sites, you might want to use a cloud vector database:
Pinecone
npm install @pinecone-database/pinecone @langchain/pineconeimport { Pinecone } from "@pinecone-database/pinecone";
import { PineconeStore } from "@langchain/pinecone";
const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pinecone.Index("your-index");
const vectorStore = await PineconeStore.fromExistingIndex(embeddings, {
pineconeIndex: index,
});
const retriever = vectorStore.asRetriever({ k: 5 });Supabase
npm install @supabase/supabase-js @langchain/communityCheck LangChain docs for Supabase vector store integration.
Troubleshooting
AI gives generic or wrong answers
- Check that
npm run build:indexwas run after editing docs - Increase
limitin search settings - Add more specific content to your documentation
AI takes too long to respond
- Use a faster model (
gpt-4o-minivsgpt-4o) - Reduce
limitin search settings - Use a local model with Ollama
AI refuses to answer valid questions
- The answer might not be in your docs - add it!
- Adjust the system prompt to be less restrictive
- Check if the question matches indexed content
Next Steps
- Installation: Set up VectraDocs
- Backend Setup: Create backends for plugins