I’ll be sharing a step-by-step series on how to build a completely local, free, enterprise-grade knowledge base Q&A system.

The reason for choosing a local setup is simple: when you rely on external models or services, your private data and user questions are sent to a third party. That brings the risk of exposing sensitive information, and in the worst case, data leakage. Keeping everything local makes your data much safer.

There is another advantage. As your business grows, the model may no longer fit your needs very well. If the model is deployed locally, it becomes much easier to fine-tune it and adapt it to your actual scenario.

That gives you both security and flexibility.

Where a knowledge base Q&A system can be used

Knowledge base Q&A has a wide range of use cases. Here are two simple commercial examples.

For individuals, it can be used for things like organizing notes, paper interpretation, literature search, and document Q&A. To show this, I’ll use LightPDF’s document chat feature: https://lightpdf.cn/chatdoc

Upload any PDF document, wait for parsing to finish, and then ask questions about its content. You can quickly extract the part you need and find information directly from the document, which greatly improves the efficiency of reading and reviewing documents.

For enterprises, it can be used for intelligent customer service, data retention, or digital employees. I’ll also use GitMind’s online customer service as an example: https://gitmind.cn

You can ask the customer service system questions, and the AI will answer accurately. The model itself does not magically know these answers, because it does not understand your business or your internal data. Those responses come from the company’s own knowledge base. In this way, you get a 7 x 24 intelligent customer service system that reduces cost, improves efficiency, and raises customer satisfaction.

Hardware requirements

The table below gives a rough idea of the minimum and recommended configuration for different scales:

<table> <thead> <tr> <th>Environment</th> <th>Minimum</th> <th>Recommended</th> </tr> </thead> <tbody> <tr> <td>Testing</td> <td>2 cores, 2 GB</td> <td>2 cores, 4 GB</td> </tr> <tr> <td>1M vector entries</td> <td>4 cores, 8 GB, 50 GB</td> <td>4 cores, 16 GB, 50 GB</td> </tr> <tr> <td>5M vector entries</td> <td>8 cores, 32 GB, 200 GB</td> <td>16 cores, 48 GB, 200 GB</td> </tr> <tr> <td>10M vector entries</td> <td>16 cores, 48 GB, 200 GB</td> <td>32 cores, 64 GB, 500 GB</td> </tr> </tbody> </table>

The entire system is built around the architecture below.

• Retrieval: use embedding models or hybrid retrieval methods to find relevant documents in the knowledge base.
• Augmentation: attach the retrieved documents to the prompt for enhancement, also known as ICL (In Context Learning).
• Generation: use a large language model to generate the answer.

This approach has several clear advantages:

1. It helps reduce hallucinations and improves answer accuracy.
2. Unlike fine-tuning, RAG offers a degree of observability and inspectability.
3. It is low-cost and relatively easy to implement.

Install Docker and Ollama

First, install Docker Desktop: https://www.docker.com/products/docker-desktop/

Then install Ollama: https://ollama.com/

For the demo, I use Alibaba’s Qwen model. Choose the right model according to your computer’s hardware.

In general, the larger the model, the better the result, but the higher the hardware requirements.

• 4B model: 3 GB memory
• 7B model: 8 GB memory
• 13B model: 16 GB memory
• 70B model: 64 GB memory

A very important note: if you want to use it for commercial purposes, you need to apply for authorization. Otherwise, it is only for personal learning.

• Qwen license information: https://ollama.com/library/qwen:latest/blobs/41c2cf8c272f
• Apply for commercial authorization: https://dashscope.console.aliyun.com/openModelApply/qianwen

Download the model

ollama pull qwen:7b

Test the API request

After the model has been downloaded and installed, run the following command to check whether the API is working.

On Mac/Linux:

``` 1 2 3 4 5 6 7 8 9 10 11 12 curl 'http://localhost:11434/api/chat' \ --data '{ "model": "qwen:7b", "messages": [ { "role": "user", "content": "你是谁" } ], "temperature": 0.1, "stream": false }'

##### On Windows:

``` 1 2 
curl "http://localhost:11434/api/chat" ^ --data "{\"model\": \"qwen:7b\", \"messages\": [{\"role\": \"user\", \"content\": \"你是谁\"}], \"temperature\": 0.1, \"stream\": false}"

Download the embedding model

I use DmetaSoul’s Chinese embedding model. It is claimed to have reached the top position among open-source models on the MTEB Chinese leaderboard: https://huggingface.co/DMetaSoul/Dmeta-embedding-zh/blob/main/README_zh.md

Download the model

ollama pull shaw/dmeta-embedding-zh

Test the API request

After the model has been downloaded and installed, run the following command to check whether the API is working.

On Mac/Linux:

``` 1 2 3 4 5 curl 'http://localhost:11434/api/embeddings' \ --data '{ "model": "shaw/dmeta-embedding-zh", "prompt": "天空是灰色的" }'

##### On Windows:

``` 1 2 
curl "http://localhost:11434/api/embeddings" ^ --data "{\"model\": \"shaw/dmeta-embedding-zh\", \"prompt\": \"天空是灰色的\"}"

Install FastGPT and its dependencies

1. Download the Docker file and config file

You can download them manually and put them in the same folder:

• docker-compose.yml: https://harryai.cc/kbqa/docker-compose.yml
• config.json: https://harryai.cc/kbqa/config.json

You can also run the following command to download them automatically:

``` 1 2 3 4 mkdir kbqa cd kbqa curl -O https://harryai.cc/kbqa/docker-compose.yml curl -O https://harryai.cc/kbqa/config.json

### 2. Start Docker

``` 1 
docker-compose up

Notes:

• The first startup may be a little slow because the images need to be pulled. Later starts will be much faster.
• If the first startup fails, run docker-compose up again a few times.

3. Access FastGPT and OneAPI

When you visit the system for the first time, there will be no data inside it yet. The data shown in the screenshots was created during my own testing.

FastGPT

Open: http://localhost:3000

Username: root Password: 1234
You can change the password through the DEFAULT_ROOT_PSW environment variable in docker-compose.yml.

OneAPI

Open: http://localhost:3001

Username: root Password: 123456

Configure OneAPI

Add models here: http://localhost:3001/channel

1. Configure the large language model

1. Type: Ollama
2. Name: ollama-qwen:7b
3. Model: enter the custom model name, qwen:7b
4. Key: any value is fine, for example 123
5. Proxy: http://host.docker.internal:11434

2. Configure the embedding model

1. Type: Ollama
2. Name: ollama-dmeta-embedding-zh
3. Model: enter the custom model name, for example shaw/dmeta-embedding-zh
4. Key: any value is fine, for example 123
5. Proxy: http://host.docker.internal:11434

Important notes:

1. The model name must match the name of the model installed in Ollama. You can check it with the ollama list command.
2. The proxy address must be http://host.docker.internal:11434, otherwise OneAPI will not be able to reach Ollama.

By default, Docker runs services in bridge mode. That means containers use Docker’s own virtual network and can talk to each other, but they cannot directly access services running on the host machine, which is your computer.

Ollama runs on your computer, while OneAPI runs inside a Docker container. Because of that, OneAPI cannot access Ollama directly.

Docker provides a supported solution for this: point the container to host.docker.internal, which resolves to the host machine’s IP.

See the official documentation on connecting from a container to a service on the host.

Configure FastGPT

First create a simple application to build a basic AI chat workflow. The goal is to verify that both the OneAPI interface and the Ollama interface are working correctly.