RAG in Production: What the Demo Never Tells You

How RAG works, in short

There are three basic steps. Split documents into small pieces and store them in an index, find the pieces relevant to a user's question, and hand those pieces to the model to write the answer. The benefit is that answers are grounded in documents you control, and you can update knowledge without retraining a model.

What many people miss is that the model can only be as good as what was retrieved. If the right piece is not found, the best model cannot save the answer. So when debugging, check retrieval first rather than swapping the model.

Chunking: split documents the way people ask questions

Pieces that are too small lose context around the key sentence. Pieces that are too large drag in unrelated text and blur the search. A reliable approach is to split along the document's own structure, such as headings, clauses or table rows, and attach the heading and document title to every piece.

Thai needs extra care because words are not separated by spaces. A poor word segmenter or tokenizer can break keyword search. Try several real Thai queries, including typos and mixed-language phrases like "รีเฟรช token", and check whether the right piece comes back.

• Split by headings or real structure, not by character count alone.
• Attach the document title, section heading and last-updated date to each chunk.
• Test with questions real users type, not ones the team wrote.

Retrieval quality: start with hybrid search

Embedding search understands meaning but can miss product codes, section numbers and proper names. Keyword search is precise on exact terms but blind to synonyms. Combining the two, usually called hybrid search, tends to beat either one alone on real workloads.

After pulling a few dozen candidates, add a rerank step that keeps only the handful that matter before they reach the model. It improves accuracy and cuts cost because less text is sent. Also filter by the user's permissions so nobody sees a document through the AI that they could not open directly.

Citations make answers checkable

An answer with no source forces users to trust or distrust it by guesswork. Show the document name, section and a link to the original next to each main claim. Users can verify quickly, and your team can see which piece produced the answer when something goes wrong.

Also teach the system to decline. If the retrieved pieces do not contain the answer, it should say it found nothing in the available documents and point to a person who can help, instead of writing something that sounds sure.

A confident answer with no source is more dangerous than an honest "I found nothing".

The evaluation loop: measure retrieval and answers separately

Measure two layers apart. The first is retrieval: is the right piece in the top five results? The second is the answer: is it faithful to the documents and complete? One blended score hides where to fix things.

Collect real questions from your logs, add a thumbs up or down button, and review the misses regularly. Every miss should become a new test case. That loop is what makes the system improve a little each week instead of degrading silently.

Failure modes after the demo

Common ones include duplicate or outdated files left in the index so the system quotes a policy that was withdrawn, scanned PDFs and complex tables that convert to garbled text, and questions that need facts from several documents, which basic RAG handles poorly.

The fixes are plain. Assign an owner and an update schedule for each document set, inspect file conversion before ingesting, split complex questions into sub-questions or tell the user to ask a specialist, and watch cost and response time, since multi-step retrieval raises both.