Why Your RAG System Works in a Demo but Fails in Production

The demo is a controlled environment

A retrieval-augmented generation system can appear dependable when it runs against a small, familiar corpus. Production adds changing source material, ambiguous questions, and the operational limits that a local test rarely reveals.

Retrieval and generation fail differently

When an answer is wrong, the first useful question is whether the relevant context was absent or whether the model did not use it. Capturing that distinction makes evaluation more actionable than a single aggregate score.

• Track whether a relevant document was retrieved.
• Record the context supplied to the model for each request.
• Keep evaluation cases representative of real questions.

1. Define the failure you want to observe.
2. Collect traces for that failure mode.
3. Review retrieval and generation separately.

Small details such as a changed chunk boundary or an unhandled timeout can affect user-facing behavior long before an offline metric moves.

def answer_question(question: str) -> str:
    context = retrieve(question)
    return generate_answer(question=question, context=context)

A production RAG system is a system of dependencies, not only a prompt connected to a vector database.

For a broader introduction to retrieval patterns, see the Next.js documentation.

The implementation details will change as the system evolves, but observing the full request path is the foundation for making improvements with confidence.