Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Basic Functionality Verification Items: End-to-End Test Cases from Knowledge Base Upload to Retrieval to Conversation

The value of end-to-end testing lies in confirming that the system is not just “a certain page can be opened” but that the entire chain from document ingestion to answer output is functional.

This topic has relatively clear public support. Dify officially provides a knowledge retrieval test entry point, Question Classifier node documentation, and knowledge base related documentation. Public articles also include practices around RAG retrieval, automated evaluation, and knowledge pipelines. Therefore, this can be written as a training piece on “the minimum reproducible test set for delivery sites.”

1. End-to-End Test Framework Confirmed by Public Sources

1. Knowledge Base Retrieval Testing Is Already an Officially Published Capability

Dify officially provides Knowledge Test Retrieval capabilities, indicating that “verify retrieval after upload, then verify Q&A” is a product-endorsed testing path.

2. End-to-End Verification Should Cover at Least Three Stages

Based on a synthesis of public sources, the minimum chain should cover:

  • Document enters the knowledge base
  • Whether retrieval produces a hit
  • Whether the final application answer is generated based on the retrieved content

3. Extended Test Cases Should Cover Complex Documents and Error Queries

Public RAG articles repeatedly emphasize that complex PDFs, parameter tuning, and error query handling significantly impact the experience. These should all be included in delivery testing.

  1. Upload a document
  2. Wait for indexing to complete
  3. Create an application bound to the knowledge base
  4. Submit a question that should definitely produce a hit
  5. Observe whether the answer correctly cites the relevant content

3. Extended Test Cases

  • Upload multiple documents
  • Upload a PDF containing tables
  • Adjust Top-K and Rerank settings
  • Verify whether the system appropriately declines to answer incorrect questions

4. Delivery Recommendations

During training, provide partners with a standard set of test texts and expected results for quick on-site validation.

Public Source References

note.com

  • “Information that should be there can’t be found” – Pitfalls encountered in Dify RAG chatbot development and building an automated evaluation system | https://note.com/kadinche/n/n87b77918dab9
  • AI that “re-searches on its own”: The forefront of advanced RAG construction with DeepSeek-R1 and Dify | https://note.com/nocode_solutions/n/nbe6c159a5460

zenn.dev / Official Documentation / Other Public Pages

  • Knowledge Retrieval Test | https://docs.dify.ai/ja/use-dify/knowledge/test-retrieval
  • Question Classifier - Dify Docs | https://docs.dify.ai/ja/use-dify/nodes/question-classifier
  • [Dify] RAG Complete Guide: Thorough Explanation of Mechanisms and Settings | https://zenn.dev/upgradetech/articles/ac9099a6489abe

Confirmed Information from Public Sources

  • The official knowledge retrieval test capability is suitable as a minimum verification step before acceptance
  • End-to-end testing should cover at least the three-stage chain of “upload -> retrieval -> conversation”
  • Complex PDFs, parameter tuning, and error query handling should be included as extended test items