Knowledge discovery enhancements

The following are the knowledge discovery enhancements.

Knowledge base benchmarking

This release of TotalAgility introduces benchmarking capabilities to measure and evaluate knowledge discovery performance for accuracy, relevance, and response quality over time.

The Benchmarking feature provides visibility into AI answer quality and enables continuous improvement, helping organizations validate results, build trust, and confidently scale knowledge discovery initiatives.

The robust, standardized benchmarking methodology for knowledge discovery follows a consistent approach based on established best practices.

Benchmarking involves systematically evaluating system performance using a curated set of user-provided prompts and corresponding reference answers, and automated scoring. This approach ensures the evaluation is tailored to the user-defined information needs and expectations, providing a more relevant and accurate measure of system effectiveness.

The structured benchmarking framework not only supports ongoing system evaluation and improvement, but also aligns with industry standards for knowledge discovery and benchmarking.

A key prerequisite for effective benchmarking is that the knowledge base must be pre-populated using an appropriate chunking strategy. The choice of chunking strategy directly impacts the relevance and quality of retrieved information, making it foundational for any benchmarking process.

Users can create benchmark tests, run them against a knowledge base, view results, and then compare the results of two different runs of the same benchmark test.

You can export and import the benchmark tests into another system via the main Import/Export menu options.

To manage storage of test results, a new retention policy, Benchmark test runs, is available on the Time tab.

A new System Benchmark Evaluate Response system process is available in the System category. It helps you customize how the knowledge base response and references are evaluated and rated against the expected response within a benchmark test.

Split knowledge base calls

The knowledge base functionality is enhanced by splitting calls between Azure AI Search and LLMs, enabling a search-only option.

General knowledge base enhancements

General knowledge base enhancements include the following updates.

Override knowledge base and index for knowledge base activities using skins

When configuring skins on a template that contains a "Knowledge discovery agent" activity, you can override the provider and index. At runtime, the overridden knowledge base and index are automatically applied, giving you greater flexibility and control in tailoring the provider to different use cases.

Enable search only without using the LLM call

The new Search only mode in the "Knowledge discovery agent" activity enables independent, efficient document search and retrieval. You can use the returned references directly for your specific use cases (such as retrieving precise, unaltered excerpts from compliance documents for a legal case) without requiring a Large Language Model (LLM) to generate a response. This mode helps minimize the token usage and cost associated with LLMs.