Introducing New Microsoft Tool that Enables Developers to Generate AI Behavior Tests from Text Descriptions

AI researchers and labs have made noteworthy advancements in evaluating AI models concerning safety, compliance, and alignment. Nevertheless, companies and developers now encounter a new challenge: ensuring their AI systems perform as intended for their specific products or services.

To streamline this testing process, Microsoft has introduced ASSERT, which stands for Adaptive Spec-driven Scoring for Evaluation and Regression Testing.

As per Microsoft, this open-source framework facilitates the assessment of application-specific AI behavior by employing AI to transform high-level, natural-language descriptions of goals, policies, or intended actions into structured, scored tests that can be assessed.

ASSERT converts straightforward descriptions of an AI model’s expected behavior and policies into a structured list of acceptable and unacceptable actions. It produces problem scenarios and test cases, carries them out against the specified system, and evaluates the results. Furthermore, it documents the routes taken by the AI system, including intermediate actions and tool calls, allowing developers to pinpoint where failures arise.

Developers can also define system context, tools, and constraints to make evaluations more precise.

For example, a developer might set parameters for a document research AI agent, ensuring it doesn’t send emails outside the organization, restricts sensitive information to C-level executives, and offers concise summaries with prior context in mind. ASSERT would then create test cases to confirm compliance with these requirements.

Microsoft claims that this framework fills a gap that broader evaluations cannot address, especially when AI models must function according to the context, policies, and tools established by specific applications or products.

“One of the key insights we’ve gained is that evaluations are crucial for making informed decisions,” stated Sarah Bird, chief product officer of Responsible AI at Microsoft. “Without understanding the behavior of the AI system, it becomes difficult to ascertain whether it meets your organization’s standards. Our findings suggest that to achieve a trustworthy system, you need to evaluate many more dimensions specific to the application.”

Bird emphasized that ASSERT can be employed to evaluate systems during their development, after deployment, and even for ongoing monitoring.

The launch of ASSERT aligns with a wider transformation within the AI sector. As models grow increasingly advanced, researchers are emphasizing reproducible testing and regression assessments, with initiatives such as Stanford’s HELM, MLCommons’ AILuminate, and evaluation groups like METR introducing benchmarks to evaluate model behavior under various circumstances.