From Assistive to Autonomous: Building a Verifiable AI Environment for Test Engineering Workflows

Testing ensures that a device performs as specified. Introducing AI into this process raises an unresolved industry question: how can we trust AI within the very process designed to establish trust? AI can reduce time-to-market and testing costs by accelerating program development, improving result evaluation, and increasing engineering efficiency. However, general-purpose models lack two essential elements: the domain expertise of ATE vendors and customers, and a reliable method to verify output accuracy. This presentation argues that test engineering already provides the required discipline. The industry has long used golden references, correlation, guard-banding, and qualification gates to manage unproven capabilities. Applying these methods to AI offers a structured verification framework instead of relying solely on trust. We outline the components needed to establish this environment for the V93000: a curated domain knowledge base, knowledge-based retrieval, agent orchestration across test workflows, quality gates with human oversight, and a data-boundary architecture that keeps customer test IP within the customer's environment. We also share observations from prototype workflows and propose a standardized approach for evaluating AI in testing to support meaningful comparisons.

Key Technologies Covered

• Curated domain knowledge base for semiconductor test 
• Retrieval-augmented generation (RAG) grounded in that corpus rather than in general web data
• MCP-based agent orchestration: hub and servers exposing test-system capability as callable tools
• Agentic workflows with human-in-the-loop supervision and defined autonomy levels
• Execution-based evaluation harnesses for AI output, adapted from software-engineering benchmarks to test programs
• Quality gates: golden-reference comparison, correlation and guard-banding of AI-generated test content