New Exam Security Questions in the AI Era: Comparing AI-Generated Item Similarity Between Naive and Detailed Prompting Approaches

With the advent of powerful generative AI tools, exam security is a growing concern. Examinees can now easily ask an AI tool to generate exam questions in the style of a board certification exam. Meanwhile, many certifying boards are exploring these same technologies to streamline item writing. These uses raise questions about implications for exam security: could AI-generated operational items be unwittingly exposed by examinees generating practice items?

Our study examines the similarity of AI-generated exam questions from two development strategies. In the naïve approach, a model is provided with publicly available exam blueprints and minimal stylistic prompts. In the other, the model leverages proprietary guidelines and sample items used by a board. By analyzing key textual and structural features via cosine similarity1, we assess the similarity of the resulting items.

High similarity would underscore the need for greater creativity in item development to preserve the integrity of testing. Conversely, low similarity would suggest that internal materials already provide a robust safeguard against duplication. Through this comparison, our work illuminates the evolving dynamic between AI technologies and high-stakes testing, offering recommendations to other boards navigating this new landscape.

Our findings will inform policy and practice, guiding boards to adopt strategic item development processes in an era when AI is widely accessible. By offering a comparative lens on question generation methods, our investigation will help boards balance AI's efficiency gains with the imperative for test security. This ensures that as technology evolves, professional certifications remain credible, fair, and rigorous.