MWAIS 2024 Proceedings

Abstract

With the prevalence of artificial intelligence (AI) models in daily life and business (e.g., security cameras, customer service chatbots, supply chain), it is essential to develop competencies in AI safety, security, and assessment thereof. The process includes examining both exploits and unintended consequences. AI Red Teams competed for the first time in DEF CON 2023 in a program co-hosted by the White House. Red-teaming in AI still lacks a clear scope or required set of competencies, and in the 2023 competition, it manifested itself as prompt hacking (prompt injecting). However, not all AI models are large language models, and prompt hacking is one of the many possible exploits to which the broader set of AI tools can be vulnerable (e.g., HopSkipJump attacks, Chen et al. 2020). The majority of organizations will be using third-party pre-trained models (HiddenLayer Report 2024); therefore, foundational skills in both AI and security will be essential for securely and safely incorporating third-party models in production. It is important that educators and researchers in computing fields explore this landscape in order to remedy the lag in workforce development and research endeavors in the area. In this work, we review common AI vulnerabilities (in code, training, model, network, and output) and the existing literature on AI red-teaming and aim to formulate the scope. We then engage in requirement analysis to identify competencies and foundational knowledge that contribute to this area’s curriculum work and research pursuits.

Download

Share

COinS