Social engineering is a major issue for organizations. In this paper, we propose that increasing adversarial thinking can improve individual resistance to social engineering attacks. We formalize our understanding of adversarial thinking using Utility Theory. Next a measure of adversarial thinking in a text-based context. Lastly the paper reports on two studies that demonstrate the effectiveness of the newly developed measure. We show that the measure of adversarial thinking has variability, can be manipulated with training, and that it is not influenced significantly by priming. The paper also shows that social engineering training has an influence on adversarial thinking and that practicing against an adversarial conversational agent has a positive influence on adversarial thinking.