The Power of Silence - A Multimodal System to Detect Document Fraud with Nonverbal Behaviors

Deception poses significant security threats to modern society. To identify deceptive individuals carrying concealing information, an automated interviewing system was developed to collect and analyze multimodal behavioral data from participants in a rapid screening environment with a fraudulent document scenario. Features for response latency and facial action units were extracted from the silent moment after the end of a question and prior to participants’ vocal responses when participants were expected to experience high cognitive load. The sparse group lasso (SGL) method was used for dimension reduction. Predictive models with features selected from SGL achieved an accuracy rate of 78.8%. This study shows that behavioral features from the silence time could potentially be integrated with multimodal deception detection systems to further boost accuracy, support decisions and guard public security.