DECEPTIcON: Automatic Deception Detection

Dataset: DECEPTIcON and DECEPTIcON+

We constructed a large-scale multimodal dataset focused on the truthfulness of public claims made by U.S. public figures on social media. Each sample includes synchronized video, audio, and transcripts, labeled using PolitiFact's fact-checking annotations. These labels cover six levels of veracity: True, Mostly True, Half True, Mostly False, False, and Pants-on-Fire.

The resulting dataset — DECEPTIcON and its extended version DECEPTIcON+ — includes over 600 annotated samples from more than 400 speakers, making it one of the largest and most systematically labeled resources in the domain of automatic deception detection.

← →

Model and Methodology

In our final approach, we employed modality-specific backbones and developed advanced fusion strategies for multimodal deception detection.

Unimodal Representations

For text, we extended pretrained models such as MPNet, SiEBERT, and Tweeteval (RoBERTa-based) to capture semantic, sentiment, and emotion-aware features. In audio, we utilized pretrained wav2vec 2.0 variants including W2V2-Emotion, W2V2-VADE, and W2V2-Gender, among others, fine-tuned for tasks like emotion recognition, gender identification, and speech recognition. For the visual modality, we leveraged models like VideoMAE, XCLIP, TimeSformer, and MARLIN to extract high-level spatiotemporal features from facial video data.

Multimodal Fusion

We experimented with and benchmarked various fusion methods:

• Soft Voting – Averaging predictions from each modality.
• Hard Voting – Majority-based prediction decision.
• Feature Concatenation – Merging modality features before classification.
• Simple Attention – Applying attention weights over concatenated representations.
• Hybrid Attention – Combining self- and cross-attention modules across modalities.
• MulT – Cross-modal transformer fusion without self-attention.
• MMJCA – A novel method leveraging joint cross-attention across modalities and positions.
• Representation Summarization – A novel lightweight summarization of modalities using learned tokens and cross-attention.

Visualizations of Various Fusion Methods

Simple Attention Fusion

Feature Summarization Fusion

MUlT Fusion

Hybrid Attention Fusion

MMJCA Fusion

← →

Key Results

In our experiments on the DECEPTIcON dataset, unimodal text-based models consistently outperformed audio and visual modalities. Among them, MMiniLM, Tweeteval, and ME5L achieved the highest classification metrics, especially in binary and 3-class tasks. Text models generally showed better generalization to unseen subjects and more stable performance across validation and test splits.

Audio-based methods performed slightly weaker than text but still showed promise, particularly W2V2-XLSR and W2V2-VADE. In some binary classification tasks, audio even surpassed text by a narrow margin, suggesting that prosodic features may be informative in certain deception contexts.

Visual-only methods were the weakest across all modalities. While VideoMAE models showed some relative advantage within the visual domain, the lack of fine-tuning and domain adaptation led to significantly lower performance compared to text and audio.

In terms of fusion strategies, hard voting (HV) emerged as the most effective method for multimodal integration, outperforming soft voting (SV) and feature-level fusion (FLF) in nearly every setting. Properly selected multimodal combinations led to notable gains, with up to +8% CCR and +8% F1 improvement over the best unimodal baselines in binary classification.

Similar performance trends were observed in the extended DECEPTIcON+ dataset as well, reinforcing the robustness of text-based cues and the effectiveness of HV-based fusion across different subject pools.

However, not all fusion attempts succeeded—some combinations suffered from performance drops due to poorly performing backbones introducing noise. These results underscore the importance of backbone complementarity over standalone performance when designing multimodal systems.

Detailed Results

Explainability

To understand how our model makes decisions, we analyzed attention patterns across text, audio, and visual data. In the textual modality, we observed that statements labeled as highly accurate often contained concrete, easily verifiable statistics. However, similar factual-looking elements were also found in misleading content, suggesting that deceptive speakers may intentionally mimic truthful language. Additionally, longer and more verbose statements tended to correlate with lower accuracy, likely reflecting attempts to bury misinformation in excessive detail.

On the audio side, the model's attention was closely aligned with the loudness of the speaker’s voice. It focused more on intense speech segments while ignoring silences and background noise, indicating a learned preference for high-information regions. Interestingly, pitch and tone had minimal impact, suggesting that vocal intensity mattered more than tonal nuance for deception detection.

Visually, the model concentrated its attention on facial action units, especially around the eyes and nose. These areas correspond to subtle muscle movements and micro-expressions, which are known to carry psychological cues related to deception. The attention heatmaps confirmed that facial expressions played a significant role in the model’s predictions, aligning with established behavioral research.

3-Modality Explainability Videos

2-Modality Explainability Videos

Personality Analysis

To model apparent personality (AP), we designed and trained deep learning architectures from scratch due to the lack of suitable pre-trained models. Our approach was centered around visual data from the FirstImpressionsV2 dataset, where each video is labeled using the OCEAN personality dimensions.

The models used spatial attention layers, recurrent memory connections, and temporal summarization to handle image sequences. Multiple architectures were explored—including CNN-RNN hybrids and transformer-based designs—with a focus on efficient temporal modeling and representation learning.

Despite extensive experimentation and integration of advanced fusion methods and attention mechanisms, the results did not reach expected levels of accuracy. The focus of our efforts remains on understanding and refining the modeling pipeline, particularly for challenging conditions such as in-the-wild face videos and label noise in personality traits.

Team

• Project Coordinator: Asst. Prof. Hamdi Dibeklioğlu
• PhD Students: Berat Biçer, Ergün Batuhan Kaynak
• MSc Students: Ali Azak, Murat Şahin, Uğur Can Altun, Kaan Gönç, Maria Raluca Eskin
• Engineers: Serhat Aras, Bahadır Durmaz
• Consultant: Prof. Dr. Mehmet Harma

Publications & Manuscripts

• Kaan Gönç, Hamdi Dibeklioğlu. Affect and Personality Aided Modeling of Transcribed Speech for Depression Severity Estimation. IEEE Transactions on Affective Computing, In Press, 2025. DOI:10.1109/TAFFC.2025.3560476
• Berat Biçer, Bahadır Durmaz, Serhat Aras, Hamdi Dibeklioğlu. DECEPTIcON: Bridging Gaps in In-the-Wild Deception Research. Under Review, 2025.