Affective Audio Annotation of Public Speeches with Convolutional Clustering Neural Network

Public speaking is a critical skill in daily communication. While more practicing such as rehearsal is helpful to improve such a skill, lack of personalized feedback limits the effectiveness of practicing. Therefore, we formulate the task of personalized feedback as an affective audio annotation problem by learning knowledge from online public speech videos. Considering the great success of deep learning techniques such as convolutional neural networks in a wide range of applications including speech recognition and object recognition, we propose a novel convolutional clustering neural network (CCNN) to solve this multi-label classification problem. Instead of aggregating the features of different channels through pooling, we introduce a novel clustering layer to derive intermediate representation for improved annotation performance. In order to evaluate the performance of our proposed method, we purposely built an affective audio annotation dataset by collecting more than 2,000 video clips from the TED website. Experimental results on this dataset demonstrate that our proposed method outperforms traditional CNN-based approaches with a lower hamming loss for affective annotation.