Enhanced New Words Recognition Based on Multi-level Semantic Vectors and Multi-task Learning Models

Abstract

Neologism discovery is a basic task in natural language processing, and it is very important to improve the performance of various downstream tasks. In order to solve problems such as word segmentation errors easily caused by existing technologies and incomplete capture of word semantic information, this paper proposes an enhanced new words recognition based on multi-level semantic vectors and multi-task learning models, aiming to solve the problems of word segmentation errors, incomplete semantic capture and dynamic recognition in existing technologies. First, an improved hash algorithm is used to generate dictionaries. A multi-level time series model is used to identify potential neologism candidates and map them to a high-dimensional vector space to generate synthetic semantic vectors. Then, a context-semantic graph model is constructed to analyze the context compatibility of words, and the sentiment score and domain relevance score are calculated through the multi-task learning model. Finally, the comprehensive score is used to identify new words. Experimental results show that this method has significant advantages in accuracy, semantic understanding and application range.