Open Access Open Access  Restricted Access Subscription Access

Voice-Image Cross-Modal Human Fatigue Detection Based on CNN-ELM Hybrid Model

Shuxi Chen,
Yiyang Sun,
Jianlin Qiu,
Haifei Zhang,
Hao Chen,

Abstract


The deepening of human fatigue will lead to the reduction of exercise ability and work efficiency, the increase of errors and accidents, and even the occurrence of organic diseases. Obviously, it is significant to understand the impact of human fatigue on the health, safe production and safe work of different people. At present, fatigue detection is mostly carried out through EEG and EMG signals. These methods usually have the disadvantages of contact and non-realtime.
In response to the aforementioned issues in the process of human fatigue detection, this article effectively applies the visual image analysis method of spectrograms to human fatigue detection and proposes a cross-modal human fatigue detection method based on speech spectral image recognition. First, Mel spectrograms of speech segments in the corpus are extracted, and a fatigue spectrogram data set is established. A deep learning model is established through convolutional neural network (CNN) and extreme learning machine (ELM) for spectral image recognition and fatigue detection. CNN is used to extract features from the input image. The feature mapping will eventually be encoded into a one-dimensional vector and sent to ELM for classification. The experimental results indicate that the speech spectrum image features extracted by this method have better fatigue characterization ability than traditional acoustic features.

Keywords


Fatigue, Mel spectrogram, Cross-modal, CNN, ELM

Citation Format:
Shuxi Chen, Yiyang Sun, Jianlin Qiu, Haifei Zhang, Hao Chen, "Voice-Image Cross-Modal Human Fatigue Detection Based on CNN-ELM Hybrid Model," Journal of Internet Technology, vol. 26, no. 5 , pp. 597-606, Sep. 2025.

Full Text:

PDF

Refbacks

  • There are currently no refbacks.





Published by Executive Committee, Taiwan Academic Network, Ministry of Education, Taipei, Taiwan, R.O.C
JIT Editorial Office, Office of Library and Information Services, National Dong Hwa University
No. 1, Sec. 2, Da Hsueh Rd., Shoufeng, Hualien 974301, Taiwan, R.O.C.
Tel: +886-3-931-7314  E-mail: jit.editorial@gmail.com