Special attention has been paid to SI (speaker identification) of whispered speech with perceptible mood. The database is developed for SI of whispers with different purposes. To prove the efficiency of delivering the speaker’s feelings, the perceptual test is carried out on the whispered speech, and the A-V (Arousal-Valance) space is introduced to judge the speaker’s status. The LFA (Latent Factor Analysis) algorithm, following the classification of the speaker’s mood, is introduced in detail and proved to be efficient. The proposed SI scheme with modified spectrum parameters provides much more satisfactory accuracy than the baseline system.