As the human brain performs cognitive functions or generates spontaneous mental processes within ever-changing, real-world environments, states of the brain are inevitably nonstationary. This calls for innovative approaches to both obtain objective and quantitative insights into hidden cognitive and mental states and study the dynamics of brain states that give rise to behaviors and mental disorders. Despite electroencephalography (EEG) offering a noninvasive, portable, real-time measurement of brain activity, an urgent need remains for computational tools to effectively decode brain states from continuous, unlabeled EEG data, to quantitatively assess state changes, and to provide neuroscientific insights. In this dissertation, I present three computational approaches for quantitative assessment of brain-state dynamics by modeling multichannel, nonstationary EEG data at the level of functional brain sources. I first describe a hypothesis-driven approach which uses independent component analysis (ICA) to model distinct source activities under different brain states, characterized as stationary processes that involve specific brain networks. Two ICA models corresponding to alert and drowsy states in a sustained-attention experiment are identified, and a quantitative measure is proposed to assess the fluctuations in drowsiness. Next, I present a data-driven approach - Adaptive Mixture ICA (AMICA) - as an unsupervised-learning method for exploring nonstationary dynamics of unlabeled data. AMICA effectively characterizes EEG dynamics during sleep for automatic staging, reveals transitions between alert and drowsy states at millisecond resolution, and provides neuroscientific insights into active brain sources in each brain state. In the remaining chapters, I introduce the Online Recursive ICA (ORICA) approach for adaptive tracking of nonstationary sources that underlie continuous state changes, which incrementally updates the ICA model upon presentation of new data. A comparison study demonstrates ORICA’s fast convergence property and its feasibility for real-time and online processing. An adaptive forgetting factor is proposed to facilitate ORICA's adaptive capacity. For maximum utility, an open-source Real-time EEG Source-mapping Toolbox (REST) is built that integrates ORICA for the source-level analysis and interactive visualization of live-streaming data. These advances in computational tools provide a framework to study unlabeled, hidden cognitive or mental states and support developments toward objective, quantitative, continuous, and real-time assessments of the dynamic brain and its health.