Nonlinear effects in optical fiber set the ultimate limit to the channel capacity in long-haul optical transmission systems. Advanced nonlinear compensation techniques such as digital backpropagation (DBP) have been proposed as a solution to overcome the channel capacity crunch. However, given theircomputational complexity, in a practical environment their performance gainremains very limited. This triggered a search for a novel communication system design that takes fiber nonlinearity into consideration. A new nonlinearcommunication method, based on the theory of the inverse spectral transform, has been proposed to overcome the nonlinear capacity crunch. Thismethod, originally proposed by Hasegawa in 1993 and called eigenvalue (ormulti-soliton) communication, is based on the fundamental observation thatthe nonlinear spectrum of an optical signal is invariant (except for a triviallinear phase shift) upon propagation in the fiber channel, as described bythe nonlinear Schrödinger equation (NLSE). This means that if the directspectral transform (also known as nonlinear Fourier transform (NFT)) ofthe received signal can be computed, the eigenvalue spectrum can be fullyrecovered.This thesis focuses on a NFT-based communication technique known as nonlinear frequency division multiplexing (NFDM). The NFDM optical systemis numerically assessed and experimentally demonstrated. First, the structure of the proposed single-polarization NFDM system using the continuousspectrum in the normal dispersion regime is presented. To that end, theNFT of the vector NLSE, or Manakov system, was numerically developed.Based on these algorithms the NFDM method was extended to polarizationdivision multiplexed (PMD) systems, and experimentally validated for thefirst time using the continuous spectrum. Finally, the experiment will bereplicated in the anomalous dispersion regime.Additional numerical studies are presented, in order to investigate the implementation challenges of the proposed NFDM techniques for the continuousspectrum modulation.