Bi-dimensional modulation formats based on amplitude and phase signal modulation, are now commonly used in optical communications thanks to breakthroughs in the field of electronic and digital signal processing (DSP) required in coherent optical receivers. Photonic solutions could compensate for nowadays limitations of electrical circuits bandwidth by facilitating the signal processing parallelization. Photonic is particularly interesting for signal sampling thanks to available stable optical clocks. The heart of the present work concerns analog-to-digital conversion (ADC) as a key element in coherent detection. A prototype of linear optical sampling using an original solution for the optical sampling source, is built and validated with the successful equivalent time reconstruction of NRZ, QPSK and 16-QAM signals. Some optical and electrical limitations of the system are experimentally and numerically analyzed, notably the extinction ratio of the optical source or the ADC parameters (bandwidth, integration time, effective number of bits ENOB). Moreover, some new DSPs tools are developed for optical transmission using bi-dimensional modulation formats (amplitude and phase). Two solutions are proposed for IQ quadrature imbalance compensation in single carrier optical coherent transmission: an original method of maximum signal-to-noise ratio estimation (MSEM) and a new structure for joint compensation and equalization; these methods are experimentally and numerically validated with 16-QAM signals. Moreover, an improved solution for carrier recovery (frequency offset and phase estimation) based on a circular harmonic expansion of a maximum loglikelihood function is studied for the first time in the context of optical telecommunications. This solution which can operate with any kind of bi-dimensional modulation format signal is numerically validated up to 128-QAM. All the DSP tools developed in this work are finally used in a demonstration of a 10 Gbaud QPSK 100 km transmission experiment, featuring a strong non-linear phase noise limitation and regenerated using a phase preserving and power limiting function based on a photonic crystal nanocavity.